Water conservancy monitoring data abnormal state identification method based on multi-modal learning

Abstract

The invention relates to a water conservancy monitoring data abnormal state identification method based on multi-modal learning, and the method specifically comprises the following steps: deploying a heterogeneous sensor network at a water conservancy facility, collecting original monitoring data, combining the text modal data of a work log, and carrying out the abnormal state marking to form a training data set; constructing uniform time grid tensor alignment multi-modal data, and obtaining a complete alignment tensor by adopting a low-rank tensor completion algorithm of fusion modal mutual information constraint; constructing a machine learning model comprising a cross-modal feature collaborative enhancement module, a heterogeneous feature projection and gating fusion module and a spatio-temporal context sensing anomaly recognition module, inputting a complete alignment tensor into the model to obtain an anomaly probability value, and training the model through a loss function; new data is collected, preprocessed and input into the trained model, an abnormal probability value is compared with a set threshold value, and an abnormal type is judged. According to the method, the feature representation capability is enhanced through multi-module cooperation, and the accuracy and timeliness of water conservancy facility anomaly recognition can be improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and water conservancy monitoring technology, and in particular to a method for identifying abnormal states in water conservancy monitoring data based on multimodal learning. Background Technology

[0002] With the continuous development of water conservancy projects and intelligent monitoring systems, conventional water conservancy system monitoring methods have gradually revealed some shortcomings, especially in multimodal data fusion and anomaly detection. Water conservancy systems involve a wide variety of sensors, including temperature and humidity sensors, flow meters, and pressure sensors. The data they generate often have different temporal characteristics and frequencies, which poses a significant challenge to data fusion and anomaly detection. At the same time, existing anomaly detection methods are usually based on single-modal data and lack effective multimodal data fusion mechanisms. This makes it difficult to fully leverage the advantages of different data sources in practical applications, easily overlooking complex correlations between data, and affecting the accuracy and robustness of the system.

[0003] Chinese invention patent CN120611329B proposes a method and system for monitoring and early warning of dam safety in water conservancy projects. It achieves comprehensive perception and accurate early warning of the dam's structural health status through composite sensing technology and intelligent analysis algorithms. A three-dimensional monitoring network integrating the interface and structure is constructed using the coordinated deployment of micromechanical resonant sensors and distributed fiber optic sensors. A three-dimensional interface stripping feature spectrum is built based on time-frequency joint analysis technology to accurately identify the bonding degradation state between the sensor and the dam body. An anomaly distribution matrix of the strain field is established through spatial correlation modeling to achieve precise localization of internal damage. A dual-channel feature fusion network based on an attention mechanism and a deep neural network evaluator are designed to perform multi-dimensional correlation analysis between the interface state and structural damage characteristics. Finally, a three-level linkage early warning decision tree is used to achieve a progressive response from data verification, multi-source verification to emergency linkage.

[0004] Existing technologies have the following shortcomings in practical applications: Conventional anomaly identification methods often ignore the potential correlation between multimodal data, and cannot effectively handle the frequency inconsistencies and missing data between different modalities, resulting in distorted fusion features and an inability to accurately reflect the true state; Current technologies mostly use shallow cross-feature modeling in feature representation, failing to fully capture complex high-order nonlinear feature relationships, resulting in insufficient feature representation capabilities for multimodal data and an inability to effectively adapt to the complexity of water conservancy monitoring data; Existing anomaly identification methods often ignore the dynamic propagation characteristics of spatiotemporal information, lack modeling of the propagation mechanism of anomalies in water conservancy systems, and are prone to inaccurate identification of instantaneous anomalies or complex propagation paths; Conventional loss functions usually only optimize classification accuracy, ignoring the coordination between multimodal data and the dependence of spatiotemporal propagation, making it difficult for the model to generalize to different water conservancy monitoring scenarios in complex environments.

[0005] Therefore, this invention proposes a method for identifying abnormal states in water conservancy monitoring data based on multimodal learning to solve the above problems. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention develops a method for identifying abnormal states in water conservancy monitoring data based on multimodal learning. This invention achieves effective alignment and completion of multimodal data, enhances feature representation capabilities through multi-module collaboration, improves the accuracy and timeliness of anomaly identification in water conservancy facilities, and provides reliable assurance for the safe operation of water conservancy facilities.

[0007] The technical solution of this invention to solve the technical problem is a method for identifying abnormal states in water conservancy monitoring data based on multimodal learning, comprising the following steps: S1. Deploy a heterogeneous sensor network on water conservancy facilities to collect multimodal raw monitoring data, including time-series modal data and discrete modal data, and obtain text modal data from work logs. Then, label the multimodal raw monitoring data with abnormal states to form a training dataset. S2. Construct a unified time grid tensor to align the original multimodal monitoring data, and use a low-rank tensor completion algorithm that integrates modal mutual information constraints to obtain a complete aligned tensor; S3. Construct and train a machine learning model. The model includes a cross-modal feature collaborative enhancement module, a heterogeneous feature projection and gating fusion module, and a spatiotemporal context-aware anomaly recognition module. Input the complete alignment tensor into the machine learning model, and pass it through each module in turn to obtain the anomaly probability value. Then calculate the model's loss function and train the model based on the loss function. S4. Collect new data, preprocess it, and input it into the trained model to obtain the anomaly probability value. Compare it with the set threshold to determine the anomaly type.

[0008] S1 is as follows: Time-series modal data includes data continuously collected by water level gauges, flow meters, and pressure sensors, with floating-point values ​​recorded at a frequency of minutes. Discrete modal data is collected through the gate opening controller and pump operation module, and the equipment status code is recorded in an event-triggered manner. Text modal data is obtained from system logs entered by maintenance personnel; Data labeling is done manually based on water conservancy expert rules and historical fault records, labeling each sensor with an abnormal status at each moment. 0 indicates normal and 1 indicates abnormal. Abnormal categories include equipment failure, environmental abnormality, and text description of fault.

[0009] S2 is as follows: S2.1 Construct a unified spatiotemporal grid tensor by mapping non-uniformly sampled multimodal data onto a unified spatiotemporal grid, where each element stores the original observation or missing label according to its index position; S2.2. The complete tensor is obtained by optimizing the objective function that combines low-rank constraints and modal mutual information regularization, and by iteratively solving the problem using the alternating direction multiplier method. Specifically, the complete tensor is obtained by solving the objective function through optimization. Based on the iterative optimization process of the alternating direction multiplier method, the complete tensor of the unified spatiotemporal grid is estimated. The final output dimension is A fully aligned tensor is a complete tensor after tensor completion and feature unification. Indicates the maximum time step. Indicates the number of sensors. This represents a unified feature dimension.

[0010] S3 is as follows: A machine learning model is constructed and trained. The fully aligned tensor is input into the model and first passes through a cross-modal feature co-enhancement module. This module uses a tensor ring decomposition algorithm to extract spatiotemporal modal co-factors from the completed tensor and concatenates them with local features. Then, adversarial training with mutual information regularization is used to generate an enhanced feature tensor. The enhanced feature tensor then passes through a heterogeneous feature projection and gating fusion module. Modality-specific projection maps heterogeneous features to a unified space, and a dual-gating mechanism is used to dynamically aggregate cross-modal information to generate cross-modal fused features. Finally, the cross-modal fused features pass through a spatiotemporal context-aware anomaly detection module. This module uses a classification network based on multi-head spatiotemporal graph attention and temporal convolutional gating. It models sensor spatial dependencies through graph structure, captures multi-scale temporal patterns, and combines an adaptive anomaly propagation mechanism for dynamic classification to generate anomaly probability values.

[0011] The cross-modal feature collaborative enhancement module is as follows: The complete aligned tensor is decomposed into time factor matrix, spatial factor matrix and modality factor tensor by tensor ring decomposition algorithm, and the collaborative features are reconstructed by outer product operation. Then, it is concatenated with the local features extracted by 3D convolution to form an enhanced collaborative feature tensor. The generator combines collaborative features with random noise to generate enhanced features. The discriminator distinguishes between real features and enhanced features. During adversarial training, mutual information regularization is used to maximize the mutual information between real features and enhanced features, resulting in an enhanced feature tensor.

[0012] The heterogeneous feature projection and gating fusion module is as follows: By constructing an independent projection network for each modality, the enhanced features are concatenated with the spatiotemporal factors and then processed by layer normalization and Gaussian error linear unit activation function to obtain the projection features of each modality. By calculating the gating weights and reset weights for each modality, where the gating weights are generated based on the projected features after average pooling and the reset weights are generated based on the projected features of other modalities, the projected features are then modulated and weighted to obtain cross-modal fusion features.

[0013] The spatiotemporal context-aware anomaly detection module is as follows: By constructing a sensor graph structure, a multi-head graph attention network is used to aggregate spatial neighbor information, and gated temporal convolution is applied along the time dimension to extract multi-scale features, outputting spatiotemporal context features; By compressing the spatiotemporal context features through multiple parallel pooling layers of different scales, the short-term fluctuations and long-term trends in water conservancy monitoring data are captured, and a fixed-length multi-scale representation vector is output. Adaptive propagation weights are calculated based on multi-scale representation vectors and spatiotemporal context features. Then, the current features are fused with the historical abnormal states of neighbors by combining propagation delay. Finally, the abnormal probability of each sensor at each time point is output through the Sigmoid activation function.

[0014] The loss function is calculated as follows: A composite loss function combining adaptive spatiotemporal propagation regularization and multimodal feature consistency constraints is adopted; The weighted cross-entropy loss is calculated based on the true anomaly labels and anomaly probability outputs, where the weights are adaptively adjusted according to the anomaly propagation strength. Based on the complete alignment tensor and modal projection features, the reconstruction loss and mutual information regularization are calculated.

[0015] The training of machine learning models is as follows: The backpropagation algorithm is used to iteratively optimize the model parameters. In each round of training, the collected multimodal data is input, and the spatiotemporal alignment and missing data compensation, cross-modal feature collaborative enhancement, heterogeneous feature projection and gating fusion, spatiotemporal context-aware anomaly recognition modules are executed in sequence to calculate the composite loss function. Update parameters using the AdamW optimizer, and set the initial learning rate and batch size; Set up a training termination mechanism to end training and output the trained model when the training termination condition is met.

[0016] S4 is as follows: Newly acquired multimodal water conservancy monitoring data is input into the trained model. First, a unified spatiotemporal grid tensor is constructed and missing data is compensated for, outputting a complete aligned tensor. Then, an enhanced feature tensor is generated by the cross-modal feature collaborative enhancement module, and cross-modal fused features are obtained through a heterogeneous feature projection and gating fusion module. Finally, the spatiotemporal context awareness module calculates the anomaly probability of each sensor at each time step, sets a threshold of 0.5 for binarization, and if the anomaly probability is... Time of the first If the probability of an anomaly of a sensor is greater than or equal to 0.5, then an anomaly alarm is triggered on the s-th sensor at time t, and the anomaly type is output.

[0017] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. The above technical solutions have the following advantages or beneficial effects: This invention discloses a method for identifying abnormal states in water conservancy monitoring data based on multimodal learning. It addresses the distortion of intermodal correlations caused by frequency inconsistencies and data gaps in conventional data processing methods by using spatiotemporal alignment of multimodal data and low-rank tensor completion algorithms, thereby preserving the potential correlations of the data and improving the accuracy of abnormal state identification. Tensor ring decomposition and mutual information regularization are employed to synergistically enhance multimodal features, and multi-head spatiotemporal graph attention and temporal convolutional gating networks are used to capture complex spatiotemporal patterns and spatial dependencies. These methods are rarely used in existing technologies and can effectively improve the accuracy of multimodal data identification. Based on the robustness and feature representation capabilities of the fusion, this study enhances the modeling ability for the dynamic propagation characteristics of abnormal states in water conservancy systems by utilizing an adaptive anomaly propagation mechanism combined with the spatial topology of sensor networks. This spatiotemporal context-aware classification method overcomes the limitations of conventional classification methods in failing to consider temporal dynamics and spatial dependencies. By combining a composite loss function with spatiotemporal propagation regularization and multimodal feature consistency constraints, the study optimizes the accuracy of anomaly classification and feature alignment capabilities, avoiding the information loss problem in the multimodal data fusion process in conventional methods, thereby improving the model's generalization ability and practical application effects. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0019] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0020] Figure 2 This is a heatmap of the anomaly probability of the method of the present invention.

[0021] Figure 3 This is a heatmap of anomaly probabilities using the conventional LSTM method.

[0022] Figure 4 This is a performance distribution feature map of different anomaly detection methods. Detailed Implementation

[0023] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0024] Example 1 like Figure 1 As shown, a method for identifying abnormal states in water conservancy monitoring data based on multimodal learning includes the following steps: S1. Deploy a heterogeneous sensor network on water conservancy facilities to collect multimodal raw monitoring data, including time-series modal data and discrete modal data, and obtain text modal data from work logs. Then, label the multimodal raw monitoring data with abnormal states to form a training dataset. S2. Construct a unified time grid tensor to align the original multimodal monitoring data, and use a low-rank tensor completion algorithm that integrates modal mutual information constraints to obtain a complete aligned tensor; S3. Construct and train a machine learning model. The model includes a cross-modal feature collaborative enhancement module, a heterogeneous feature projection and gating fusion module, and a spatiotemporal context-aware anomaly recognition module. Input the complete alignment tensor into the machine learning model, and pass it through each module in turn to obtain the anomaly probability value. Then calculate the model's loss function and train the model based on the loss function. S4. Collect new data, preprocess it, and input it into the trained model to obtain the anomaly probability value. Compare it with the set threshold to determine the anomaly type.

[0025] In a specific implementation, S1 is as follows: Multimodal raw monitoring data is collected through a heterogeneous sensor network deployed in water conservancy facilities (such as reservoirs, rivers, and pumping stations). Time-series modal data is continuously generated by devices such as water level gauges, flow meters, and pressure sensors, recording floating-point values ​​at a frequency of minutes. Discrete modal data originates from equipment status monitoring units, such as gate opening controllers and pump operation modules, recording equipment status codes in an event-triggered manner. Text modal data is obtained through system logs entered by maintenance personnel. Data labeling is completed based on water conservancy expert rules and historical fault records: each sensor at each moment is manually labeled with an abnormal status mark, with 0 indicating normal and 1 indicating abnormal. The labeling categories include three types: equipment failure (such as sudden changes in sensor readings), environmental anomalies (such as water levels exceeding warning values), and text description faults (such as the keyword "leakage" in the logs).

[0026] In a specific implementation, S2 is as follows: Water conservancy monitoring involves time-series sensor data, discrete equipment status codes, and text maintenance logs, which suffer from inconsistent collection frequencies and random missing data. Conventional processing methods use interpolation techniques to fill in missing data, but this destroys the potential correlation between different modal data, leading to distorted fused features and an inability to accurately reflect the true state. This invention aligns multimodal data by constructing a unified spatiotemporal grid tensor and employs a low-rank tensor completion algorithm constrained by fused modal mutual information to simultaneously solve the problems of missing data and spatiotemporal misalignment. The specific steps are as follows: 1) Construct a unified spatiotemporal grid tensor By mapping non-uniformly sampled multimodal data onto a unified spatiotemporal grid to construct a unified spatiotemporal grid tensor, where each element stores the original observation or missing value based on its index position, the problem of inconsistent multimodal data acquisition frequencies is resolved. This is represented as: In the formula, It represents a unified spatiotemporal grid tensor, which solves the problem of inconsistent sampling frequencies by mapping non-uniformly sampled multimodal data onto a unified spatiotemporal grid; The tensor represents the tensor in the first place. Time of the first The first sensor Element values ​​at modalities; Represents the original observations, including floating-point values ​​for time-series data, one-hot encoded vectors for discrete data, or BERT embedding vectors for text data; Indicates a missing identifier, used to mark the first... Time of the first The first sensor The location of missing data at the modality; Indicates the first Time of the first The first sensor Index position at the modality; This represents a time index, with a value range of [value range missing]. ; For the maximum time step, e.g., Corresponding to minute-level data over 30 days; This represents the sensor index, with a value range of [value range missing]. ; This represents the total number of sensors; This represents the modal index, with a value range of 100. Separately Corresponding to timing modes, Corresponding to discrete modes, Corresponding text modality.

[0027] In practical implementation, timing data is directly read from the sensor's floating-point values, such as the readings of a water level sensor. The unit is meters. Discrete data converts device status codes into one-hot codes, such as encoding "normal" as... The text data is embedded into the maintenance logs using a pre-trained BERT model. For example, the log "abnormal pump vibration" is converted into a 768-dimensional vector. .

[0028] 2) Tensor completion for modal mutual information constraints By optimizing the objective function that combines low-rank constraints and modal mutual information regularization, and iteratively solving it using the alternating direction multiplier method, a complete tensor is obtained. This compensates for missing data while maintaining statistical dependencies between modes, ensuring both low-rank data and modal correlation. This is expressed as: In the formula, The complete tensor is obtained by optimizing the objective function. This optimization process, based on the alternating direction multiplier method, applies to a unified spatiotemporal grid tensor. Perform a complete estimate to ensure low-rank data and intermodal statistical dependencies, and mutual information constraints to ensure that the completed data maintains intermodal statistical dependencies; Indicates the first The auxiliary matrix of the 3D modal expansion introduces low-rank constraints into the optimization objective, simplifying the nuclear norm minimization problem; This represents the modal dimension index, with a value range of [value range missing]. ; This indicates the completion of the complete tensor. and the Auxiliary matrix for 3D modal expansion The minimization operation is to optimize the objective function by minimizing the two variables. This represents the projection operator, which retains only the observation set. For elements in the array, the missing positions are set to zero; Denotes the Frobenius norm; Represents the nuclear norm; This represents the low-rank constraint weight, controlling the low-rank strength of different modalities. It is set through cross-validation or based on the data missing rate; for example, the low-rank constraint weight is increased when the missing rate is high. To enhance low-rank properties, the empirical value range is: ; Represents the mutual information regularization strength, balancing low-rank constraints and mutual information regularization, and sets... ; This represents the mutual information function, used to quantify intermodal correlations. The calculation method for the item is expressed as follows: ; This indicates that the complete tensor is in the 1st... A two-dimensional slice on the modality, including data from all time and sensor dimensions, with dimensions of . ; This indicates that the complete tensor is in the 1st... A two-dimensional slice on the modality, including data from all time and sensor dimensions, with dimensions of . ; The modal index represents the mutual information, which is used to calculate the mutual information. Its value range is [value range missing]. Separately Corresponding to timing modes, Corresponding to discrete modes, Corresponding text modality; Indicates difference from Mutual information modal index; express and The first joint singular value decomposition One singular value; This represents a logarithmic function, with the default base being the natural constant. express The A marginal distribution feature function is used to calculate the modal feature distribution; express The A marginal distribution feature function is used to calculate the modal feature distribution; This represents the singular value index, with a value range of 1000. ; This indicates the number of singular values ​​to be truncated, based on a preset cumulative contribution rate of singular values, for example, retaining the first... The singular values ​​make the cumulative contribution rate Experience value .

[0029] It should be noted that the first Auxiliary matrix for 3D modal expansion These are introduced auxiliary variables used to simplify optimization with low-rank constraints. In the alternating direction multiplier method, they are obtained through tensor expansion, i.e., by completing the tensor. Expand along the nth dimension into a matrix. These correspond to time, sensor, and modal dimensions, respectively. During the optimization process Alternating updates; observation set It is the set of indices of known data points, that is, the set of non-missing locations in the original data. When the projection operator is applied to the tensor, only the observation set is retained. Set the element corresponding to the index in the middle, and set all other positions to zero.

[0030] In the specific implementation process, The marginal distribution characteristic functions The calculation is approximated using a neural network. Specifically, a fully connected layer maps the input features to the feature space, and then extracts the k-th feature component, which is the k-th column of the output matrix of the fully connected layer. Similarly, The marginal distribution characteristic functions The implementation method is the same.

[0031] Output dimension is Fully aligned tensor It is the complete tensor after tensor completion and feature unification, where, Indicates the number of sensors. To represent a unified feature dimension, after tensor completion, the features of each modality are projected to unify the original modal feature dimensions to a unified dimension. Specifically, through the completion of the tensor Each modal slice is obtained by applying a linear transformation, which projects the features of each modality from the original dimensions to 64 dimensions.

[0032] S3 specifically involves: building and training a machine learning model; In a specific implementation, the cross-modal feature collaborative enhancement module is as follows: Aligned multimodal data suffers from differences in feature scales and loss of implicit associations. Conventional feature cross-multiplication methods typically only explicitly model second-order interactions, making it difficult to capture higher-order nonlinear feature relationships, resulting in insufficient feature representation capabilities. This invention utilizes tensor ring decomposition to extract spatiotemporal modal co-factors and concatenates them with local features. Then, adversarial training with mutual information regularization is used to generate enhanced features, thereby improving the robustness of feature representation. The specific steps are as follows: 1) Co-factorization of tensor rings The complete aligned tensor is decomposed into a time factor matrix, a spatial factor matrix, and a modal factor tensor using a tensor ring decomposition algorithm. Cooperative features are then reconstructed through outer product operations and concatenated with local features extracted by 3D convolution. This captures temporal feature patterns, sensor spatial distribution features, and intermodal interactions, forming an enhanced cooperative feature tensor, represented as: In the formula, Representing the time factor matrix The Column vectors are used to capture feature patterns in the time dimension; Represents the space factor matrix The Column vectors are used to capture the spatial distribution characteristics of the sensors; Represents the modality factor tensor The Row vectors are used to model the interaction relationships between modalities; The ring rank of the tensor representing the time dimension controls the complexity of the time factor matrix; the default setting is... ; The ring rank of the tensor represents the spatial dimension, controlling the complexity of the spatial factor matrix; the default setting is... ; Represents a time-rank index. ; Represents a spatial rank index. ; Indicates the outer product operation; Represents the collaborative feature tensor with dimension . It is a synergistic feature that integrates tensor ring decomposition factor and 3D convolution local features; This represents a 3D convolution kernel, which is a trainable parameter with dimension 1. This is used to extract local spatiotemporal features; Represents convolution operation This represents the bias term, which is a trainable parameter used to adjust the baseline of the convolution output; This represents the modified linear unit activation function; Represents the time factor matrix, with dimension 1. It is obtained from the complete aligned tensor through the tensor ring decomposition algorithm. The trainable parameter matrix obtained from the training has a dimension of It represents long-term dependence and cyclical patterns in the time dimension; Represents a spatial factor matrix with dimension . It is obtained from the complete aligned tensor through the tensor ring decomposition algorithm. The trainable parameter matrix obtained from the training has a dimension of This characterizes the geographical correlation in the spatial distribution of sensors; This indicates a splicing operation.

[0033] 2) Mutual information maximization and adversarial enhancement Enhanced features are generated by combining collaborative features with random noise through a generator. A discriminator distinguishes between real and enhanced features. During adversarial training, mutual information regularization is used to maximize the mutual information between real and enhanced features, thereby increasing data diversity and robustness while maintaining semantic consistency of features. This can be represented as follows: In the formula, Represents the augmented feature tensor, with dimension . Enhanced features are generated by introducing random noise into the generator, aiming to increase data diversity and robustness. This is represented by the computational method. ; The generator takes co-factorial features and noise as input and outputs enhanced features as output, with dimensions of... The three fully connected layers and Activation function composition; This represents a discriminator used to distinguish between real features and augmented features, consisting of dimensions... Two fully connected layers and Output layer composition; Represents a random noise vector with a mean of 1 / 2. The covariance matrix follows a normal distribution, which is the identity matrix. Indicates that it follows a certain distribution, for example express It follows a standard normal distribution; This indicates maximizing the parameters of the generator; This indicates that the parameters of the discriminator are minimized; This represents the expectation operation; Represents the mutual information weight coefficient, balancing adversarial loss and mutual information regularization, and sets... ; The mutual information function quantifies the dependency between two random variables; It is a natural constant.

[0034] In a specific implementation, the heterogeneous feature projection and gating fusion module is as follows: The enhanced features still retain modal heterogeneity. Conventional fully connected fusion methods ignore the nonlinear dependencies between feature channels, leading to insufficient information exchange and affecting the fusion effect. This invention maps heterogeneous features to a unified space through modality-specific projection and uses a dual-gating mechanism to dynamically aggregate cross-modal information, achieving efficient fusion of cross-modal information. The specific steps are as follows: 1) Modality-specific projection By constructing an independent projection network for each modality, the enhanced features are concatenated with the spatiotemporal factors and then processed through layer normalization and Gaussian error linear unit activation functions to obtain the projected features of each modality. This maps heterogeneous features to a unified feature space, eliminates scale differences between modalities, and enhances spatiotemporal consistency, as expressed below: In the formula, Indicates the first The projection features of the modality, with dimension 1 By fusing enhanced features and spatiotemporal factors, intermodal scale differences are eliminated, and spatiotemporal consistency is improved. This represents the modal index, with a value range of 100. Separately Corresponding to timing modes, Corresponding to discrete modes, Corresponding text modality; Represents the augmented feature tensor The Middle Modal slices, with dimensions of This is obtained through channel segmentation, ensuring that each modal branch processes heterogeneous features; It is the first Feature dimensions of a modality; This indicates concatenation along the feature dimension; Indicates the first The projection matrix of the modality is a trainable parameter with dimension 1. , used for feature space transformation; Presentation layer normalization operation; This represents the Gaussian error linear unit activation function.

[0035] 2) Gated cross-modal aggregation By calculating the gating weights and reset weights for each modality, where the gating weights are generated based on the projected features after average pooling and the reset weights are generated based on the projected features of other modalities, and then using these weights to modulate and weightedly sum the projected features, the information flow of each modality is dynamically controlled, thereby achieving cross-modal feature fusion, as expressed in: In the formula, Represents cross-modal fusion features, with dimensions of This characterizes the multimodal joint features after scale unification and dynamic weighting; Indicates the first The modal gating weights, used to control the amount of information passed through that modality, are calculated as follows: ; Indicates the first Modal reset weights, used to modulate interaction information from other modalities, are calculated as follows: ; This indicates the Sigmoid activation function, which compresses the output to... interval; Indicates the first The modality's gating weight matrix is ​​a trainable parameter with dimension 1. This is used to project the average pooled features into the gate space; This represents the average pooling operation, which aggregates features along the spatiotemporal dimension. The gating bias vector is a trainable parameter with dimension . This is used to adjust the threshold of the gated activation function; Indicates difference from The modal index, with a value range of Separately Corresponding to timing modes, Corresponding to discrete modes, Corresponding text modality; Indicates the first Projection characteristics of the modality; Indicates from the first Modal to the first The modal interaction weight matrix is ​​a trainable parameter with dimension 1. Implicit associations between modalities are learned through training; This represents element-wise multiplication; Tensor product operation In a specific implementation, the spatiotemporal context-aware anomaly detection module is as follows: Conventional classification methods, such as fully connected networks, may fail to capture the complex spatiotemporal patterns in water conservancy monitoring data and neglect the spatial topological relationships of sensor networks and the dynamic propagation characteristics of transient anomalies. This invention employs a classification network based on multi-head spatiotemporal graph attention and temporal convolutional gating. It models sensor spatial dependencies through graph structure, captures multi-scale temporal patterns, and combines an adaptive anomaly propagation mechanism for dynamic classification. The specific steps are as follows: 1) Multi-head spatiotemporal graph attention encoding By constructing a sensor graph structure, a multi-head graph attention network is used to aggregate spatial neighbor information, and gated temporal convolution is applied along the time dimension to extract multi-scale features, outputting spatiotemporal context features, represented as: In the formula, Represents spatiotemporal context features, with dimensions of It enhances spatial dependence and temporal dynamics through the fusion of graph attention and temporal convolution. This indicates the number of attention heads in the graph; the default setting. ; This represents the graph attention head index, with a value range of 1. ; The normalized adjacency matrix is ​​constructed based on the spatial distance between sensors and is calculated as follows: ; It is an adjacency matrix. Adjacency matrix The Middle Line number The element of the column represents the first element. The sensor and the first The connection weights between sensors, when ,otherwise ; It is a degree matrix, a diagonal matrix, where the degree of each sensor is the sum of the corresponding rows in the adjacency matrix; Indicates the s-th sensor and the s-th sensor. The Euclidean distance between the sensors is calculated based on the geographic coordinates of the sensors; Indicates difference from The sensor index, with a value range of 100. ; This represents the distance scaling parameter, which has a default value. The effect of distance on connection weights is adjusted by using a Gaussian kernel function; This represents a distance threshold used to limit the connection of remote sensors; the default value is... The unit is meters; Indicates the first The projection matrix of each attention head is a trainable parameter with dimension . ; The graph feature dimension is set through cross-validation based on model complexity and task requirements; the default setting is... ; This represents gated temporal convolution, composed of one-dimensional convolution and sigmoid gating, and is calculated as follows: ; This indicates the first convolutional layer, which uses a one-dimensional convolutional kernel, and the default convolutional size is set to 5. This indicates the second convolutional layer, which uses a one-dimensional convolutional kernel, and the default convolutional size is set to 5.

[0036] 2) Multi-scale temporal pyramid pooling By compressing spatiotemporal context features through multiple parallel pooling layers of different scales, short-term fluctuations and long-term trends in water conservancy monitoring data are captured, and a fixed-length multi-scale representation vector is output, as follows: In the formula, Represents a multi-scale representation vector with dimension . It is a global feature aggregated through pyramid pooling. The dimension of the multi-scale representation vector; Represents the pooling scale set, defaulting to These correspond to short-term, medium-term, and long-term patterns. Representing spatiotemporal context features At pooling scale The following characteristics; For pooling scalar index; This represents the adaptive average pooling operation, which compresses the time dimension of the input features to a fixed length. ; Indicates the pooling scale The output time length is set to the default value. ; This indicates that all pooling results are concatenated along the feature dimension.

[0037] 3) Adaptive anomaly propagation classification Adaptive propagation weights are calculated based on multi-scale representation vectors and spatiotemporal context features. Then, the current features are fused with the historical anomaly states of neighbors by combining the propagation delay. Finally, the anomaly probability of each sensor at each time point is output through the Sigmoid activation function, as follows: In the formula, Indicates the first Time of the first The anomaly probability of each sensor, calculated through an adaptive propagation mechanism, is the output with a range of [value range missing]. scalar; Indicates the first Time of the first The probability of anomalies in each sensor; Indicates from the first The sensor to the first The anomaly propagation delay time of each sensor is calculated based on the Euclidean distance between the sensors and the water flow velocity. The calculation method is as follows: Simulates the propagation delay of hydraulic anomalies along the direction of water flow; This indicates the water flow velocity, which can be estimated from historical hydraulic data; the default setting is... , used to adjust the propagation speed; This represents the floor function; The anomaly classification weight vector represents trainable parameters with dimension 1. This is used to map spatiotemporal features to anomaly scores; Representing spatiotemporal context features In the Time of the first Feature vectors of each sensor; Indicates the first The set of neighboring sensors of each sensor, based on an adjacency matrix. Determined, that is, when hour, ; Indicates the first Time from the first The sensor to the first The anomaly propagation weights of each sensor are used to quantify the intensity of anomaly propagation, and are calculated as follows: ; The feature interaction parameter vector is a trainable parameter vector used to capture the interactions between features; for transpose; Representing spatiotemporal context features In the Time of the first Feature vectors of each sensor; Representing spatiotemporal context features In the Time of the first Feature vectors of each sensor; Indicates the first The sensor and the first The Euclidean distance between the sensors is calculated based on the geographic coordinates of the sensors, and the unit is meters; Different from and The sensor index, with a value range of 100. .

[0038] It should be noted that the first Time from the first The sensor to the first Anomaly propagation weights of individual sensors The calculation utilizes multi-scale representation vectors This enables it to adapt to global sequence patterns while incorporating feature vectors. and Capturing transient spatial dependencies and anomaly propagation delay time Designed based on the background of hydraulic physics, this model reflects the propagation characteristics of anomalies in water flow networks, enhancing its practicality in hydraulic monitoring scenarios. In its specific implementation, for... In the default case This indicates that there are no historical anomalies, and the final output is the [number]. Time of the first The probability of anomalies in each sensor It can be used for real-time anomaly alarms or subsequent analysis.

[0039] In a specific implementation, the loss function is as follows: Water conservancy monitoring data anomaly identification models need to simultaneously optimize anomaly classification accuracy and maintain the consistency of multimodal features. Conventional cross-entropy loss ignores the spatiotemporal dependence of anomaly propagation and multimodal feature alignment constraints, resulting in insufficient generalization ability of the model in complex water conservancy scenarios. This invention adopts a composite loss function combining adaptive spatiotemporal propagation regularization and multimodal feature consistency constraints. By combining anomaly propagation consistency loss and modality alignment loss, the model's ability to model the dynamic propagation of water conservancy anomalies and its robustness to multimodal feature fusion are enhanced. The specific steps are as follows: 1) Propagation of perceptual cross-entropy loss The weighted cross-entropy loss is calculated based on the true anomaly labels and anomaly probability outputs. The weights are adaptively adjusted according to the anomaly propagation intensity to emphasize anomalous samples along the propagation path and improve the model's sensitivity to the propagation of hydraulic anomalies. This is expressed as: , In the formula, This represents the propagation-perceptual cross-entropy loss, used to optimize anomaly classification accuracy and incorporate propagation information; This represents the propagation enhancement coefficient, used to control the impact of propagated information on the loss weights. A preferred setting is... ; Indicates the first Time of the first The true anomaly label of each sensor, with a value of 0 or 1; Indicates the first Time of the first The true anomaly label of each sensor has a value of 0 or 1.

[0040] It should be noted that, Based on real anomaly labels and propagation parameters, this approach enables the model to focus more on samples along the propagation path during training, enhancing its ability to model the dynamic propagation of water conservancy anomalies. In specific implementation, for... In the default case This indicates that there are no historical anomalies.

[0041] 2) Multimodal feature consistency loss Based on the complete alignment tensor and modal projection features, the reconstruction loss and mutual information regularization are calculated to ensure feature alignment and intermodal dependencies, thereby improving the robustness of multimodal feature fusion. This is expressed as: , In the formula, This represents the multimodal feature consistency loss, ensuring that multimodal features maintain data consistency and intermodal correlation during the fusion process; Represents a fully aligned tensor In the Time of the first The first sensor The value of the feature dimension; Represents cross-modal fusion features In the Time of the first The first sensor The value of the feature dimension; This represents the feature dimension index, with a value range of 100. ; Represents the L2 norm; Indicates the first Modal projection characteristics Indicates the first Projection characteristics of the modality; This represents the mutual information weighting coefficient, balancing the reconstruction loss and mutual information regularization; the default setting is... .

[0042] It should be noted that, Item representation reconstruction loss, forced cross-modal feature fusion Approximating the complete aligned tensor Reduce feature distortion and preserve the information integrity of the original data. Term representation mutual information regularization maximizes the projected features of different modalities. and Mutual information between modalities enhances statistical dependence between modalities, avoids modal isolation, and improves the robustness of feature fusion.

[0043] In a specific implementation, the model training is as follows: The model parameters are iteratively optimized using the backpropagation algorithm. In each training round, multimodal data is input, and the following steps are executed sequentially: S2 spatiotemporal alignment and missing data compensation, S3 cross-modal feature collaborative enhancement, heterogeneous feature projection and gating fusion, and spatiotemporal context-aware anomaly recognition module. The composite loss function defined in S304 (propagation-aware cross-entropy loss) is calculated. Consistency loss with multimodal features (Weighted sum). Parameters are updated using the AdamW optimizer, with an initial learning rate of 0.001 and a batch size of 32. The F1 score on the validation set is continuously monitored during training: if the F1 score does not improve after 10 consecutive validation rounds, an early stopping mechanism is activated to terminate training; the maximum number of iterations is set to 200, and the model parameters of the last iteration are saved when the maximum number of iterations or the early stopping condition is reached.

[0044] In a specific implementation, S5 is as follows: The real-time collected multimodal water conservancy monitoring data is input into the trained model.

[0045] First, a unified spatiotemporal grid tensor is constructed and missing data is compensated using S2, and a complete aligned tensor is output. Then, the cross-modal feature collaborative enhancement module of S3 generates an enhanced feature tensor, and then the cross-modal fused feature is obtained through the heterogeneous feature projection and gating fusion module; Finally, the spatiotemporal context awareness module calculates the anomaly probability of each sensor at each time step, sets a threshold of 0.5, and performs binarization for determination: if the... Time of the first The probability of anomalies in each sensor If the value is ≥0.5, then the abnormal alarm of the s-th sensor at time t will be triggered, and the abnormality type will be output.

[0046] The identification results are pushed to the water conservancy monitoring platform in real time to provide a basis for operation and maintenance decisions.

[0047] Example 2 like Figure 2 and Figure 3As shown, a heatmap analysis of spatiotemporal context-aware anomaly detection is performed to visually demonstrate the anomaly detection performance of the spatiotemporal context-aware module in a sensor network. The experiment simulates anomaly detection by 10 sensors over 24 hours, comparing the detection performance of the method of this invention with that of conventional long short-term memory networks. The horizontal axis of the heatmap represents the time axis in hours, the vertical axis represents different sensor numbers, and the color intensity indicates the anomaly probability. Figure 2 As shown, the heatmap of the method of this invention reveals the spatiotemporal distribution pattern of anomalies in the sensor network. Certain sensors exhibit significant anomaly clustering within specific time periods; for example, sensor 3 shows high anomaly probability areas in the morning and sensor 7 shows high anomaly probability areas in the afternoon, reflecting the temporal regularity of equipment failure anomalies. Simultaneously, multiple adjacent sensors simultaneously generate anomalous signals at night, demonstrating the diffusion characteristics of environmental anomalies. In contrast, as... Figure 3 As shown, the heatmaps of conventional long short-term memory network methods appear more cluttered, with anomalous signals distributed in a scattered manner, failing to clearly demonstrate the spatiotemporal propagation patterns of anomalies. Furthermore, the probability of detecting anomalies is generally low, indicating that they are ineffective in capturing complex spatiotemporal dependencies.

[0048] Example 3 like Figure 4 As shown, this paper analyzes the performance distribution characteristics of different anomaly detection methods and compares the performance differences between the multimodal anomaly identification method proposed in this invention and mainstream anomaly detection algorithms in water conservancy monitoring scenarios. Experiments compare traditional techniques such as Isolation Forest, Local Anomaly Factor Algorithm, Autoencoder, and Long Short-Term Memory Network. These methods represent different technical routes, including isolation-based anomaly detection, density-based anomaly detection, reconstruction-based anomaly detection, and time-series modeling-based anomaly detection, respectively. The experiments use the same multimodal water conservancy monitoring dataset, and each method is tested multiple times to obtain its performance distribution. The evaluation metric is the F1 score of anomaly detection, which combines precision and recall to comprehensively reflect the model's detection capability. The kernel density distribution plot shows significant differences in the F1 score distribution of different methods. The distribution of traditional methods is generally skewed to the left and relatively dispersed, indicating unstable performance and low average scores. Isolation Forest and Local Anomaly Factor Algorithm have a wider distribution range and lower peak values, indicating poor adaptability of these unsupervised methods in complex water conservancy monitoring scenarios. The distribution of Autoencoder and Long Short-Term Memory Network is relatively concentrated, showing improved performance, but still not reaching ideal levels. The distribution of the method in this invention is clearly located on the far right, with a compact distribution range and the highest peak value, indicating that it not only has the best average performance but also the best stability. Experimental results show that the combined effect of cross-modal feature synergistic enhancement and spatiotemporal context awareness mechanism in this invention can fully utilize the complementary information in multimodal data to accurately identify various hydraulic anomalies.

[0049] Although the specific embodiments of the invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the invention. Based on the technical solutions of the invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the invention.

Claims

1. A method for identifying abnormal states in water conservancy monitoring data based on multimodal learning, characterized in that, Includes the following steps: S1. Deploy a heterogeneous sensor network on water conservancy facilities to collect multimodal raw monitoring data, including time-series modal data and discrete modal data, and obtain text modal data from work logs. Then, label the multimodal raw monitoring data with abnormal states to form a training dataset. S2. Construct a unified time grid tensor to align the original multimodal monitoring data, and use a low-rank tensor completion algorithm that integrates modal mutual information constraints to obtain a complete aligned tensor; S3. Construct and train a machine learning model. The model includes a cross-modal feature collaborative enhancement module, a heterogeneous feature projection and gating fusion module, and a spatiotemporal context-aware anomaly recognition module. Input the complete alignment tensor into the machine learning model, and pass it through each module in turn to obtain the anomaly probability value. Then calculate the model's loss function and train the model based on the loss function. S4. Collect new data, preprocess it, and input it into the trained model to obtain the anomaly probability value. Compare it with the set threshold to determine the anomaly type.

2. The method for identifying abnormal states in water conservancy monitoring data based on multimodal learning according to claim 1, characterized in that, S1 is as follows: Time-series modal data includes data continuously collected by water level gauges, flow meters, and pressure sensors, with floating-point values ​​recorded at a frequency of minutes. Discrete modal data is collected through the gate opening controller and pump operation module, and the equipment status code is recorded in an event-triggered manner. Text modal data is obtained from system logs entered by maintenance personnel; Data labeling is done manually based on water conservancy expert rules and historical fault records, labeling each sensor with an abnormal status at each moment. 0 indicates normal and 1 indicates abnormal. Abnormal categories include equipment failure, environmental abnormality, and text description of fault.

3. The method for identifying abnormal states in water conservancy monitoring data based on multimodal learning according to claim 1, characterized in that, S2 is as follows: S2.1 Construct a unified spatiotemporal grid tensor by mapping non-uniformly sampled multimodal data onto a unified spatiotemporal grid, where each element stores the original observation or missing label according to its index position; S2.

2. The complete tensor is obtained by optimizing the objective function that combines low-rank constraints and modal mutual information regularization, and by iteratively solving the problem using the alternating direction multiplier method. Specifically, the complete tensor is obtained by solving the objective function through optimization. Based on the iterative optimization process of the alternating direction multiplier method, the complete tensor of the unified spatiotemporal grid is estimated. The final output dimension is A fully aligned tensor is a complete tensor after tensor completion and feature unification. Indicates the maximum time step. Indicates the number of sensors. This represents a unified feature dimension.

4. The method for identifying abnormal states in water conservancy monitoring data based on multimodal learning according to claim 1, characterized in that, S3 is as follows: A machine learning model is constructed and trained. The fully aligned tensor is input into the model and first passes through a cross-modal feature collaborative enhancement module. This module uses the tensor ring decomposition algorithm to extract the spatiotemporal modal collaborative factors from the completed tensor and concatenates them with local features. Then, it generates an enhanced feature tensor through adversarial training with mutual information regularization. The enhanced feature tensor then passes through a heterogeneous feature projection and gating fusion module. The module maps heterogeneous features to a unified space through modality-specific projection and uses a dual-gating mechanism to dynamically aggregate cross-modal information to generate cross-modal fused features. Finally, the cross-modal fusion features are processed by a spatiotemporal context-aware anomaly detection module. This module uses a classification network based on multi-head spatiotemporal graph attention and temporal convolutional gating. It models sensor spatial dependence through graph structure, captures multi-scale temporal patterns, and combines an adaptive anomaly propagation mechanism for dynamic classification to generate anomaly probability values.

5. The method for identifying abnormal states in water conservancy monitoring data based on multimodal learning according to claim 4, characterized in that, The cross-modal feature collaborative enhancement module is as follows: The complete aligned tensor is decomposed into time factor matrix, spatial factor matrix and modality factor tensor by tensor ring decomposition algorithm, and the collaborative features are reconstructed by outer product operation. Then, it is concatenated with the local features extracted by 3D convolution to form an enhanced collaborative feature tensor. The generator combines collaborative features with random noise to generate enhanced features. The discriminator distinguishes between real features and enhanced features. During adversarial training, mutual information regularization is used to maximize the mutual information between real features and enhanced features, resulting in an enhanced feature tensor.

6. The method for identifying abnormal states in water conservancy monitoring data based on multimodal learning according to claim 4, Its key feature is that the heterogeneous feature projection and gating fusion module is specifically as follows: By constructing an independent projection network for each modality, the enhanced features are concatenated with the spatiotemporal factors and then processed by layer normalization and Gaussian error linear unit activation function to obtain the projection features of each modality. By calculating the gating weights and reset weights for each modality, where the gating weights are generated based on the projected features after average pooling and the reset weights are generated based on the projected features of other modalities, the projected features are then modulated and weighted to obtain cross-modal fusion features.

7. The method for identifying abnormal states in water conservancy monitoring data based on multimodal learning according to claim 4, characterized in that, The spatiotemporal context-aware anomaly detection module is as follows: By constructing a sensor graph structure, a multi-head graph attention network is used to aggregate spatial neighbor information, and gated temporal convolution is applied along the time dimension to extract multi-scale features, outputting spatiotemporal context features; By compressing the spatiotemporal context features through multiple parallel pooling layers of different scales, the short-term fluctuations and long-term trends in water conservancy monitoring data are captured, and a fixed-length multi-scale representation vector is output. Adaptive propagation weights are calculated based on multi-scale representation vectors and spatiotemporal context features. Then, the current features are fused with the historical abnormal states of neighbors by combining propagation delay. Finally, the abnormal probability of each sensor at each time point is output through the Sigmoid activation function.

8. The method for identifying abnormal states in water conservancy monitoring data based on multimodal learning according to claim 4, characterized in that, The loss function is calculated as follows: A composite loss function combining adaptive spatiotemporal propagation regularization and multimodal feature consistency constraints is adopted; The weighted cross-entropy loss is calculated based on the true anomaly labels and anomaly probability outputs, where the weights are adaptively adjusted according to the anomaly propagation strength. Based on the complete alignment tensor and modal projection features, the reconstruction loss and mutual information regularization are calculated.

9. The method for identifying abnormal states in water conservancy monitoring data based on multimodal learning according to claim 4, characterized in that, The training of machine learning models is as follows: The backpropagation algorithm is used to iteratively optimize the model parameters. In each round of training, the collected multimodal data is input, and the spatiotemporal alignment and missing data compensation, cross-modal feature collaborative enhancement, heterogeneous feature projection and gating fusion, spatiotemporal context-aware anomaly recognition modules are executed in sequence to calculate the composite loss function. Update parameters using the AdamW optimizer, and set the initial learning rate and batch size; Set up a training termination mechanism to end training and output the trained model when the training termination condition is met.

10. The method for identifying abnormal states in water conservancy monitoring data based on multimodal learning according to claim 5, characterized in that, S4 is as follows: The newly collected multimodal water conservancy monitoring data is input into the trained model. First, a unified spatiotemporal grid tensor is constructed and missing data is compensated, and a complete aligned tensor is output. Then, the cross-modal feature collaborative enhancement module generates an enhanced feature tensor, and the cross-modal fused feature is obtained through the heterogeneous feature projection and gating fusion module; Finally, the spatiotemporal context awareness module calculates the anomaly probability of each sensor at each time step, sets a threshold of 0.5 for binarization judgment, and if the... Time of the first If the probability of an anomaly of a sensor is greater than or equal to 0.5, then an anomaly alarm is triggered on the s-th sensor at time t, and the anomaly type is output.

Images