Water conservancy comprehensive monitoring method and system based on multi-modal data fusion

Abstract

The invention relates to a water conservancy comprehensive monitoring method and system based on multi-modal data fusion, and the method comprises the following steps: collecting water conservancy comprehensive data containing remote sensing images, sensor readings, and meteorological and hydrological historical records, and carrying out the normalization and noise reduction processing; and extracting features from the treated data, and fusing to form water conservancy comprehensive features. And the feature dimension is further reduced through the encoder. And inputting the dimension-reduced water conservancy comprehensive features and the preliminarily processed data into a support vector machine (SVM), and assisting in executing anomaly detection and classification by adopting an isolated forest algorithm. And outputting a classification result to realize comprehensive monitoring of the water conservancy system. According to the method, multi-source data are fused, the monitoring precision is improved, misjudgment caused by dependence on a single data source is effectively prevented, and comprehensiveness and accuracy of water conservancy monitoring are ensured.

Description

Technical Field

[0001] The present invention relates to a water conservancy comprehensive monitoring method and system based on multi-modal data fusion, belonging to the technical field of water conservancy projects. Background Art

[0002] With the development of water conservancy engineering technology, the importance of water conservancy monitoring systems in aspects such as water resource management, flood control and drought relief, and water environment protection has become increasingly prominent. Traditional water conservancy monitoring methods mainly rely on single data sources, such as manual monitoring, fixed-site monitoring, or simple sensor networks. These methods can provide some basic hydrological information to a certain extent, but there are many deficiencies and defects. Traditional methods usually only rely on one or a few data sources, such as manual carrying of portable instruments for on-site monitoring or continuous monitoring at fixed sites. This method cannot comprehensively reflect the complex state of the water conservancy system because different modalities of data (such as remote sensing images, sensor data, meteorological data, hydrological historical data) each provide different information, and it is difficult for a single data source to integrate this information. Traditional monitoring methods rely more on manual operations, which are time-consuming and laborious, and the analysis results lag far behind the actual water quality changes. For example, manual monitoring requires staff to collect data on-site regularly, which is not only inefficient but also has poor data timeliness and is difficult to reflect the dynamic changes of the water conservancy system in real time. Traditional methods are usually relatively simple in data processing and lack effective data fusion and analysis means. For example, the data formats, scales, sampling rates, etc. of different modalities may be different, and it is difficult to directly fuse them. Moreover, the data of each modality may contain noise, and direct fusion may lead to noise superposition, affecting the performance of the model.

[0003] The patent document with the patent number "CN117236565A" discloses a method for intelligent management of the water environment in a basin. The problems of this method are as follows: The data sources are mainly concentrated on water quality parameters, meteorological conditions, water flow velocity, and image data, lacking the full utilization of remote sensing images and hydrological historical data. In terms of data fusion, it mainly conducts simple weighted fusion through image analysis and a preliminary water environment quality index, lacking in-depth integration of multi-modal data. In terms of anomaly detection, it mainly relies on isolation forests and simple image analysis, and has limited ability to identify complex, multi-factor-induced anomalies (such as sudden pollution incidents). Summary of the Invention

[0004] In order to solve the problems existing in the above-mentioned prior art, the present invention proposes a water conservancy comprehensive monitoring method and system based on multi-modal data fusion.

[0005] The technical solution of the present invention is as follows:

[0006] On the one hand, the present invention provides a water conservancy comprehensive monitoring method based on multi-modal data fusion, including the following steps:

[0007] Collect comprehensive water conservancy data, and perform normalization and denoising processing on the comprehensive water conservancy data, where the comprehensive water conservancy data includes remote sensing image data, sensor data, meteorological data, and hydrological historical data;

[0008] Extract the features of the comprehensive water conservancy data after normalization and denoising processing, and fuse the features to obtain comprehensive water conservancy features;

[0009] Use an encoder to reduce the dimension of the comprehensive water conservancy features;

[0010] Input the reduced comprehensive water conservancy features and the comprehensive water conservancy data after normalization and denoising processing into a support vector machine (SVM), and combine with an isolation forest to perform anomaly detection and classification results of the comprehensive water conservancy data;

[0011] Output the classification results to achieve comprehensive water conservancy monitoring.

[0012] As a preferred implementation manner, the normalization and denoising processing method is:

[0013]

[0014] where X new represents the comprehensive water conservancy data after normalization and denoising processing, X old represents the comprehensive water conservancy data before normalization and denoising processing, min() represents the minimum value function, max() represents the maximum value function, WT() represents the wavelet transform processing function, φ represents the wavelet basis function, and j represents the preset number of wavelet decomposition layers.

[0015] As a preferred implementation manner, the feature extraction method is:

[0016]

[0017] where X new (t) represents the remote sensing image data, sensor data, meteorological data, or hydrological historical data after normalization and denoising processing at time t, N represents the preset maximum number, β represents the weight of the preset autoregressive model, AR() represents the autoregressive model prediction function, represents the complex exponential function, F(k) represents the eigenvalue of X new (t) at frequency k, k represents the frequency index, q represents the imaginary unit, and t represents the time index.

[0018] As a preferred implementation manner, the comprehensive water conservancy feature fusion method is:

[0019]

[0020] where F fuseDenote the comprehensive water conservancy features, M denote the preset number of modes, i denote the quantity index, Q i Denote the query matrix of F(k), Denote the transpose of the key matrix of F(k), d k Denote the number of columns of the key matrix, Softmax() denote the Softmax function, γ denote the preset weights of the convolutional neural network, V i Denote the value matrix of F(k), CNN() denote the convolutional operation function.

[0021] As a preferred embodiment, the method for reducing the dimension of the comprehensive water conservancy features is as follows:

[0022] F low = ReLU(W e F fuse , + b e );

[0023] Among them, F low Denote the reduced comprehensive water conservancy features, b e Denote the encoder bias vector, W e Denote the encoder weight matrix, ReLU() denote the dimension reduction function.

[0024] As a preferred embodiment, the method for anomaly detection and classification is as follows:

[0025]

[0026] Among them, y denote the classification result, a i Denote the Lagrange multiplier of the support vector machine, y i Denote the preset label of the comprehensive water conservancy data after normalization and denoising, X new (i) denote the i-th comprehensive water conservancy data after normalization and denoising, K() denote the kernel function, γ denote the preset anomaly detection weight, IF() denote the Isolation Forest function, sgn() denote the sign function.

[0027] On the other hand, the present invention also provides a comprehensive water conservancy monitoring system based on multi-modal data fusion, including:

[0028] Preprocessing module: Collect comprehensive water conservancy data, and perform normalization and denoising processing on the comprehensive water conservancy data, where the comprehensive water conservancy data includes remote sensing image data, sensor data, meteorological data, and hydrological historical data;

[0029] Multi-modal data fusion module: Extract the features of the comprehensive water conservancy data after normalization and denoising, and fuse the features to obtain comprehensive water conservancy features;

[0030] Dimensionality reduction module: Use an encoder to reduce the dimensionality of the comprehensive water conservancy features;

[0031] Monitoring module: Input the reduced comprehensive water conservancy features and the normalized and denoised comprehensive water conservancy data into a support vector machine (SVM), and combine the Isolation Forest for anomaly detection and classification results of the comprehensive water conservancy data; Output the classification results to achieve comprehensive water conservancy monitoring.

[0032] The present invention has the following beneficial effects:

[0033] The present invention comprehensively collects various water conservancy-related data such as remote sensing image data, sensor data, meteorological data, and hydrological historical data. These data reflect the state and changes of the water conservancy system from different angles and levels. For example, remote sensing images can intuitively show macroscopic situations such as water body distribution and vegetation coverage, sensor data can accurately obtain specific values such as real-time water level and flow velocity, meteorological data helps to understand the impact of rainfall, temperature, etc. on water conservancy, and hydrological historical data provides comparison and reference for the current situation. By fusing these multi-modal data, the overall situation of the water conservancy system can be grasped more comprehensively and accurately, thereby improving the accuracy of monitoring results and avoiding misjudgments caused by the limitations of a single data source. Use the method of combining a support vector machine (SVM) with an Isolation Forest to perform anomaly detection and classification on comprehensive water conservancy data. SVM is an effective classification algorithm that can find the optimal classification hyperplane in a high-dimensional space and accurately classify water conservancy data; the Isolation Forest is an efficient anomaly detection algorithm that can identify anomaly points in the data. The combination of the two enables the present invention to accurately identify the normal state and abnormal state in the water conservancy system, such as abnormal situations like floods, droughts, and water quality pollution, issue early warnings in a timely manner, provide a scientific basis for the water conservancy management department to take corresponding countermeasures, and effectively improve the accuracy and reliability of water conservancy monitoring. Extract the features of the normalized and denoised comprehensive water conservancy data through methods such as autoregressive models, and then use an encoder to reduce the dimensionality of the comprehensive water conservancy features. The feature extraction process can extract key information from the original data to form more representative and distinguishable feature vectors; the dimensionality reduction operation further reduces the number and complexity of features, reducing the problem of dimensionality disaster. This not only speeds up the data processing speed but also improves the operation efficiency of the algorithm, enabling the monitoring system to analyze and process a large amount of water conservancy data more quickly, output monitoring results in a timely manner, and meet the requirements of real-time water conservancy monitoring. The output classification results can clearly reflect the current state of the water conservancy system, including normal operation, abnormal situations, and specific abnormal types, etc. These detailed monitoring results provide comprehensive and accurate information for the water conservancy management department, helping it better understand the operation status of the water conservancy system and timely discover potential problems and risks. Description of the Drawings

[0034] Figure 1 This is the flowchart of the method implementation of the present invention. Specific implementation manner

[0035] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

[0036] It should be understood that the step numbers used in the text are only for convenience of description and do not limit the execution order of the steps.

[0037] It should be understood that the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0038] The terms "comprising" and "including" indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their combinations.

[0039] The term " / or" refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0040] Embodiment 1:

[0041] Refer to Figure 1 , a water conservancy comprehensive monitoring method based on multi-modal data fusion, including the following steps:

[0042] Collect water conservancy comprehensive data, and perform normalization and denoising processing on the water conservancy comprehensive data. The water conservancy comprehensive data includes remote sensing image data (including water body distribution images, vegetation coverage images, land use images, etc., which can be obtained by methods such as satellite remote sensing, UAV remote sensing, and high-resolution cameras), sensor data (including real-time water level data, flow velocity data, water quality data, soil moisture data, etc., which can be collected by deploying sensors), meteorological data (real-time rainfall data, temperature data, wind speed data, relative humidity data, etc., which can be obtained through meteorological bureaus, meteorological satellites, weather stations, etc.), and hydrological historical data (historical water level records, historical flow records, historical precipitation records, etc., which can be obtained from the historical database of the water conservancy department);

[0043] Extract the features of the integrated water conservancy data after normalization and denoising, and fuse the above to obtain the integrated water conservancy features;

[0044] Use an encoder to reduce the dimension of the integrated water conservancy features;

[0045] Input the reduced integrated water conservancy features and the integrated water conservancy data after normalization and denoising into the support vector machine SVM, and combine the isolation forest to obtain the anomaly detection and classification results of the integrated water conservancy data;

[0046] Output the classification results to achieve integrated water conservancy monitoring.

[0047] As a preferred embodiment, the normalization and denoising method is:

[0048]

[0049] Among them, X new represents the integrated water conservancy data after normalization and denoising, X old represents the integrated water conservancy data before normalization and denoising, min() represents the minimum value function, max() represents the maximum value function, WT() represents the wavelet transform processing function, φ represents the wavelet basis function, and j represents the preset number of wavelet decomposition layers.

[0050] As a preferred embodiment, the feature extraction method is:

[0051]

[0052] Among them, X new (t) represents the remote sensing image data, sensor data, meteorological data, or hydrological historical data at time t after normalization and denoising, N represents the preset maximum number (preset in advance according to the total number of data), β represents the weight of the preset autoregressive model, AR() represents the autoregressive model prediction function, represents the complex exponential function, 2π is a periodic constant, representing a complete period (360 degrees or 2π radians). Through the 2π Fourier transform, the time-domain signal can be decomposed into sine and cosine components of different frequencies, and F(k) represents the eigenvalue of X new (t) at frequency k, k represents the frequency index, q represents the imaginary unit, which is used to convert the time-domain signal into the frequency-domain signal and q 2 =-1, and t represents the time index.

[0053] As a preferred embodiment, the method for fusing the integrated water conservancy features is:

[0054]

[0055] Among them, Ffuse represents the comprehensive water conservancy feature, M represents the preset number of modes, i represents the quantity index, Q i represents the query matrix of F(k), represents the transpose of the key matrix of F(k), d k represents the number of columns of the key matrix, Softmax() represents the Softmax function, which converts the input vector into a probability distribution, ensuring that each weight value is between 0 and 1 and the sum of all weights is 1, γ represents the preset weight of the convolutional neural network, V i represents the value matrix of F(k), and CNN() represents the convolution operation function.

[0056] As a preferred embodiment, the method for reducing the dimension of the comprehensive water conservancy feature is:

[0057] F low = ReLU(W e F fuse , + b e );

[0058] wherein, F low represents the reduced comprehensive water conservancy feature, b e represents the encoder bias vector, W e represents the encoder weight matrix, ReLU() represents the dimension reduction function, and the encoder bias vector and the encoder weight matrix can be obtained through conventional pre-training methods, which are not limited here.

[0059] As a preferred embodiment, the method for anomaly detection and classification is:

[0060]

[0061] wherein, y represents the classification result, a i represents the Lagrange multiplier of the support vector machine, y i represents the preset label of the comprehensive water conservancy data after normalization and denoising, and the label is remote sensing image data, sensor data, meteorological data, or hydrological historical data, X new (i) represents the i-th comprehensive water conservancy data after normalization and denoising, K() represents the kernel function used to calculate the similarity between F low and X new (i), γ represents the preset anomaly detection weight, IF() represents the isolation forest function, and sgn() represents the sign function, which is used to convert the classification result into a classification label.

[0062] Embodiment 2:

[0063] A comprehensive water conservancy monitoring system based on multi-modal data fusion, comprising:

[0064] Preprocessing module: Collect comprehensive water conservancy data, and perform normalization and denoising processing on the comprehensive water conservancy data, where the comprehensive water conservancy data includes remote sensing image data, sensor data, meteorological data, and hydrological historical data;

[0065] Multimodal data fusion module: Extract the features of the comprehensive water conservancy data after normalization and denoising processing, and fuse the features to obtain comprehensive water conservancy features;

[0066] Dimensionality reduction module: Use an encoder to reduce the dimensionality of the comprehensive water conservancy features;

[0067] Monitoring module: Input the reduced comprehensive water conservancy features and the comprehensive water conservancy data after normalization and denoising processing into a support vector machine (SVM), and perform anomaly detection and classification results of the comprehensive water conservancy data in combination with an isolation forest; Output the classification results to achieve comprehensive water conservancy monitoring.

[0068] This system is used to implement the method in Embodiment 1, which will not be elaborated here.

[0069] In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent the situation where A exists alone, A and B exist simultaneously, or B exists alone. Where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one of the following" and its similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0070] Those of ordinary skill in the art can realize that the units and algorithm steps described in the embodiments disclosed herein can be implemented by a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0071] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be elaborated here.

[0072] In several embodiments provided by the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs.

[0073] The above are only the embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A water conservancy comprehensive monitoring method based on multimodal data fusion, characterized in that: The following steps are involved: Collecting comprehensive water conservancy data, and normalizing and removing noise from the comprehensive water conservancy data, wherein the comprehensive water conservancy data includes remote sensing image data, sensor data, meteorological data, and hydrological historical data; Extracting the features of the water conservancy comprehensive data after normalization and denoising, and fusing the features to obtain the water conservancy comprehensive features; Use encoders to reduce the dimensionality of water conservancy comprehensive features; Input the reduced comprehensive water conservancy features and the comprehensive water conservancy data after normalization and denoising into a support vector machine (SVM), and perform abnormal detection and classification of the comprehensive water conservancy data in combination with an isolation forest; Output the classification results to achieve comprehensive water conservancy monitoring.

2. The water conservancy integrated monitoring method based on multimodal data fusion according to claim 1 is characterized in that: The normalization and denoising processing method is: Among them, X new represents the comprehensive water conservancy data after normalization and denoising, X old represents the comprehensive water conservancy data before normalization and denoising, min() represents the minimum function, max() represents the maximum function, WT() represents the wavelet transform processing function, φ represents the wavelet basis function, and j represents the preset number of wavelet decomposition layers.

3. The water conservancy integrated monitoring method based on multimodal data fusion according to claim 2 is characterized in that: The feature extraction method is: Among them, X new (t) represents the normalized and denoised remote sensing image data, sensor data, meteorological data, or hydrological historical data at time t, N represents the preset maximum number, β represents the preset weight of the autoregressive model, AR() represents the autoregressive model prediction function, represents the complex exponential function, F(k) represents X new (t) The eigenvalue at frequency k, where k represents the frequency index, q represents the imaginary unit, and t represents the time index.

4. The water conservancy integrated monitoring method based on multimodal data fusion according to claim 3 is characterized in that: The method for integrating the comprehensive water conservancy features is as follows: Among them, F fuse represents the comprehensive characteristics of water conservancy, M represents the preset modal number, i represents the quantity index, Q i represents the query matrix of F(k), represents the transpose of the key matrix of F(k), d k represents the number of columns of the key matrix, Softmax() represents the Softmax function, γ represents the preset convolutional neural network weight, V i Represents the value matrix of F(k), and CNN() represents the convolution operation function.

5. The water conservancy integrated monitoring method based on multimodal data fusion according to claim 4 is characterized in that: The dimensionality reduction method of the water conservancy comprehensive characteristics is: F low =ReLU(W e F fuse +b e ); Among them, F low represents the comprehensive characteristics of water conservancy after reduction, b e represents the encoder bias vector, W e Represents the encoder weight matrix, and ReLU() represents the special dimensionality reduction function.

6. The water conservancy integrated monitoring method based on multimodal data fusion according to claim 5 is characterized in that: The method of anomaly detection and classification is: Among them, y represents the classification result, a i represents the Lagrange multiplier of the support vector machine, y i represents the preset label of the water conservancy comprehensive data after normalization and denoising, X new (i) represents the i-th normalized and denoised water conservancy comprehensive data, K() represents the kernel function, γ represents the preset anomaly detection weight, IF() represents the isolation forest function, and sgn() represents the sign function.

7. A water conservancy integrated monitoring system based on multimodal data fusion, characterized in that: include: Preprocessing module: collects comprehensive water conservancy data, and performs normalization and noise removal on the comprehensive water conservancy data, wherein the comprehensive water conservancy data includes remote sensing image data, sensor data, meteorological data, and hydrological historical data; Multimodal data fusion module: extracts the features of the water conservancy comprehensive data after normalization and denoising, and fuses the features to obtain the water conservancy comprehensive features; Dimensionality reduction module: Use encoder to reduce the dimension of comprehensive water conservancy features; Monitoring module: input the reduced comprehensive water conservancy features and the comprehensive water conservancy data after normalization and denoising into a support vector machine SVM, and perform abnormal detection and classification of the comprehensive water conservancy data in combination with isolation forest; Output the classification results to achieve comprehensive water conservancy monitoring.

8. The water conservancy integrated monitoring system based on multimodal data fusion according to claim 7 is characterized in that: The preprocessing module, normalization and denoising processing methods are: Among them, X new represents the comprehensive water conservancy data after normalization and denoising, X old represents the comprehensive water conservancy data before normalization and denoising, min() represents the minimum function, max() represents the maximum function, WT() represents the wavelet transform processing function, φ represents the wavelet basis function, and j represents the preset number of wavelet decomposition layers.

9. The water conservancy integrated monitoring system based on multimodal data fusion according to claim 8 is characterized in that: The multimodal data fusion module, the feature extraction method is: Among them, X new (t) represents the normalized and denoised remote sensing image data, sensor data, meteorological data, or hydrological historical data at time t, N represents the preset maximum number, β represents the preset weight of the autoregressive model, AR() represents the autoregressive model prediction function, represents the complex exponential function, F(k) represents X new (t) The eigenvalue at frequency k, where k represents the frequency index, q represents the imaginary unit, and t represents the time index.

10. The water conservancy integrated monitoring method based on multimodal data fusion according to claim 9 is characterized in that: The multimodal data fusion module and the method for fusing comprehensive water conservancy features are as follows: Among them, F fuse represents the comprehensive characteristics of water conservancy, M represents the preset modal number, i represents the quantity index, Q i represents the query matrix of F(k), represents the transpose of the key matrix of F(k), d k represents the number of columns of the key matrix, Softmax() represents the Softmax function, γ represents the preset convolutional neural network weight, V i Represents the value matrix of F(k), and CNN() represents the convolution operation function.

Images