Sewage outlet AI intelligent real-time monitoring method based on multi-modal data processing

Abstract

The invention relates to a sewage outlet AI intelligent real-time monitoring method based on multi-modal data processing, and the method comprises the steps: collecting and preprocessing sewage data, and obtaining a sewage data set; marking and dividing the sewage data set; a Fusion Transform-Net model is constructed, and the Fusion Transform-Net model is trained; and obtaining sewage data to be monitored and preprocessing the sewage data to be monitored, and inputting the preprocessed sewage data to be monitored into the trained model to obtain a real-time monitoring result. The method has the advantages of low cost, high precision, strong continuity and good real-time performance; the deep learning algorithm is used for recognizing the multi-modal information of the sewage discharge port in real time, robustness is high, various sewage ports can be recognized, and the application range is wide; a back-end cloud platform is used for storing an identification result, performing visualization processing on sewage port information, and performing intelligent analysis on the sewage information, so that the system has the advantage of high intuition; not only can the sewage discharge port be monitored, but also other targets can be monitored by replacing the data set.

Description

Technical Field

[0001] The present invention relates to the field of environmental monitoring technology, and in particular to an AI intelligent real-time monitoring method for sewage outlets based on multimodal data processing. Background Art

[0002] Water pollution caused by illegal wastewater discharge has become the main factor in water shortage. How to effectively regulate wastewater discharge has become one of the key and difficult points in the current water pollution prevention and control. With the advent of the industrial age, the economy has developed rapidly, but the ecological environment on which human beings depend for survival has also been severely damaged, especially the water environment has been seriously polluted.

[0003] my country's sewage source monitoring technology is still in its infancy, mainly relying on manual operation and lagging. At present, my country's main sewage discharge monitoring is mainly the following: First, manual inspection and offline sampling, both of which are inefficient and costly; Second, through the sewage monitoring system based on contact sensors, the system can realize real-time online monitoring and early warning based on sensor contact, and can grasp the changes of sewage in real time, but the sensor is easily corroded, the cost is high, it is not easy to maintain, the background intelligence is low, and it cannot complete complex logical judgments; Third, there are also systems that use image recognition to monitor sewage in China, but they can only identify whether there is sewage, but cannot classify sewage. The degree of automation is low and cannot be automatically identified. There are problems with low automation and intelligence. There is a lack of an intelligent online monitoring platform that can integrate and analyze data, making it difficult to monitor emissions continuously and in real time.

[0004] Foreign countries started monitoring sewage outlets earlier. So far, most foreign water quality monitoring technologies use automatic water pollution monitoring systems, which integrate computer communication technology, sensor technology and automatic control technology, but are also based on contact water quality sensors, and have high deployment costs. Foreign image monitoring of sewage mainly focuses on the detection of sewage pipes and sewer pipes, and there is little research on sewage outlet monitoring. Summary of the invention

[0005] In order to solve the problems of high sensor deployment cost, difficult maintenance, and difficulty in continuous and real-time monitoring of emissions, the purpose of the present invention is to provide an AI intelligent real-time monitoring method for sewage outlets based on multimodal data processing, which is low in cost, easy to maintain, and can monitor the sewage discharge situation of sewage outlets in real time with high precision.

[0006] To achieve the above object, the present invention adopts the following technical solution: an AI intelligent real-time monitoring method for sewage outlets based on multimodal data processing, the method comprising the following steps in order:

[0007] (1) Collecting multimodal sewage data and preprocessing: the multimodal sewage data includes sewage images, sewage multispectral data and sewage sensor data, and the preprocessed multimodal sewage data constitutes a multimodal sewage data set;

[0008] (2) Use a semi-supervised deep learning network model to annotate the multimodal sewage dataset and randomly divide it into training, test, and validation sets;

[0009] (3) Build the FusionTransformer-Net model;

[0010] (4) Inputting the training set into the FusionTransformer-Net model for training to obtain a trained FusionTransformer-Net model;

[0011] (5) Obtain the sewage data to be monitored and preprocess it, input the preprocessed sewage data to be monitored into the trained FusionTransformer-Net model to obtain real-time monitoring results, and send the real-time monitoring results and the sewage data to be monitored to the semi-supervised deep learning network model for further learning and optimization.

[0012] The step (1) specifically refers to: the sewage data is screened after collecting images using a camera on site, and the screened data finally includes data of sewage outlets in spring, summer, autumn and winter, data of sewage outlets at different scales, and data of sewage outlets under sunny, cloudy, overcast, rainy and night conditions;

[0013] The pre-processing refers to:

[0014] For sewage image data, denoise, enhance contrast, and uniformly adjust to a fixed size;

[0015] For sewage multispectral data, the Savitzky-Golay filter was first used to remove noise, then the denoised data was standardized using zero mean unit variance, and finally principal component analysis was used to reduce the dimension of the standardized results;

[0016] For sewage sensor data, interpolation is first used to fill missing values, and then time alignment and unified dimension are performed with the preprocessed sewage image data and sewage multispectral data.

[0017] The step (2) specifically refers to: firstly labeling part of the sewage data, and then labeling the remaining data set using a semi-supervised deep learning network model.

[0018] The step (3) specifically refers to: the FusionTransformer-Net model includes a feature extraction module, a multimodal fusion module, a fusion processing module and a prediction module;

[0019] The feature extraction module uses Vision Transformer (ViT) to extract global image features, uses 1DSwin Transformer to extract spectral features, and uses Time Series Transformer (TST) to process time series sensor data;

[0020] The multimodal fusion module uses a shared embedding space, i.e., Shared Embedding Space, to map the image, spectral data, and sensor data features to vector spaces of the same dimension respectively; then, a Cross-Attention Transformer, i.e., CAT, is used for cross-modal feature interaction, with image features as query, spectral features and sensor features as key and value respectively, to generate image-enhanced spectral and sensor data features, and then with spectral features as query, interaction is performed again to generate a fused high-dimensional feature representation, and finally, 4 layers of multi-layer interaction are used to ensure deep information transmission between modalities; for the Cross-Attention Transformer in the multimodal fusion module, a learnable weight coefficient is added to each modality in each layer of the Cross-Attention Transformer to adjust the importance of the modality, and each modality refers to any one of image, spectrum, and sensor. The features of each modality are multiplied by the weight coefficient and input into the multi-head attention mechanism. The weighted feature calculation formula is as follows:

[0021] X′ I =w I *X I

[0022] X′ S =w S *X S

[0023] X′ T =w T *X T

[0024] Where I is the image data modality, S is the spectral data modality, and T is the sensor data modality. The feature matrices corresponding to I, S, and T are X I , X S , X T , the weight coefficients corresponding to I, S, and T are w I 、w S 、w T, X' T , X' S , X' T They are the weighted image, spectral and sensor data features;

[0025] The calculation formula of the multi-head attention mechanism is as follows:

[0026]

[0027] Among them, d k is the dimension of the spectral feature;

[0028] The fusion processing module first concatenates the interactive features of multiple modalities to form a final fusion feature vector, and then uses a feedforward neural network to extract high-level information of the fusion features. The Swish activation function is used in the feedforward neural network instead of the traditional activation function, and the residual connection is used to ensure the smoothness of the gradient;

[0029] The prediction module uses a fully connected layer to calculate and predict the feature vector to achieve the sewage classification task; for regression prediction, 4 layers of fully connected layers and activation functions are used to calculate and predict the feature vector to achieve the regression task of numerical prediction of water quality parameters.

[0030] In step (3), the FusionTransformer-Net model is loaded on the front-end edge computing monitoring device or on the back-end server; when the FusionTransformer-Net model is loaded on the front-end edge computing monitoring device, the front-end edge computing monitoring device fuses multimodal data and performs calculation prediction, and sends the results to the back-end server via a wireless network; when the FusionTransformer-Net model is loaded on the back-end server, the front-end edge computing monitoring device sends the sewage data to the back-end server via a wireless network, and the back-end server identifies it, and at the same time sends the real-time monitoring results and the sewage data to be monitored to the semi-supervised deep learning network model for further learning and optimization.

[0031] It can be seen from the above technical scheme that the beneficial effects of the present invention are: first, the present invention uses a method combining non-contact and contact monitoring to monitor the sewage outlet, which has the advantages of low cost, high accuracy, strong continuity and good real-time performance; second, the present invention uses a deep learning algorithm to identify the sewage outlet, which has strong robustness and can identify various types of sewage outlets, and has a wide range of applications; third, the present invention uses a back-end cloud platform to store the recognition results, visualizes the sewage outlet information, and intelligently analyzes the sewage information, which has the advantage of strong intuitiveness; fourth, the present invention can not only monitor the sewage outlet, but also realize the monitoring of other targets by replacing the data set, such as in the field of fire monitoring and security, which makes the system have the advantage of high application potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of the method of the present invention;

[0033] Figure 2 It is a structural diagram of the FusionTransformer-Net model in the present invention. DETAILED DESCRIPTION

[0034] like Figure 1 As shown, an AI intelligent real-time monitoring method for sewage outlets based on multimodal data processing includes the following steps in sequence:

[0035] (1) Collecting multimodal sewage data and preprocessing: the multimodal sewage data includes sewage images, sewage multispectral data and sewage sensor data, and the preprocessed multimodal sewage data constitute a multimodal sewage data set; the sewage multispectral data includes infrared spectrum and ultraviolet spectrum, and the sewage sensor data includes total N, P, K, COD, etc.;

[0036] (2) Use a semi-supervised deep learning network model to annotate the multimodal sewage dataset and randomly divide it into training, test, and validation sets;

[0037] (3) Build the FusionTransformer-Net model;

[0038] (4) Inputting the training set into the FusionTransformer-Net model for training to obtain a trained FusionTransformer-Net model;

[0039] (5) Obtain the sewage data to be monitored and preprocess it, input the preprocessed sewage data to be monitored into the trained FusionTransformer-Net model to obtain real-time monitoring results, and send the real-time monitoring results and the sewage data to be monitored to the semi-supervised deep learning network model for further learning and optimization.

[0040] The step (1) specifically refers to: the sewage data is screened after collecting images using a camera on site, and the screened data finally includes data of sewage outlets in spring, summer, autumn and winter, data of sewage outlets at different scales, and data of sewage outlets under sunny, cloudy, overcast, rainy and night conditions;

[0041] The pre-processing refers to:

[0042] For sewage image data, denoise, enhance contrast, and uniformly adjust to a fixed size;

[0043] For sewage multispectral data, the Savitzky-Golay filter was first used to remove noise, then the denoised data was standardized using zero mean unit variance, and finally principal component analysis was used to reduce the dimension of the standardized results;

[0044] For sewage sensor data, interpolation is first used to fill missing values, and then time alignment and unified dimension are performed with the preprocessed sewage image data and sewage multispectral data.

[0045] The step (2) specifically means: since the amount of multimodal sewage data is large, part of the sewage data is first manually labeled, and then the remaining data set is labeled using a semi-supervised deep learning network model.

[0046] like Figure 2 As shown, the step (3) specifically refers to: the FusionTransformer-Net model includes a feature extraction module, a multimodal fusion module, a fusion processing module and a prediction module;

[0047] The feature extraction module uses Vision Transformer (ViT) to extract global image features, uses 1DSwin Transformer to extract spectral features, and uses Time Series Transformer (TST) to process time series sensor data;

[0048] The input image of Vision Transformer is an RGB image with a size of H×W×3, where H is the number of pixels in the vertical direction and W is the number of pixels in the horizontal direction. The output image feature is Z image ; The input of 1D Swin Transformer is a spectral image containing multiple bands with a size of B×H×W, where B is the number of bands, H and W are the height and width of the image respectively, and the output spectral feature is represented by Z spectrum ; The input of Time Series Transformer is an N×M matrix, where N is the number of sensors and M is the data dimension of each sensor. The output is a matrix R Tsensor×D , where Tsensor is the number of sample points of sensor data and D is the feature dimension after embedding.

[0049] The multimodal fusion module uses a shared embedding space, i.e., Shared Embedding Space, to map the image, spectral data, and sensor data features to vector spaces of the same dimension respectively; then, a Cross-Attention Transformer, i.e., CAT, is used for cross-modal feature interaction, with image features as query, spectral features and sensor features as key and value respectively, to generate image-enhanced spectral and sensor data features, and then with spectral features as query, interaction is performed again to generate a fused high-dimensional feature representation, and finally, 4 layers of multi-layer interaction are used to ensure deep information transmission between modalities; for the Cross-Attention Transformer in the multimodal fusion module, a learnable weight coefficient is added to each modality in each layer of the Cross-Attention Transformer to adjust the importance of the modality, and each modality refers to any one of image, spectrum, and sensor. The features of each modality are multiplied by the weight coefficient and input into the multi-head attention mechanism. The weighted feature calculation formula is as follows:

[0050] X′ I =w I *X I

[0051] X′ S =w S *X S

[0052] X′ T =w T *X T

[0053] Where I is the image data modality, S is the spectral data modality, and T is the sensor data modality. The feature matrices corresponding to I, S, and T are X I , X S , X T , the weight coefficients corresponding to I, S, and T are w I 、w S 、w T , X' T , X' S , X' T They are the weighted image, spectral and sensor data features;

[0054] The calculation formula of the multi-head attention mechanism is as follows:

[0055]

[0056] Among them, d k is the dimension of the spectral feature;

[0057] The fusion processing module first concatenates the interactive features of multiple modalities to form a final fusion feature vector, and then uses a feedforward neural network to extract high-level information of the fusion features. The Swish activation function is used in the feedforward neural network instead of the traditional activation function, and the residual connection is used to ensure the smoothness of the gradient;

[0058] The prediction module uses a fully connected layer to calculate and predict the feature vector to achieve the sewage classification task; for regression prediction, 4 layers of fully connected layers and activation functions are used to calculate and predict the feature vector to achieve the regression task of numerical prediction of water quality parameters.

[0059] In step (3), the FusionTransformer-Net model is loaded on the front-end edge computing monitoring device or on the back-end server; when the FusionTransformer-Net model is loaded on the front-end edge computing monitoring device, the front-end edge computing monitoring device fuses multimodal data and performs calculation prediction, and sends the results to the back-end server via a wireless network; when the FusionTransformer-Net model is loaded on the back-end server, the front-end edge computing monitoring device sends the sewage data to the back-end server via a wireless network, and the back-end server identifies it, and at the same time sends the real-time monitoring results and the sewage data to be monitored to the semi-supervised deep learning network model for further learning and optimization.

[0060] The front-end edge computing equipment mainly includes a camera, a spectral probe, a water quality sensor, an edge computing development board and a solar cell.

[0061] For areas with poor network signals, high latency and packet loss rates, the FusionTransformer-Net model is deployed on the edge computing development board of the front-end edge computing device to perform real-time fusion calculation and analysis on the data transmitted by the camera, spectral probe and water quality sensor and give the test results. The test results are converted into Json format and stored in the memory of the edge computing development board, and the test results are sent to the backend server through the wireless network according to the http post protocol at regular intervals.

[0062] For areas with good signals, the FusionTransformer-Net model is deployed on the backend server. The front-end edge computing device converts the information from the camera, spectral probe, and water quality sensor into Json format and sends it to the backend server via the http post protocol. The FusionTransformer-Net model on the backend server performs calculations and tests on the received information and stores the test results in the database on the server.

[0063] In summary, the present invention uses a method combining non-contact (image and spectral monitoring) and contact monitoring (water quality sensor monitoring) to monitor sewage outlets, which has the advantages of low cost, high accuracy, strong continuity and good real-time performance; the present invention uses a deep learning algorithm, namely the FusionTransformer-Net model, to identify sewage outlets, which has strong robustness and can identify various types of sewage outlets, and has a wide range of applications; the present invention uses a back-end cloud platform to store the recognition results, visualizes the sewage outlet information, and intelligently analyzes the sewage information, which has the advantage of strong intuitiveness; the present invention can not only monitor sewage outlets, but also monitor other targets by replacing data sets, such as in the fields of fire monitoring and security, so that the system has the advantage of high application potential.

Claims

1. An AI intelligent real-time monitoring method for sewage outlets based on multimodal data processing, characterized in that: The method comprises the following steps in order: (1) Collecting multimodal sewage data and preprocessing: the multimodal sewage data includes sewage images, sewage multispectral data and sewage sensor data, and the preprocessed multimodal sewage data constitutes a multimodal sewage data set; (2) Use a semi-supervised deep learning network model to annotate the multimodal sewage dataset and randomly divide it into training, test, and validation sets; (3) Build the FusionTransformer-Net model; (4) Inputting the training set into the FusionTransformer-Net model for training to obtain a trained FusionTransformer-Net model; (5) Obtain the sewage data to be monitored and preprocess it, input the preprocessed sewage data to be monitored into the trained FusionTransformer-Net model to obtain real-time monitoring results, and send the real-time monitoring results and the sewage data to be monitored to the semi-supervised deep learning network model for further learning and optimization.

2. The AI ​​intelligent real-time monitoring method for sewage outlets based on multimodal data processing according to claim 1 is characterized in that: The step (1) specifically refers to: the sewage data is screened after collecting images using a camera on site, and the screened data finally includes data of sewage outlets in spring, summer, autumn and winter, data of sewage outlets at different scales, and data of sewage outlets under sunny, cloudy, overcast, rainy and night conditions; The pre-processing refers to: For sewage image data, denoise, enhance contrast, and uniformly adjust to a fixed size; For sewage multispectral data, the Savitzky-Golay filter was first used to remove noise, then the denoised data was standardized using zero mean unit variance, and finally principal component analysis was used to reduce the dimension of the standardized results; For sewage sensor data, interpolation is first used to fill missing values, and then time alignment and unified dimension are performed with the preprocessed sewage image data and sewage multispectral data.

3. According to the AI ​​intelligent real-time monitoring method for sewage outlets based on multimodal data processing according to claim 1, it is characterized in that: The step (2) specifically refers to: firstly labeling part of the sewage data, and then labeling the remaining data set using a semi-supervised deep learning network model.

4. The AI ​​intelligent real-time monitoring method for sewage outlets based on multimodal data processing according to claim 1 is characterized in that: The step (3) specifically refers to: the FusionTransformer-Net model includes a feature extraction module, a multimodal fusion module, a fusion processing module and a prediction module; The feature extraction module uses Vision Transformer (ViT) to extract global image features, uses 1D SwinTransformer to extract spectral features, and uses Time Series Transformer (TST) to process time series sensor data; The multimodal fusion module uses a shared embedding space, i.e., Shared Embedding Space, to map the image, spectral data, and sensor data features to vector spaces of the same dimension respectively; then, a Cross-Attention Transformer, i.e., CAT, is used for cross-modal feature interaction, with image features as query, spectral features and sensor features as key and value respectively, to generate image-enhanced spectral and sensor data features, and then with spectral features as query, interaction is performed again to generate a fused high-dimensional feature representation, and finally, 4 layers of multi-layer interaction are used to ensure deep information transmission between modalities; for the Cross-Attention Transformer in the multimodal fusion module, a learnable weight coefficient is added to each modality in each layer of the Cross-Attention Transformer to adjust the importance of the modality, and each modality refers to any one of image, spectrum, and sensor. The features of each modality are multiplied by the weight coefficient and input into the multi-head attention mechanism. The weighted feature calculation formula is as follows: X′ I =w I *X I X′ S =w S *X S X T =w T *X T Where I is the image data modality, S is the spectral data modality, and T is the sensor data modality. The feature matrices corresponding to I, S, and T are X I , X S , X T , the weight coefficients corresponding to I, S, and T are w I 、w S 、w T , X' T , X' S , X' T They are the weighted image, spectral and sensor data features; The calculation formula of the multi-head attention mechanism is as follows: Among them, d k is the dimension of the spectral feature; The fusion processing module first concatenates the interactive features of multiple modalities to form a final fusion feature vector, and then uses a feedforward neural network to extract high-level information of the fusion features. The Swish activation function is used in the feedforward neural network instead of the traditional activation function, and the residual connection is used to ensure the smoothness of the gradient; The prediction module uses a fully connected layer to calculate and predict the feature vector to achieve the sewage classification task; for regression prediction, 4 layers of fully connected layers and activation functions are used to calculate and predict the feature vector to achieve the regression task of numerical prediction of water quality parameters.

5. The AI ​​intelligent real-time monitoring method for sewage outlets based on multimodal data processing according to claim 1 is characterized in that: In step (3), the FusionTransformer-Net model is loaded on the front-end edge computing monitoring device or on the back-end server; when the FusionTransformer-Net model is loaded on the front-end edge computing monitoring device, the front-end edge computing monitoring device fuses multimodal data and performs calculation prediction, and sends the results to the back-end server via a wireless network; when the FusionTransformer-Net model is loaded on the back-end server, the front-end edge computing monitoring device sends the sewage data to the back-end server via a wireless network, and the back-end server identifies it, and at the same time sends the real-time monitoring results and the sewage data to be monitored to the semi-supervised deep learning network model for further learning and optimization.

Images