An AI intelligent real-time monitoring method for sewage outlet based on multi-modal data processing

Abstract

The present application relates to a kind of sewage outlet AI intelligent real-time monitoring method based on multimodal data processing, comprising: collecting sewage data and pre-processing, obtain sewage dataset;Sewage dataset is labeled and divided;FusionTransformer-Net model is constructed and trained;Get the sewage data to be monitored and pre-processing, the sewage data to be monitored after pre-processing is input into the model after training, and real-time monitoring result is obtained.The present application has the advantages of low cost, high precision, strong continuity, good real-time performance;Using deep learning algorithm, the multimodal information of sewage discharge port is identified in real time, and the robustness is strong, can identify various sewage ports, and the application range is wide;Using back-end cloud platform, the identification result is stored, the sewage port information is visualized, and the sewage information is intelligently analyzed, with the advantage of strong intuitiveness;Not only can sewage outlet be monitored, but also other targets can be monitored by replacing dataset.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of environmental monitoring, and particularly relates to a sewage outlet AI intelligent real-time monitoring method based on multi-modal data processing. BACKGROUND

[0002] Water pollution caused by illegal discharge of wastewater has become a major factor of water resource shortage, and how to effectively supervise the discharge of wastewater has become one of the key and difficult points of current water pollution prevention and control. With the advent of the industrial era, the economy has developed rapidly, but the ecological environment on which human beings depend has been seriously damaged, especially the water environment has been seriously polluted.

[0003] The sewage source monitoring technology in China is still in its infancy and mainly relies on manual operation, which is highly lagging. At present, the main sewage discharge monitoring in China mainly includes the following: first, manual inspection and offline sampling, both of which are inefficient and costly; second, monitoring by a sewage monitoring system based on a contact type sensor, which can realize contact type real-time online monitoring and early warning based on the sensor and can realize real-time monitoring of the change of the sewage, but the sensor is easy to be eroded, the cost is high, it is not easy to maintain, the background intelligence is low, and complex logic discrimination cannot be completed; third, a sewage monitoring system using image recognition has appeared in China at present, but it can only identify whether there is sewage or not, cannot classify the sewage, has low automation, cannot realize automatic identification, has low automation and intelligence, lacks an intelligent online monitoring platform that can integrate and analyze data, and it is difficult to continuously and real-timely monitor the discharge.

[0004] Foreign sewage outlet monitoring started early, and so far, most of the water quality monitoring technologies abroad adopt an automatic water quality monitoring system, which integrates computer communication technology, sensor technology and automatic control technology, but is also based on a contact type water quality sensor, and has high deployment cost. Foreign image monitoring of sewage mainly focuses on the detection of sewage pipes and sewer pipes, and less on the monitoring of sewage outlet. SUMMARY

[0005] In order to solve the problems of high sensor deployment cost, difficulty in maintenance and difficulty in continuously and real-timely monitoring the discharge, the present application aims to provide a sewage outlet AI intelligent real-time monitoring method based on multi-modal data processing, which has low cost, is easy to maintain and can high-precision real-timely monitor the sewage discharge of the sewage outlet.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical scheme: a sewage outlet AI intelligent real-time monitoring method based on multi-modal data processing, which comprises the following sequential steps:

[0007] (1) Collecting and preprocessing multi-modal sewage data: the multi-modal sewage data includes sewage images, sewage multispectral data and sewage sensor data, and the preprocessed multi-modal sewage data constitutes a multi-modal sewage data set;

[0008] (2) Labeling the multi-modal sewage data set using a semi-supervised deep learning network model, randomly dividing it into a training set, a test set and a validation set;

[0009] (3) Constructing a FusionTransformer-Net model;

[0010] (4) Input the training set into the FusionTransformer-Net model for training to obtain the trained FusionTransformer-Net model;

[0011] (5) Obtain the sewage data to be monitored and preprocess it, input the preprocessed sewage data to be monitored into the trained FusionTransformer-Net model to obtain the real-time monitoring result, and send the real-time monitoring result and the sewage data to be monitored to the semi-supervised deep learning network model for further learning and optimization.

[0012] The step (1) is specifically referring to: the sewage data is collected by using a camera device on site to collect images and then screened, and the finally screened data contains data of sewage outlets in spring, summer, autumn and winter, data of sewage outlets in different scales, data of sewage outlets in sunny, cloudy, rainy and night conditions;

[0013] The preprocessing refers to:

[0014] For sewage image data, denoising, contrast enhancement and uniform adjustment to a fixed size are performed;

[0015] For sewage multispectral data, first use the Savitzky-Golay filter to remove noise, then use zero-mean unit variance to standardize the denoised data, and finally use principal component analysis to reduce the dimension of the standardized result;

[0016] For sewage sensor data, first use interpolation replacement to fill in missing values, and then time align and unify the dimension with the preprocessed sewage image data and sewage multispectral data.

[0017] The step (2) is specifically referring to: first, label part of the sewage data, then use the semi-supervised deep learning network model to label the remaining data set.

[0018] The step (3) is specifically referring to: the FusionTransformer-Net model comprises a feature extraction module, a multi-modal fusion module, a fusion processing module and a prediction module;

[0019] The feature extraction module uses Vision Transformer (ViT) to extract image global features, uses 1D Swin Transformer to extract spectral features, and uses Time Series Transformer (TST) to process time series sensor data;

[0020] The multi-modal fusion module uses Shared Embedding Space to map image, spectral data and sensor data features to the same dimensional vector space respectively; then uses Cross-Attention Transformer (CAT) for cross-modal feature interaction, taking image features as Query, spectral features and sensor features as Key and Value respectively, to generate image-enhanced spectral and sensor data features, then taking spectral features as Query, interacting again to generate fused high-dimensional feature representation, and finally using 4-layer multi-layer interaction to ensure the transmission of depth information between modalities; for the Cross-Attention Transformer in the multi-modal fusion module, a learnable weight coefficient is added to each modality in each layer of Cross-Attention Transformer to adjust the importance of the modality, and each modality refers to any one of image, spectrum and sensor; the features of each modality are multiplied by the weight coefficient and then input into the multi-head attention mechanism, and the weighted feature calculation formula is as follows:

[0021] X′ I =w I *X I

[0022] X′ S =w S *X S

[0023] X′ T =w T *X T

[0024] In the formula, I is the image data modality, S is the spectral data modality, and T is the sensor data modality; the feature matrices corresponding to I, S and T are X I , X S , X T , and the weight coefficients corresponding to I, S and T are w I , w S , w T, X' T , X' S , X' T are weighted image, spectral and sensor data features, respectively;

[0025] The calculation formula of the multi-head attention mechanism is as follows:

[0026]

[0027] Wherein, d k is the dimension of the spectral feature;

[0028] The fusion processing module first splices the features after the interaction of multiple modalities to form a final fusion feature vector, and then uses a feedforward neural network to extract high-level information of the fusion features, uses a Swish activation function instead of a traditional activation function in the feedforward neural network, and ensures smooth gradient flow through the use of residual connection;

[0029] The prediction module uses a fully connected layer to calculate and predict the feature vector, realizing the sewage classification task; for regression prediction, a 4-layer fully connected layer and an activation function are used to calculate and predict the feature vector, realizing the regression task of water quality parameter numerical prediction.

[0030] In step (3), the FusionTransformer-Net model is loaded on the front-end edge computing monitoring device or the background server; when the FusionTransformer-Net model is loaded on the front-end edge computing monitoring device, the front-end edge computing monitoring device performs fusion and calculation prediction on multi-modal data, and sends the result to the background server through a wireless network; when the FusionTransformer-Net model is loaded on the background server, the front-end edge computing monitoring device sends the sewage data to the background server through a wireless network, which identifies the data, and simultaneously sends the real-time monitoring result and the sewage data to be monitored to the semi-supervised deep learning network model for further learning and optimization.

[0031] From the above technical solutions, the beneficial effects of the present application are: first, the present application uses a method combining non-contact and contact monitoring to monitor the sewage discharge outlet, which has the advantages of low cost, high precision, strong continuity, and good real-time performance; second, the present application uses a deep learning algorithm to identify the sewage discharge outlet, which has strong robustness and can identify various sewage outlets, and has a wide range of applications; third, the present application uses a back-end cloud platform to store the identification results, visualizes the sewage outlet information, and intelligently analyzes the sewage information, which has the advantage of strong intuitiveness; fourth, the present application not only can monitor sewage discharge outlets, but also can monitor other targets by replacing the data set, such as in the field of fire monitoring and security, so that the system has the advantage of high application potential. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 Flow chart of the method of the present application;

[0033] Figure 2 Structure diagram of the FusionTransformer-Net model in the present application. DETAILED DESCRIPTION

[0034] As Figure 1 shown, an AI intelligent real-time monitoring method for sewage outlet based on multi-modal data processing comprises the following sequential steps:

[0035] (1) Collecting multi-modal sewage data and performing preprocessing: the multi-modal sewage data comprises sewage images, sewage multi-spectral data and sewage sensor data, and the preprocessed multi-modal sewage data constitutes a multi-modal sewage data set; the sewage multi-spectral data comprises infrared spectrum and ultraviolet spectrum, and the sewage sensor data comprises total N, P, K, COD, etc.;

[0036] (2) Labeling the multi-modal sewage data set using a semi-supervised deep learning network model, and randomly dividing to obtain a training set, a test set and a validation set;

[0037] (3) Constructing a FusionTransformer-Net model;

[0038] (4) Inputting the training set into the FusionTransformer-Net model for training to obtain a trained FusionTransformer-Net model;

[0039] (5) Obtaining sewage data to be monitored and performing preprocessing, inputting the preprocessed sewage data to be monitored into the trained FusionTransformer-Net model to obtain a real-time monitoring result, and simultaneously sending the real-time monitoring result and the sewage data to be monitored to the semi-supervised deep learning network model for further learning and optimization.

[0040] The step (1) specifically refers to: the sewage data is screened after the images are collected by the on-site camera equipment, and the finally screened data contains data of the sewage outlet in spring, summer, autumn and winter, data of the sewage outlet in different scales, data of the sewage outlet under sunny, cloudy, rainy and night conditions;

[0041] The preprocessing refers to:

[0042] For sewage image data, denoising, contrast enhancement and uniform adjustment to a fixed size are performed;

[0043] For wastewater multispectral data, we first used Savitzky-Golay filter to remove noise, then used zero mean unit variance to standardize the denoised data, and finally used principal component analysis to reduce the dimensionality of the standardized results.

[0044] For wastewater sensor data, interpolation is first used to fill in missing values, and then time-aligned and dimension-unified with preprocessed wastewater image data and wastewater multispectral data.

[0045] Step (2) specifically refers to the following: due to the large amount of multimodal sewage data, some sewage data are first manually labeled, and then a semi-supervised deep learning network model is used to label the remaining dataset.

[0046] like Figure 2 As shown, step (3) specifically refers to the following: The FusionTransformer-Net model includes a feature extraction module, a multimodal fusion module, a fusion processing module, and a prediction module;

[0047] The feature extraction module uses Vision Transformer (ViT) to extract global image features, 1DSwin Transformer to extract spectral features, and Time Series Transformer (TST) to process time series sensor data.

[0048] The Vision Transformer takes an RGB image as input, with dimensions H×W×3, where H is the number of pixels vertically and W is the number of pixels horizontally. The output image features are Z... image The 1D Swin Transformer takes a spectral image containing multiple bands as input, with dimensions B×H×W, where B is the number of bands, and H and W are the height and width of the image, respectively. The output spectral features are represented by Z. spectrum The Time Series Transformer takes an N×M matrix as input, where N is the number of sensors and M is the data dimension of each sensor, and outputs a matrix R. Tsensor×D Where Tsensor is the number of sample points of the sensor data, and D is the feature dimension after embedding.

[0049] The multi-modal fusion module uses a shared embedding space to map image, spectral data and sensor data features into the same dimensional vector space; then uses Cross-Attention Transformer (CAT) for cross-modal feature interaction, taking image features as Query, spectral features and sensor features as Key and Value respectively, to generate image-enhanced spectral and sensor data features, and then taking spectral features as Query to interact again to generate fused high-dimensional feature representation, and finally using 4 layers of multi-layer interaction to ensure the transmission of depth information between modalities; for the Cross-Attention Transformer in the multi-modal fusion module, a learnable weight coefficient is added to each modality in each layer of Cross-Attention Transformer to adjust the importance of the modality, and each modality refers to any one of image, spectrum, sensor, and the weighted feature calculation formula is as follows:

[0050] X' I =w I *X I

[0051] X' S =w S *X S

[0052] X' T =w T *X T

[0053] In the formula, I is the image data modality, S is the spectral data modality, and T is the sensor data modality, and the feature matrices corresponding to I, S and T are X I , X S , X T , the weight coefficients corresponding to I, S and T are w I , w S , w T , X' T , X' S , X' T are the weighted image, spectral and sensor data features respectively;

[0054] The calculation formula of the multi-head attention mechanism is as follows:

[0055]

[0056] Where d k is the dimension of the spectral feature;

[0057] The fusion processing module first splices the features of the interaction of multiple modalities to form a final fusion feature vector, and then uses a feedforward neural network to extract high-level information of the fusion features, wherein a Swish activation function is used instead of a traditional activation function in the feedforward neural network, and a residual connection is used to ensure smooth gradient flow.

[0058] The prediction module uses a fully connected layer to calculate and predict the feature vector, thereby realizing the sewage classification task; for regression prediction, a 4-layer fully connected layer and an activation function are used to calculate and predict the feature vector, thereby realizing the regression task of water quality parameter value prediction.

[0059] In step (3), the FusionTransformer-Net model is loaded on a front-end edge computing monitoring device or a background server; when the FusionTransformer-Net model is loaded on the front-end edge computing monitoring device, the front-end edge computing monitoring device performs fusion and calculation prediction on multi-modal data, and sends the results to the background server through a wireless network; when the FusionTransformer-Net model is loaded on the background server, the front-end edge computing monitoring device sends the sewage data to the background server through a wireless network, and the background server performs identification, and simultaneously sends the real-time monitoring results and the sewage data to be monitored to the semi-supervised deep learning network model for further learning and optimization.

[0060] The front-end edge computing device mainly includes a camera, a spectrum probe, a water quality sensor, an edge computing development board and a solar cell.

[0061] For areas with poor network signals, high delay and high packet loss rate, the FusionTransformer-Net model is deployed on the edge computing development board of the front-end edge computing device, which performs fusion calculation and analysis on the data transmitted by the camera, the spectrum probe and the water quality sensor in real time and gives the detection results. The detection results are converted into Json format and stored in the memory of the edge computing development board, and the detection results are sent to the background server through a wireless network according to the http post protocol at regular intervals.

[0062] For areas with good signals, the FusionTransformer-Net model is deployed on the background server, and the front-end edge computing device converts the information of the camera, the spectrum probe and the water quality sensor into Json format and sends it to the background server according to the http post protocol. The FusionTransformer-Net model on the background server calculates and detects the received information, and stores the detection results in the database on the server.

[0063] In summary, the present application uses a method combining non-contact (image and spectral monitoring) and contact monitoring (water quality sensor monitoring) to monitor sewage outlets, has the advantages of low cost, high precision, strong continuity, good real-time performance; the present application uses a deep learning algorithm, namely the FusionTransformer-Net model, to identify sewage outlets, has strong robustness, can identify various sewage outlets, and has a wide range of applications; the present application uses a back-end cloud platform to store the identification results, visually processes sewage outlet information, and intelligently analyzes sewage information, has the advantage of strong intuitiveness; the present application can not only monitor sewage outlets, but also can monitor other targets by replacing the data set, for example, in the field of fire monitoring and security, so that the system has the advantage of high application potential.

Claims

1. A real-time AI-powered monitoring method for wastewater discharge outlets based on multimodal data processing, characterized in that: The method includes the following steps in sequence: (1) Collecting and preprocessing multimodal wastewater data: The multimodal wastewater data includes wastewater images, wastewater multispectral data and wastewater sensor data. The preprocessed multimodal wastewater data constitutes a multimodal wastewater dataset. (2) The multimodal sewage dataset was labeled using a semi-supervised deep learning network model and randomly divided to obtain a training set, a test set and a validation set; (3) Construct the FusionTransformer-Net model; The FusionTransformer-Net model includes a feature extraction module, a multimodal fusion module, a fusion processing module, and a prediction module; The feature extraction module uses Vision Transformer (ViT) to extract global image features, 1D SwinTransformer to extract spectral features, and Time Series Transformer (TST) to process time series sensor data. The multimodal fusion module uses a shared embedding space to map image, spectral, and sensor data features to a vector space of the same dimension. Then, a Cross-Attention Transformer (CAT) is used for cross-modal feature interaction. Using image features as the query, and spectral and sensor features as the key and value, respectively, image-enhanced spectral and sensor data features are generated. Then, using spectral features as the query again, interaction is performed to generate a fused high-dimensional feature representation. Finally, four layers of multi-level interaction are used to ensure the transfer of deep information between modalities. For the Cross-Attention Transformer in the multimodal fusion module, a learnable weight coefficient is added to each modality in each layer to adjust the importance of the modality. Each modality refers to any one of image, spectrum, or sensor data. The features of each modality are multiplied by this weight coefficient and then input into the multi-head attention mechanism. The fusion processing module first concatenates the features after interaction of multiple modalities to form the final fused feature vector. Then, it uses a feedforward neural network to extract high-level information of the fused features. In the feedforward neural network, the Swish activation function is used instead of the traditional activation function, and the gradient is kept smooth by using residual connections. The prediction module uses a fully connected layer to calculate and predict the feature vector to achieve the wastewater classification task; for regression prediction, it uses four fully connected layers stacked one after the other and activation functions to calculate and predict the feature vector to achieve the regression task of predicting water quality parameter values. (4) Input the training set into the FusionTransformer-Net model for training to obtain the trained FusionTransformer-Net model; (5) Acquire the wastewater data to be monitored and preprocess it. Input the preprocessed wastewater data to be monitored into the trained FusionTransformer-Net model to obtain the real-time monitoring results. At the same time, send the real-time monitoring results and the wastewater data to be monitored to the semi-supervised deep learning network model for further learning and optimization.

2. The AI-based intelligent real-time monitoring method for sewage outlets based on multimodal data processing according to claim 1, characterized in that: The step (1) specifically refers to: the sewage data is collected by the camera equipment on site and then filtered. The final filtered data includes sewage outlet data in spring, summer, autumn and winter, sewage outlet data at different scales, and sewage outlet data under sunny, cloudy, rainy and nighttime conditions. The preprocessing refers to: The wastewater image data was denoised, contrast enhanced, and resized to a fixed size. For wastewater multispectral data, we first used Savitzky-Golay filter to remove noise, then used zero mean unit variance to standardize the denoised data, and finally used principal component analysis to reduce the dimensionality of the standardized results. For wastewater sensor data, interpolation is first used to fill in missing values, and then time-aligned and dimension-unified with preprocessed wastewater image data and wastewater multispectral data.

3. The AI-powered real-time monitoring method for wastewater discharge outlets based on multimodal data processing according to claim 1, characterized in that: Step (2) specifically refers to: first, labeling a portion of the sewage data, and then using a semi-supervised deep learning network model to label the remaining dataset.

4. The AI-based intelligent real-time monitoring method for sewage outlets based on multimodal data processing according to claim 1, characterized in that: Step (3) specifically refers to the following formula for calculating the weighted features: ； ； ； In the formula, I represents the image data mode, S represents the spectral data mode, and T represents the sensor data mode. The feature matrices corresponding to I, S, and T are respectively... , , The weighting coefficients for I, S, and T are respectively , , , , These are the weighted features of the image, spectrum, and sensor data, respectively. The formula for calculating the multi-head attention mechanism is as follows: ； in, It is a dimension of spectral characteristics.

5. The AI-based intelligent real-time monitoring method for sewage outlets based on multimodal data processing according to claim 1, characterized in that: In step (3), the FusionTransformer-Net model is loaded onto the front-end edge computing monitoring device or the back-end server. When the FusionTransformer-Net model is loaded onto the front-end edge computing monitoring device, the front-end edge computing monitoring device sends the results to the back-end server via a wireless network by fusing multimodal data and performing calculations and predictions. When the FusionTransformer-Net model is loaded onto the back-end server, the front-end edge computing monitoring device sends the sewage data to the back-end server via a wireless network for identification. At the same time, the back-end server sends the real-time monitoring results and the sewage data to be monitored to the semi-supervised deep learning network model for further learning and optimization.

Images