Water coagulation effect prediction method based on multi-modal data and deep learning model

Abstract

The invention discloses a water quality coagulation effect prediction method based on multi-modal data and a deep learning model, and the method comprises the steps: obtaining a group of to-be-detected multi-modal data sets of target water quality, respectively inputting the to-be-detected floc image and the sensor acquisition parameters at each moment into a first branch and a second branch of a pre-trained improved deep learning model to obtain respective prediction results, and fusing to obtain a fused coagulation effect prediction result of the target water quality; the first branch comprises a ResNet model and a first LSTM model which are connected in sequence, and the second branch comprises a second LSTM model; the coagulation effect prediction result comprises a coagulation effect grade and / or coagulation effect parameters; the coagulation effect grades are used for representing different grades of coagulation effects, and the coagulation effect parameters comprise water quality parameters representing the final coagulation effect. According to the invention, the improved deep learning model is utilized to realize water quality coagulation effect prediction, intelligent optimization of the adding amount of the flocculant can be realized, and the water treatment efficiency and the effluent quality are improved.

Description

Technical Field

[0001] The present invention belongs to the field of water treatment, and specifically relates to a method for predicting water quality coagulation effect based on multimodal data and deep learning models. Background Art

[0002] Water coagulation is a key step in water treatment, and its effectiveness directly impacts the efficiency of subsequent treatment processes and effluent quality. During the coagulation stage, suspended solids and colloids are removed from the water by adding coagulants. Predicting the effectiveness of this process is crucial for optimizing water treatment processes, reducing costs, and improving water quality. However, coagulation effectiveness is influenced by numerous factors, including the type and dosage of coagulant, water quality parameters, and hydraulic conditions. The dynamic nature of these factors makes coagulation effect prediction a complex problem.

[0003] In order to quickly determine the appropriate coagulant dosage under the constraints of nonlinearity and large time lag, the current practice of water companies is to conduct a six-beaker experiment in the laboratory every day (see the existing six-beaker experiment equipment in water plants for details). Figure 1 As shown), that is, different doses of coagulant are added to six groups of the same raw water. After the cup coagulation is completed, the turbidity of the supernatant of the water sample in the beaker is measured, that is, the turbidity after sedimentation, and the corresponding relationship between different coagulant dosages and turbidity is obtained. Then, the coagulant dosage is determined based on the experience of the technicians.

[0004] Although the method of determining the coagulant dosage through the six-beaker test can basically cope with the daily water treatment of water treatment companies, it still has the following disadvantages:

[0005] 1. Unable to cope with sudden changes in water quality. In water treatment, sudden changes in water quality are inevitable. For example, reservoir discharge caused by summer rainstorms or reservoir overflow caused by water shortages will cause a sharp increase in turbidity. The turbidity of raw water can quickly rise from a few NTU to hundreds or even thousands of NTU within a few hours. In addition, when the output or process of sewage discharge enterprises changes, the water quality of industrial wastewater will suddenly change. When the raw water suddenly changes, the coagulant dosage must be adjusted quickly to achieve the effluent water quality standards. At this time, the dosage determined by the previous six-beaker experiment is no longer appropriate. In actual treatment, the dosage can only be determined entirely by manual experience.

[0006] 2. It is difficult to treat low-temperature and low-turbidity water in winter. The coagulant is not fully dissolved at low temperatures, and the low turbidity makes the flocs grow insufficiently. In this state, the coagulant dosage is too small, the turbidity removal after coagulation is insufficient, and the effluent turbidity does not meet the standard. When the coagulant dosage is excessive, the coagulant itself that is not fully dissolved will become a turbidity component, making the effluent turbidity substandard. Therefore, it is usually necessary to repeatedly conduct six-beaker experiments and actual dosage adjustments to determine the coagulant dosage that meets the effluent turbidity requirements;

[0007] 3. Excessive addition of coagulants. Since the amount of coagulant added cannot be automatically adjusted in real time according to the changes in the incoming water, in order to ensure that the effluent quality meets the standards, water companies will choose to add excessive coagulants. This excessive addition will increase the cost of water treatment for the company. The increased cost consists of two parts: one is the coagulant cost, and the other is the subsequent treatment cost of other units. For example, excessive coagulant brings more sludge, which increases the sludge treatment fee; excessive coagulant PAC brings excessive residual aluminum, which requires additional cost to treat.

[0008] 4. Increase carbon emissions. In addition to increased costs, excessive addition of coagulants will also cause more carbon emissions. Water treatment companies are not major contributors to direct carbon emissions, but they are major contributors to indirect carbon emissions. Among indirect carbon emissions, electricity consumption accounts for the largest proportion, and chemicals rank second. Summary of the Invention

[0009] In order to solve the above problems existing in the prior art, the present invention provides a method for predicting water quality coagulation effect based on multimodal data and deep learning models. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0010] A method for predicting water quality coagulation effect based on multimodal data and deep learning models, including:

[0011] Acquire a set of multimodal data sets of target water quality to be measured, including a set of floc images of the target water quality to be measured collected at multiple times and sensor acquisition parameters at each time; the sensor acquisition parameters include water quality parameters that characterize the coagulation effect;

[0012] The group of floc images to be tested and the sensor acquisition parameters at each moment are respectively input into the first branch and the second branch of the pre-trained improved deep learning model, and the first coagulation effect prediction result and the second coagulation effect prediction result are obtained accordingly. The first coagulation effect prediction result and the second coagulation effect prediction result are fused to obtain the fused coagulation effect prediction result of the target water quality; wherein, the first branch includes a ResNet model and a first LSTM model connected in sequence, and the second branch includes a second LSTM model; the improved deep learning model is trained based on multiple groups of sample multimodal data sets of water quality samples; the coagulation effect prediction result includes coagulation effect grade and / or coagulation effect parameters; the coagulation effect grade is used to characterize different levels of coagulation effects, and the coagulation effect parameters include water quality parameters characterizing the final coagulation effect.

[0013] In one embodiment of the present invention, the coagulation effect grades include perfect coagulation, very good coagulation, good coagulation, relatively good coagulation, appropriate coagulation, general coagulation, slightly poor coagulation, significantly poor coagulation, very poor coagulation, and no coagulation, a total of 10 grades, indicating that the coagulation effect deteriorates in order;

[0014] The coagulation effect parameters include pH value, temperature, turbidity NTU, chemical oxygen demand COD, absorbance at A275, absorbance at A220 and absorbance at A520.

[0015] In one embodiment of the present invention, after obtaining the prediction result of the fusion coagulation effect of the target water quality, the method further includes:

[0016] The coagulant dosage of the target water quality is determined according to the prediction result of the fusion coagulation effect of the target water quality and the pre-established correspondence between the water quality coagulation effect prediction result and the coagulant dosage.

[0017] In one embodiment of the present invention, the training process of the improved deep learning model includes:

[0018] By collecting floc images and sensor acquisition parameters at each moment during a water quality coagulation experiment with multiple water quality samples, multiple groups of sample multimodal data sets are obtained to form a data set; wherein, one sample in the data set is a group of sample multimodal data sets; each group of sample multimodal data sets corresponds to a single water quality coagulation experiment, including a group of sample floc images and sensor acquisition parameters at multiple moments, wherein the group of sample floc images is floc images collected at multiple moments during the water quality coagulation experiment with the same water quality sample, and the acquisition moments of the sensor acquisition parameters at the multiple moments correspond one-to-one to the acquisition moments of the floc images;

[0019] Labeling the samples in the data set; wherein the label data of each sample includes a coagulation effect level and / or a coagulation effect parameter; the coagulation effect parameter used as the label data is a sensor acquisition parameter at the last moment of the water quality coagulation experiment;

[0020] Perform data preprocessing on the dataset after labeling the labeled data, and divide the training set from the dataset after data preprocessing;

[0021] In each iterative training, the samples of the training set are input into the improved deep learning model, and a group of sample floc images in the sample are processed by the first branch to output a first coagulation effect prediction result; the sensor acquisition parameters at multiple moments in the sample are processed by the second branch to output a second coagulation effect prediction result, and the first coagulation effect prediction result and the second coagulation effect prediction result are fused by weighted fusion to obtain a fused coagulation effect prediction result;

[0022] The loss function is calculated using the fused coagulation effect prediction results of the input sample and the corresponding label data, and the model parameters are adjusted by backpropagation; wherein the model parameters include the weight parameters corresponding to the first coagulation effect prediction result and the second coagulation effect prediction result in the weighted fusion method;

[0023] Repeat multiple iterations of training until an improved deep learning model is obtained.

[0024] In one embodiment of the present invention, in each iterative training, samples of the training set are input into the improved deep learning model, a group of sample floc images in the sample are processed using the first branch, and a first coagulation effect prediction result is output, including:

[0025] In each iterative training, the samples of the training set are input into the first branch of the improved deep learning model, and the image features are extracted using the ResNet model to obtain the image features of each floc image in the sample;

[0026] The image features of each floc image in the sample are combined into an image feature time series sequence according to the time sequence of the floc images, and the image features are input into a first LSTM model for time series feature extraction, and a first coagulation effect prediction result corresponding to the sample is output.

[0027] In one embodiment of the present invention, the second branch is used to process the sensor acquisition parameters at multiple moments in the sample and output a second coagulation effect prediction result, including:

[0028] The sensor acquisition parameters at multiple moments in the sample are arranged in time sequence to form a sensor parameter feature time series sequence, which is input into the second LSTM model for time series feature extraction, and the second coagulation effect prediction result corresponding to the sample is output.

[0029] In one embodiment of the present invention, data preprocessing for flocculent images includes:

[0030] Image culling, image cropping, and image enhancement.

[0031] In one embodiment of the present invention, data preprocessing for sensor acquisition parameters includes:

[0032] Data cleaning, handling of missing values ​​and outliers.

[0033] In one embodiment of the present invention, the ResNet model includes a ResNet18 network or a ResNet50 network.

[0034] In one embodiment of the present invention, the method further comprises:

[0035] A training set is divided from the data set after data preprocessing, and a validation set is divided at the same time. The validation set is used to evaluate the model performance after the training is completed.

[0036] Beneficial effects of the present invention:

[0037] The present invention combines the ResNet and LSTM models to construct an improved deep learning model for predicting water quality coagulation effects. This model's advantage lies in its ability to keenly capture the dynamic changes in the water quality coagulation process. Through the deep structure and residual learning mechanism of the ResNet model, key features can be effectively extracted from continuously captured floc images. These features are crucial for understanding subtle changes in the coagulation process. The introduction of the LSTM model enables the capture of temporal dynamics. It can learn temporal patterns in image sequences, thereby better understanding the evolution of the coagulation process. From the floc image data sequence extracted by the ResNet model, the LSTM can identify potential key factors affecting the coagulation effect and predict how these factors affect the coagulation effect over time. Simultaneously, a separate LSTM model can learn temporal patterns in the sensor-acquired parameter sequence, similarly understanding the evolution of the coagulation process. Thanks to the image feature extraction capabilities of ResNet and the temporal analysis capabilities of LSTM, the prediction accuracy of this hybrid model is significantly improved. Furthermore, the model can adapt to new data and changing conditions, demonstrating good generalization and flexibility, which is particularly important for addressing the uncertainties and changes in actual water treatment processes. The experimental results show that compared with the classic LSTM model and other machine learning models, the improved deep learning model can better handle the nonlinearity and non-stationarity of water quality data by processing multimodal data and integrating coagulation effect prediction results, and provide more accurate coagulation effect prediction results.

[0038] Furthermore, the coagulant dosage can be determined based on the water quality and coagulation effect prediction results, enabling precise placement. This will help advance water treatment technology, improve the efficiency and stability of water purification process operations, enhance the level of water purification process operation control, reduce labor costs, and ensure the safety of drinking water quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a physical picture of the existing six-beaker experimental equipment;

[0040] Figure 2 A schematic flow chart of a method for predicting water quality coagulation effect based on multimodal data and deep learning models provided by an embodiment of the present invention;

[0041] Figure 3Schematic diagram of the overall process of the water quality coagulation effect prediction method based on multimodal data and deep learning model according to an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of a single iterative training process of a hybrid deep learning model according to an embodiment of the present invention;

[0043] Figure 5 Schematic diagram of the residual mapping structure;

[0044] Figure 6 This is a schematic diagram of the structure of the ResNet model obtained based on the 34-layer convolutional neural network structure with residual connections in an embodiment of the present invention;

[0045] Figure 7 Schematic diagram of the overall computing framework of the LSTM model according to an embodiment of the present invention;

[0046] Figure 8 Schematic diagram of the internal structure of the ResNet18 model;

[0047] Figure 9 Schematic diagram of the internal structure of basic block 1 in the ResNet18 model;

[0048] Figure 10 Schematic diagram of the internal structure of basic block 2 in the ResNet18 model;

[0049] Figure 11 Schematic diagram of the internal structure of basic block 3 in the ResNet18 model;

[0050] Figure 12 This is a schematic diagram of the internal structure of the ResNet50 model;

[0051] Figure 13 Schematic diagram of the internal structure of bottleneck block 1 in the ResNet50 model;

[0052] Figure 14 Schematic diagram of the internal structure of bottleneck block 2 in the ResNet50 model;

[0053] Figure 15 Schematic diagram of the internal structure of bottleneck block 3 in the ResNet50 model;

[0054] Figure 16 Schematic diagram of the internal structure of bottleneck block 4 in the ResNet50 model. DETAILED DESCRIPTION

[0055] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0056] At present, there are some existing technologies for predicting coagulation effects based on artificial intelligence algorithms. There are mainly two categories. One is to predict turbidity through flocs generated during the coagulation process, and then determine the amount of coagulant added. In the paper "Dosage optimization of polyaluminum chloride by the application of convolutional neural network to the floc images captured in jar tests" by Yamamura et al., the cup coagulation process was photographed, and the CNN neural network algorithm was used to train the model based on the floc images to achieve rapid prediction of the turbidity of the supernatant after sedimentation; in the patent "A method for establishing a coagulation intelligent monitoring linkage system" published by the China Patent Office on November 22, 2022, a camera was used to capture images of alum flocs in the coagulation tank, and combined with the influent parameters to jointly predict the amount of coagulant added and the turbidity of the supernatant after sedimentation at the tail of the horizontal sedimentation tank.

[0057] Another method is to use the historical operation data of the water plant to predict the coagulation effect. In the paper "Coagulant dosage determination using deep learning-based graph attention multivariate time series forecasting model" by Lin et al., a graph attention neural network algorithm is used for training based on the historical data of the water plant to predict the dosage of the incoming water coagulant; in the patent "Coagulation and sedimentation control method, device, equipment and storage medium" published by the China Patent Office on March 14, 2023, the inlet parameters of the water inlet end of the sedimentation container in the preset water purification plant and the preset dosage of the preset coagulant are obtained; according to the inlet parameters and dosage, a pre-trained turbidity time series prediction model is used for processing to obtain the predicted value of the effluent turbidity at the outlet end of the sedimentation container; according to the predicted value of the effluent turbidity, the preset dosage is adjusted to obtain the target dosage; according to the target dosage, the dosing equipment is controlled to add the preset coagulant of the target dosage into the sedimentation container.

[0058] However, in the cup tank coagulation process, a group of continuously taken floc images have a clear time sequence between the photos, while the CNN network predicts the samples as isolated individuals. Therefore, simply using the CNN neural network for turbidity prediction cannot effectively utilize the time sequence information of the floc images. At the same time, simply predicting turbidity cannot achieve the regulation and optimization of the dosage. Using an underwater camera to photograph the alum flocs at the water inlet of the sedimentation tank, the quality of the alum floc photos taken is difficult to guarantee due to the dim light underwater. In addition, the humid environment in the water will also cause great damage to the service life of equipment such as cameras, resulting in more frequent equipment replacement and higher use costs. The model trained using the historical operation data of the water plant can only determine the historical dosage corresponding to the current water quality based on the inlet parameters, but cannot fit a better coagulant dosage relationship, that is, it cannot achieve optimization of the dosage.

[0059] In order to solve the above problems, the embodiment of the present invention provides a method for predicting water quality coagulation effect based on multimodal data and deep learning model. Figure 2 As shown, the method may include the following steps:

[0060] S1, obtaining a set of multimodal data sets of target water quality to be measured, including a set of floc images of the target water quality to be measured collected at multiple times and sensor acquisition parameters at each time; the sensor acquisition parameters include water quality parameters that characterize the coagulation effect;

[0061] In embodiments of the present invention, an image acquisition device can be used to sample target water quality, such as a reservoir or river, at multiple consecutive moments and capture floc images of the water samples. This produces multiple floc images of the target water quality, arranged according to the acquisition moments. Simultaneously, sensor acquisition parameters for the target water quality are also obtained at each acquisition moment.

[0062] After data preprocessing, multiple floc images and sensor acquisition parameters of the target water quality are generated into a multimodal dataset of the target water quality. This dataset is then fed into the pre-trained improved deep learning model as a test sample. For details on the data preprocessing process, see the instructions for the improved deep learning model training process.

[0063] The number of floc images in the set of floc images to be tested is the same as the number of floc images contained in each sample during the training of the improved deep learning model. The sensor acquisition parameters for the target water quality are also consistent in data type and number with the sensor acquisition parameters contained in each sample during the training of the improved deep learning model.

[0064] S2. Input the set of floc images to be tested and the sensor acquisition parameters at each moment into the first branch and the second branch of the pre-trained improved deep learning model respectively, and obtain the first coagulation effect prediction result and the second coagulation effect prediction result accordingly, and fuse the first coagulation effect prediction result and the second coagulation effect prediction result to obtain the fused coagulation effect prediction result of the target water quality; wherein, the first branch includes a ResNet model and a first LSTM model connected in sequence, and the second branch includes a second LSTM model; the improved deep learning model is trained based on multiple groups of sample multimodal data sets of water quality samples; the coagulation effect prediction result includes coagulation effect grade and / or coagulation effect parameters; the coagulation effect grade is used to characterize different levels of coagulation effects, and the coagulation effect parameters include water quality parameters characterizing the final coagulation effect.

[0065] In an optional embodiment, the coagulation effect level includes perfect coagulation, very good coagulation, good coagulation, better coagulation, appropriate coagulation, general coagulation, slightly poor coagulation, significantly poor coagulation, very poor coagulation, and no coagulation, totaling 10 levels, indicating that the coagulation effect becomes worse in sequence, which can be represented by 1 to 10 respectively. Of course, these numbers are only an example of an identification method for a coagulation effect level and do not constitute a limitation to the embodiments of the present invention.

[0066] The coagulation effect parameters include pH value, temperature, nephelometric turbidity unit (NTU), chemical oxygen demand (COD), absorbance at A275, absorbance at A220, and absorbance at A520.

[0067] When measuring total nitrogen by UV spectrophotometry, the uncorrected A220 (absorbance at 220 nm) includes the combined absorbance of nitrate nitrogen, organic matter, turbidity, and other factors. Correction using A275 (A220 - 2 × A275) accurately reflects the nitrate nitrogen content in the water sample. The core function of A520 (absorbance at 520 nm) is to reflect the removal of colored substances (such as organic matter, colloids, and dyes) in water. This is because 520 nm falls within the visible light region and is sensitive to yellow and orange substances (such as humic acid, fulvic acid, iron-manganese oxide colloids, and azo dyes in printing and dyeing wastewater). Unlike turbidity (NTU, scattered light), A520 is a transmitted light absorbance value that directly reflects the concentration of absorbing substances in the solution (in accordance with the Lambert-Beer law).

[0068] For the specific meanings of absorbance at A275, absorbance at A220, and absorbance at A520, please refer to the relevant technical understanding.

[0069] Of course, in the embodiment of the present invention, the water quality parameters characterizing the coagulation effect are not limited to the above parameters.

[0070] In an embodiment of the present invention, the fused coagulation effect prediction result output by the pre-trained improved deep learning model can be the coagulation effect level, the coagulation effect parameter, or both, which corresponds to the type of label data used in the training process of the improved deep learning model.

[0071] The embodiment of the present invention combines the ResNet model and the LSTM model to construct an improved deep learning model, and inputs a set of floc images to be tested of the target water quality into the first branch of the trained improved deep learning model. The image feature extraction of the ResNet model of the first branch and the time series feature extraction of the first LSTM model can obtain the first coagulation effect prediction result of the target water quality. At the same time, the sensor acquisition parameters of the target water quality at each moment are input into the second branch of the trained improved deep learning model, and the time series feature extraction of the second LSTM model of the second branch can obtain the second coagulation effect prediction result of the target water quality. The first coagulation effect prediction result and the second coagulation effect prediction result are then fused to obtain a fused coagulation effect prediction result as the final coagulation effect prediction result. Since the fused coagulation effect prediction result includes the coagulation effect grade and / or coagulation effect parameters, it can be used to evaluate the coagulation effect of the target water quality, thereby determining whether the coagulant dosage is appropriate, and can be used to guide changes in the coagulant dosage.

[0072] Furthermore, after obtaining the prediction result of the fusion coagulation effect of the target water quality, the method further includes:

[0073] The coagulant dosage of the target water quality is determined according to the prediction result of the fusion coagulation effect of the target water quality and the pre-established correspondence between the water quality coagulation effect prediction result and the coagulant dosage.

[0074] The embodiment of the present invention can use a large amount of experimental data in advance to determine the correspondence between the water quality coagulation effect prediction result and the coagulant dosage. The correspondence can be a data table, which lists the corresponding coagulant dosage under different coagulation effect levels and / or coagulation effect parameter values. In this way, after obtaining the fused coagulation effect prediction result of the target water quality, for example, the fused coagulation effect prediction result of the target water quality includes a coagulation effect level of 1 and various values ​​of the coagulation effect parameters, then the coagulant dosage corresponding to the above-mentioned fused coagulation effect prediction result of the target water quality can be found in the data table, thereby guiding the coagulant addition.

[0075] The embodiment of the present invention pre-establishes a correspondence between the water quality coagulation effect prediction results and the coagulant dosage, and determines the coagulant dosage of the target water quality according to the fusion coagulation effect prediction results of the target water quality, so as to accurately quantify the required coagulant dosage, thereby achieving precise delivery.

[0076] To facilitate understanding of the embodiments of the present invention, the training process of the improved deep learning model is described below.

[0077] The training process of the improved deep learning model includes:

[0078] Step A1, by collecting floc images and sensor acquisition parameters at each moment during a water quality coagulation experiment using a variety of water quality samples, to obtain a plurality of groups of sample multimodal data sets to form a data set;

[0079] Among them, one sample in the data set is a set of sample multimodal data sets; each set of sample multimodal data sets corresponds to a single water quality coagulation experiment, including a set of sample floc images and sensor acquisition parameters at multiple moments. The set of sample floc images is floc images collected at multiple moments during the water quality coagulation experiment of the same water quality sample, and the acquisition moments of the sensor acquisition parameters at the multiple moments correspond one-to-one to the acquisition moments of the floc images;

[0080] Specifically, to obtain multiple groups of sample multimodal datasets to constitute a dataset, data collection must be performed first.

[0081] In the water floc experiment simulation, water samples with different water quality conditions can be selected as sampling objects. For each water quality condition, the same water coagulation experiment is repeated many times, so as to fully consider the consistency and repeatability of the water quality coagulation experiment in the actual production process.

[0082] During a water quality coagulation experiment, multimodal data acquisition primarily consists of floc images and sensor parameters. First, during image collection, high-resolution industrial cameras and other image acquisition equipment can be used to capture image samples of the water quality coagulation experiment. Since water coagulation typically takes 30 to 40 minutes, the single sampling time for coagulation experiment images is limited to 50 minutes. Furthermore, the sampling interval between adjacent floc images minimizes the need for repeated floc feature acquisition, capturing floc characteristics at different stages of the coagulation experiment. Second, during this coagulation experiment, the hash probe records water quality parameters, including pH, temperature, turbidity, COD, A275, A220, and A520, in real time. The acquisition time corresponds to the image acquisition time. This completes a round of multimodal data acquisition, resulting in a sample multimodal dataset.

[0083] After completing each round of multimodal data collection, the specific information of each sampling session, including time, location, operator, and environmental conditions, is properly recorded. This ultimately forms a multimodal dataset of multiple samples, forming a dataset. The dataset contains multiple folders, each representing a sample. This corresponds to a multimodal dataset of samples from a single water quality coagulation experiment. A folder contains a set of floc images, arranged in the order of image acquisition time, along with sensor acquisition parameters for multiple times.

[0084] Step A2, labeling the samples in the dataset;

[0085] Data annotation is a key step after multimodal data collection in water quality coagulation experiments. It involves the classification, identification, and labeling of floc collection images and sensor collection parameters in order to improve deep learning models to learn from these data and make accurate predictions.

[0086] In the above dataset, each folder contains multiple time-series floc images and a table of sensor acquisition parameters that records the changing state of the coagulation experiment, thus fully reflecting the entire water quality coagulation experiment process. However, the final experimental results are not accurately and completely recorded. To address this issue, first, a set of detailed annotation guidelines needs to be developed, including different floc classification standards, feature descriptions, and identification and marking methods for floc characteristics. It should also include the specific meaning of sensor acquisition parameters and their corresponding water quality status. This helps ensure that all annotators can annotate in a consistent manner. Secondly, according to the detailed annotation guidelines, the final results of each water quality coagulation experiment are classified and annotated or parameterized to form an accurate and complete label file, so as to achieve accurate predictions for the improved deep learning model.

[0087] The label data of each sample includes the coagulation effect level and / or coagulation effect parameters; the coagulation effect parameters used as label data are the sensor collection parameters at the last moment of the water quality coagulation experiment;

[0088] For each sample, the final water quality coagulation test results can be annotated with different coagulation effect levels, that is, label data of the labeled samples, for example, using 1 to 10 to represent perfect coagulation, very good coagulation, good coagulation, relatively good coagulation, appropriate coagulation, general coagulation, slightly poor coagulation, significantly poor coagulation, very poor coagulation, and no coagulation at all, indicating that the coagulation effect deteriorates in sequence.

[0089] The sensor collected parameters at the final moment of the coagulation experiment may also be used as coagulation effect parameters and marked as label data. Some or all of the sensor collected parameters at the final moment may be selected as label data.

[0090] The label data can be set as coagulation effect level, coagulation effect parameter, or both as needed.

[0091] Step A3: preprocess the data set after labeling the labeled data, and divide the training set from the preprocessed data set;

[0092] The data preprocessing process is crucial to improving the performance of deep learning models.

[0093] In an optional embodiment, data preprocessing for the floc image includes:

[0094] Image culling, image cropping, and image enhancement.

[0095] In an optional embodiment, data preprocessing for sensor acquisition parameters includes:

[0096] Data cleaning, handling of missing values ​​and outliers.

[0097] For the floc images in the dataset:

[0098] First, image culling is performed. Specifically, multiple floc images in each folder of the dataset are selected, and floc images with low acquisition quality are deleted. The images in the folder are then uniformly reduced to the same number of floc images, thus ensuring the same number of images in each folder.

[0099] Secondly, image cropping is performed, specifically cropping all floc images in the dataset to a uniform image pixel size to meet the input requirements of the improved deep learning model;

[0100] Again, image enhancement is performed. In order to enhance image features, a series of image enhancement operations can be performed, including adjusting brightness and contrast, applying histogram equalization and CLAHE (contrast limited adaptive histogram equalization) technology, and using nonlinear methods such as gamma transform to improve the visual effect of the image.

[0101] These processes together improve image quality, enhance floc characteristics, and provide the model with more accurate and rich information, enabling effective learning and prediction.

[0102] Collect parameters for the sensors in the dataset:

[0103] The main task is to clean the data and handle missing values ​​and outliers. For missing values, statistical methods (such as mean filling) or more advanced interpolation methods (such as KNN filling or MICE) are used. For outliers, Z-Score or IQR methods are used for detection and handling.

[0104] Before entering the improved deep learning model, multimodal data also needs to undergo important data preprocessing steps such as standardization or normalization (for ease of understanding, the following Figure 3 The specific operation is as follows:

[0105] First, in multimodal data, the standardization of water floc image data is responsible for converting water floc images into standard normal distribution data, which is a key step to ensure that the model can learn effectively. Generally, standardization processing can speed up training, improve the generalization ability of the model, and reduce numerical instability during training. It mainly consists of pixel value scaling, mean and Z-score standardization. First, pixel value scaling is to scale the pixel values ​​of the image from the range of 0 to 255 to between 0 and 1, which is achieved by dividing each pixel value by 255. For each pixel value p, the image scaling value is Secondly, Z-score normalization is achieved by subtracting the mean and dividing by the standard deviation. Let μ represent the mean of the pixel values ​​in an image and σ represent the standard deviation. Then the normalized pixel value is Through these standardized steps, the improved deep learning model can more effectively process water floc image data, improving the performance and accuracy of the model.

[0106] Second, in multimodal data, the normalization of sensor acquisition parameter data is responsible for scaling the unknown distribution of acquisition parameters to a fixed range, minimizing the impact of measurement error on the characteristics of time series data. Usually, the data set of acquisition parameters mainly uses the Min-Max method to achieve the normalization of specific data. Let x be all the original sequence data of a water quality parameter in a coagulation experiment, min(x) and max(x) are the maximum and minimum values ​​of the sequence data respectively, then the acquisition parameter data after normalization is By using the above normalization processing, the improved deep learning model can effectively eliminate the dimensional effect, thereby deeply exploring the changing laws of real-time water quality parameters during the coagulation experiment.

[0107] Furthermore, for step A3, a training set is divided from the data set after data preprocessing, and a validation set is also divided. The validation set is used to evaluate the model performance after the training is completed.

[0108] The ratio of the training set to the validation set can be 4:1, which can be set according to needs.

[0109] In the embodiment of the present invention, as an experimental simulation, a set of multimodal data sets of target water quality to be tested in step S1 can be obtained from the validation set. Figure 3 Understand the overall process of the water quality coagulation effect prediction method based on multimodal data and deep learning model in the embodiment of the present invention. Among them, the floc image to be measured and the sensor acquisition parameters are respectively adopted by the first LSTM model and the second LSTM model. Figure 3 No distinction is made in the representation.

[0110] Step A4: In each iterative training, the samples of the training set are input into the improved deep learning model, a group of sample floc images in the sample are processed using the first branch, and a first coagulation effect prediction result is output; the sensor acquisition parameters of multiple moments in the sample are processed using the second branch, and a second coagulation effect prediction result is output; the first coagulation effect prediction result and the second coagulation effect prediction result are fused by weighted fusion to obtain a fused coagulation effect prediction result;

[0111] It can be understood that after completing the above steps, each folder in the training set and the validation set contains standard format data of a round of water quality coagulation experiment, which includes multiple floc images at consecutive moments with the same data format and sensor acquisition parameters at the same moment.

[0112] In each iterative training, the samples of the training set are input into the improved deep learning model, and a group of sample floc images in the sample are processed using the first branch to output a first coagulation effect prediction result, including:

[0113] 1) In each iterative training, the samples of the training set are input into the first branch of the improved deep learning model, and the image features are extracted using the ResNet model to obtain the image features of each floc image in the sample;

[0114] 2) The image features of each floc image in the sample are combined into an image feature time series sequence according to the time sequence of the floc images, and the image features are input into a first LSTM model for time series feature extraction, and a first coagulation effect prediction result corresponding to the sample is output.

[0115] Specifically, for a sample corresponding to a single round of water quality coagulation experiment process in the training set, all water quality floc images need to be input into the ResNet model separately to realize the extraction of water quality floc image features at each moment and obtain the tensor sequence obtained from continuous coagulation images; secondly, according to the time sequence of image acquisition, these obtained tensor sequences are used as image feature time series sequences and input into the independent first LSTM model in sequence to realize the time series feature extraction of floc images.

[0116] In each iterative training, the second branch is used to process the sensor acquisition parameters at multiple moments in the sample, and output the second coagulation effect prediction result, including:

[0117] The sensor acquisition parameters at multiple moments in the sample are arranged in time sequence to form a sensor parameter feature time series sequence, which is input into the second LSTM model for time series feature extraction, and the second coagulation effect prediction result corresponding to the sample is output.

[0118] Specifically, for a sample corresponding to a single-round water quality coagulation experiment process in the training set, the sensor acquisition parameter data needs to be input into the independent second LSTM model in sequence according to the time sequence of its acquisition, so as to realize the extraction of multiple time series features of water quality acquisition parameters.

[0119] See Figure 4 Understand the process of improving a deep learning model's training iteration.

[0120] At present, using programming tool platforms such as the PyTorch library or the Tensorflow library, the present invention can select the ResNet model and the LSTM model with the ability to extract local image features as the feature extraction model of multimodal data according to the specific characteristics of water quality coagulation multimodal data. The detailed description is as follows:

[0121] For example, the ResNet model may include a ResNet18 network or a ResNet50 network. The ResNet model is used to extract features from flocculent images. These image features contain key visual information in the image and are crucial for understanding microscopic changes in the coagulation process.

[0122] ResNet, short for Residual Network, is a deep learning architecture designed to address the degradation problem in deep neural network training. This degradation problem occurs when the number of network layers increases, leading to a decrease in model performance. ResNet effectively addresses this degradation issue in traditional neural network training by introducing a "residual learning" framework.

[0123] Typically, the goal of a traditional neural network is to use one or more layers of the network to learn a mapping function H(x) that maps the input x to the desired output. However, as the number of network layers increases, the learning process of the mapping function H(x) becomes very difficult. To address this problem, the ResNet model proposes that instead of letting the network learn H(x), it is better to let the network learn the residual or difference between the input and output, that is, F(x) = H(x) - x. In this way, the original mapping H(x) can be expressed as F(x) + x ，Where F(x) is the residual mapping, the network structure is as follows Figure 5 shown.

[0124] Using multiple convolutional layers and a residual link, the ResNet model can construct a basic residual unit, namely a residual block. At this time, let the input and output of the residual block be x and y respectively, and the function F(x, {W i}) represents the residual mapping that needs to be learned, W i represents the weight of the i-th convolutional layer. For example, Figure 5 There are two layers of residual mapping in the function F(x, {W i}) is expressed as F = W2γ(W1x), where γ represents the ReLU function, and W1 and W2 represent the weights of the two convolutional layers. In addition, in the specific calculation process, the function F(x, {W i The output of}) must maintain the same dimension as the input x in order to output the final residual learning result. At this point, it is also necessary to add a linear projection operation W to the input x s , to match the dimensions of the function F and the input x. In summary, similar to Figure 5 The calculation formula of the residual mapping structure in is:

[0125] y=F(x,{W i})+W s x (1);

[0126] According to the principle of the above residual mapping structure, the traditional neural network can be deeply transformed to form a convolutional neural network with strong feature extraction capabilities. For example, a 34-layer convolutional neural network structure based on residual connections can be used as a ResNet model. Figure 6 shown.

[0127] Figure 6 In the example, 7×7conv,64, / 2 means the convolution kernel size is 7×7. ， The number of channels is 64, / 2 means that the convolution layer uses a convolution operation with a stride of 2; 3×3conv,64 means that the convolution kernel size is 3×3 ， The number of channels is 64. The parameters of the remaining layers will not be explained one by one.

[0128] After extracting image features from a set of floc images using the ResNet model, the improved deep learning model requires inputting these tensor sequences into the first LSTM model in chronological order, thereby extracting temporal features of the flocs during the coagulation process. Simultaneously, the normalized tensor sequences of the time series parameter data collected by the sensor can be directly input into the second LSTM model in chronological order. It is important to note that the two independent LSTM models have the same structure.

[0129] The LSTM model can be implemented using an existing LSTM network. The LSTM gating mechanism allows the model to learn long-term dependencies, which is important for predicting coagulation effects.

[0130] The Long Short-Term Memory (LSTM) network is a special type of recurrent neural network (RNN) designed specifically to address long-term dependencies. The LSTM incorporates three gates and a memory unit to control the flow of information, enabling it to retain information over long periods of time in a time series. This structure enables the LSTM to capture complex and long-term dependencies within a sequence, resulting in its outstanding performance in fields such as natural language processing, speech recognition, and time series forecasting.

[0131] The overall computing framework of the LSTM network is as follows Figure 7 As shown in Figure 2, the LSTM network consists of components such as a forget gate, an input gate, and an output gate. Through the collaborative work of these three gating components, the LSTM model determines the flow and update of floc image feature information within the network. The core of the LSTM is the cell state, a continuous value that carries temporal sequence information throughout the LSTM network. The hidden state, a snapshot of the current state, is passed to the next moment. The LSTM design allows gradients to flow effectively across time, alleviating the problem of vanishing or exploding gradients and enabling the network to learn long-term dependencies in sequential data. The three gating mechanisms are described below:

[0132] First, the forget gate is responsible for selectively discarding old information. t-1 Indicates the long-term state information of the previous moment, h t-1 is the hidden state information of the previous moment, x t is the input information at the current moment, [] represents splicing calculation, b f is the bias matrix of the forget gate, W f is the weight matrix of the forget gate, δ is the sigmoid function, that is The calculation formula of the forget gate is as follows:

[0133] ft =δ(W f ·[h t-1 , x t ]+b f ) (2);

[0134] Among them, f t is the output of the forget gate.

[0135] It should be noted that, first, the output of the forget gate is a real number vector between 0 and 1, and each element of this vector represents the degree of retention of the element in the corresponding cell state. If the output is 0, it means that the corresponding information is completely forgotten; if the output is 1, it means that the corresponding information is completely retained; secondly, the main function of the forget gate is to control the degree to which the historical state information flows to the current state, which determines the cell state C at the previous moment. t-1 How much of the cell state at the current moment is involved. If the forget gate is fully closed (value 0), the history has no effect on the current state; if the forget gate is fully open (value 1), the historical information is transmitted to the current moment intact without any information loss; again, when updating the cell state, the forget gate will be multiplied by the cell state to determine how much historical information to retain. Specifically, the cell state C at the previous moment t-1 Will be combined with the forget gate output f t The result of element-wise multiplication is combined with the new information to form the new cell state.

[0136] Second, the input gate is responsible for determining the storage of new information. Let C t Indicates the long-term status information at the current moment, i t is the output of the input gate, δ is the sigmoid function, W i and b i are the weight matrix and bias matrix of the input gate, h t-1 is the hidden state information of the previous moment, x t is the input information at the current moment, C' t is the candidate memory unit at the current moment, W C and b C are the weight matrix and bias matrix of the candidate memory unit, respectively, and tanh represents the hyperbolic tangent function, that is, The specific calculation process of the input gate is as follows:

[0137] i t =δ(W i ·[h t-1 ,x t ]+b i ) (3);

[0138] C′ t =tanh(WC ·[h t-1 ,x t ]+b C ) (4);

[0139] C t =f t ·C t-1 +i t C′ t (5);

[0140] It should be noted that the input gate determines which parts of the memory cell should be updated at the current moment. The output of the input gate is a real number vector between 0 and 1, and each element of this vector represents the degree of update of the element corresponding to the candidate memory cell. If the output of the input gate is 0, it means that the corresponding information should not be added to the memory cell; if the output is 1, it means that the corresponding information should be completely added to the memory cell. In addition, in the input gate, the update of the memory cell depends not only on the input gate, but also on the forget gate. The final state C of the memory cell t It is calculated by combining the output of the forget gate and the output of the input gate.

[0141] Third, the output gate determines the final output state. ot is the output of the output gate, δ is the sigmoid function, and W o and b o are the weight matrix and bias matrix of the output gate, h t-1 is the hidden state information of the previous moment, xt is the input information of the current moment, C t is the final state of the memory unit, then the calculation process of the output gate is as follows:

[0142] ot=δ(W o ·[ht -1 ,xt]+b o ) (6);

[0143]

[0144] in, Represents a multiplication operation.

[0145] It should be noted that the output of the output gate is also a real number vector between 0 and 1, and each element of this vector represents the degree of output of the corresponding element in the cell state. If the output is 0, it means that the corresponding information should not be output; if the output is 1, it means that the corresponding information should be output completely. In addition, the output gate allows the LSTM model to selectively output information at each moment, which helps the model capture and remember long-term dependencies in sequence data while ignoring unimportant information. In this way, LSTM can process time series data more efficiently, especially in scenarios where long-distance dependent information needs to be remembered.

[0146] Through the above training process, a trained improved deep learning model can be obtained, and the validation set can be used to evaluate the model performance.

[0147] Specifically, for each group of multimodal data sets in the validation set, the floc images are firstly extracted using the trained ResNet model to obtain the image feature time series sequence, and then the trained first LSTM model is used to predict the image feature time series sequence to obtain the first coagulation effect prediction result. At the same time, the trained second LSTM model is used to predict the sensor acquisition parameters in this group of multimodal data sets to obtain the second coagulation effect prediction result. Then, the two are weightedly fused using the weight parameter to obtain the fused coagulation effect prediction result.

[0148] The fused coagulation effect prediction results are then fully compared and verified with the annotated labeled data, generating statistical metrics such as accuracy, MSE (Mean Squared Error), and MAE (Mean Absolute Error). If the performance indicators meet the requirements, a hybrid deep learning model that meets the requirements for use is obtained. If not, further training and adjustments can be performed to ensure the model's reliability in predicting coagulation effects.

[0149] In summary, the advantage of the ResNet-LSTM model in predicting water quality coagulation effects lies in its ability to discern the dynamic changes in the coagulation process from both floc image data and sensor water quality parameters. Specifically, first, from the floc image data sequence extracted by the ResNet model, the LSTM model can identify potential key factors influencing the coagulation effect and predict how these factors affect the final coagulation effect over time. Second, from the continuous sequence of sensor water quality parameters, the LSTM model can more accurately extract the changing process of the coagulation process, thereby enabling state prediction of the water quality coagulation effect. By learning the state transformations of two modal data, this multimodal learning model can adapt to coagulation effect prediction for a variety of complex raw water quality conditions, demonstrating superior generalization and flexibility, which is particularly important for coping with the uncertainty and variability in actual water treatment processes. Experimental results demonstrate that compared to classic LSTM models and other machine learning models, the ResNet-LSTM model based on multimodal data can better handle the nonlinearity and nonstationarity of water quality data, providing more accurate prediction results.

[0150] Two learnable weight parameters are added to the outputs of the two independent LSTM models, enabling the fusion of water quality acquisition parameters and floc images for coagulation prediction. It is important to note that these weight parameters dynamically adjust the importance of the two features, thereby better reflecting the contribution of different modal data to the water quality coagulation process. Assuming that the two weight parameters are λ1 and λ2, respectively, and that λ1 + λ2 = 1, and the prediction outputs of the two independent LSTM models are R1 and R2, respectively, the final prediction of the improved deep learning model, i.e., the fused coagulation effect prediction, is R = λ1R1 + λ2R2.

[0151] Step A5: Calculate the value of the loss function using the fused coagulation effect prediction result of the input sample and the corresponding label data, and perform backpropagation to adjust the model parameters;

[0152] According to the learned complex patterns and rules, the distance between the fused coagulation effect prediction results and the label data is continuously reduced, thereby completing the model parameter adjustment. The model parameters include the weight parameters corresponding to the first coagulation effect prediction result and the second coagulation effect prediction result in the weighted fusion method;

[0153] The loss function in the training process can be implemented using an existing loss function. The training process can be understood by referring to the conventional neural network training process.

[0154] Step A6: Repeat the iterative training multiple times until a trained improved deep learning model is obtained.

[0155] In view of the fact that ResNet has a strong ability in extracting spatial features of images, and LSTM has a strong advantage in processing time series data, the present invention combines the two in an embodiment of the present invention to construct an improved deep learning model. This improved deep learning model can not only extract key time series information from multimodal data at continuous moments, but also predict the coagulation effect of water quality with high accuracy, providing strong technical support for water quality management and optimization of coagulation process. Through this innovative method, in the coagulation process of water treatment, by combining deep learning, machine vision and automatic control strategies, researchers and engineers can more effectively monitor and control the coagulation stage of the water treatment process, realize intelligent optimization of coagulant dosage, thereby improving the coagulation process effect of water quality and protecting public health and environmental safety.

[0156] Specifically, the method of the present invention has the following beneficial effects:

[0157] 1. Improve prediction accuracy: By using deep learning algorithms to build an improved deep learning model, the model can simultaneously extract the spatial features and time series features of the image, achieving more efficient and accurate analysis and prediction.

[0158] 2. Consider the influence of multiple factors: The model input variables include not only the direct influencing factors of the coagulant dosage, such as influent turbidity, water temperature, etc., but also include factors affecting turbidity removal such as the floc formation time and floc size, such as influent pH, etc., so as to more realistically reflect the influence of the dosage of the coagulant on the floc precipitation effect and the turbidity of the effluent after sedimentation.

[0159] 3. Consideration of time series characteristics: The value of a single influencing factor uses the sequence value within a period of time as the input variable of the model, which can more realistically reflect the entire impact process of coagulant dosage on complete coagulation, aggregation, sedimentation, floc formation, floc particle size change and floc removal.

[0160] 4. Optimize coagulant dosage: By obtaining accurate coagulation effect prediction results, the coagulant dosage can be further accurately predicted, which is of great significance for optimizing coagulant dosage, stabilizing the turbidity of the effluent after sedimentation treatment, and ensuring the safety of water quality.

[0161] 5. Improved intelligence and automation: This technology collects and analyzes key water quality indicators and treatment process parameters in real time, and uses deep learning technology to intelligently optimize and manage the treatment process, significantly improving the efficiency and stability of the water treatment process while reducing reliance on manual operations.

[0162] 6. Adapting to environmental changes: Through the implementation of the method of the present invention, the optimal operation of water treatment facilities can be guaranteed under various environmental conditions, ensuring the continuity, quality and safety of water supply.

[0163] 7. Reduce the deviation caused by single-modal data: Single-modal image data may not be able to fully reflect the complex physical and chemical changes in the coagulation process. The present invention effectively compensates for the limitations of single-modal data by fusing image data and sensor data. Among them, sensor data provides real-time monitoring information of key parameters such as water flow, turbidity, pH value, and water temperature. By fusing these multimodal data, the model can more comprehensively understand the dynamic changes of the coagulation process, thereby reducing the prediction deviation caused by insufficient single-modal data and further improving the accuracy and reliability of the prediction results.

[0164] To facilitate understanding of the embodiments of the present invention, two specific embodiments are given below.

[0165] (1) Example 1

[0166] Data preparation: A multimodal dataset consisting of multiple sets of continuous coagulation floc images and sensor acquisition parameters was constructed, and all datasets were divided into training and validation sets in a 4:1 ratio. Regardless of the training set or validation set, each sample includes 125 continuously captured floc images (size 3*224*224) and the sensor acquisition parameters at the same time. The sample label data (the coagulation effect prediction results are consistent with it) is the coagulation effect level, including: perfect coagulation (labeled as 1), very good coagulation (labeled as 2), good coagulation (labeled as 3), good coagulation (labeled as 4), appropriate coagulation (labeled as 5), general coagulation (labeled as 6), slightly poor coagulation (labeled as 7), significantly poor coagulation (labeled as 8), very poor coagulation (labeled as 9), and no coagulation (labeled as 10).

[0167] Data preprocessing: Using PyTorch's transforms module, the floc images and sensor acquisition parameters are converted into tensors and then standardized and normalized. The standardization parameters (mean and standard deviation) are set based on the parameters of the pre-trained model, while the specific normalization parameters are determined by the acquisition parameter data itself.

[0168] Model architecture: The ResNet model uses the ResNet18 network, which is a relatively shallow network in the ResNet series, with a total of 18 layers of depth. It is mainly composed of multiple residual blocks, which are suitable for networks with fewer layers. The structure of ResNet18 starts with a 7×7 convolutional layer, followed by a maximum pooling layer, followed by four residual block layers, and finally a global average pooling layer and a fully connected layer. In ResNet18, the first to fourth layers each contain 2 different residual blocks. The internal structure of the ResNet18 network is as follows Figure 8 shown.

[0169] In the ResNet18 model, the input layer consists of a convolution layer and a normalization layer. First, the convolution layer uses a 7×7 convolution kernel with a stride of 2 to convolve the input image. The number of output channels of this convolution layer is 64, which means it will produce 64 feature maps. The convolution operation can capture local features of the input data, such as edges, textures, etc., and recognize these features by learning different convolution kernels. Let I represent the input image, K is the 7×7 convolution kernel, b is the bias term, * represents the convolution operation, and O is the output feature map. The calculation formula of this layer is:

[0170] (I*K)+b=O (8);

[0171] Second, the normalization layer, also known as the Batch Normalization (BN) layer, is used to normalize the output of the convolutional layer. Batch normalization calculates the mean and variance of each small batch of data and normalizes the data to make the output distribution more stable, making the mean of each small batch of data 0 and the variance 1, thereby speeding up the training process and improving the generalization ability of the model. Let x represent the output of the convolutional layer, μ is the mean of the small batch of data, σ is 2 is the variance of the mini-batch data, ε is a small number used to prevent division by zero, is the normalized output, then the calculation formula of this layer is:

[0172]

[0173] The feature map after the input layer needs to enter the calculation of the first layer, which is mainly composed of the maximum pooling layer and two basic blocks 1. Among them, the main function of the maximum pooling layer is to downsample and reduce the size of the feature map, thereby reducing the number of parameters and the amount of calculation, while maintaining the main features of the feature map. The maximum pooling operation does not involve the learning of weights. It divides the feature map and then selects the maximum value in each divided area as the representative of the area, thereby achieving dimensionality reduction and abstraction of features. Specifically, the window size of the maximum pooling layer is 2×2, the stride is 2, let P(i,j) represent the value of the output feature map at position (i,j), I in is the input feature map (size H×W), m and n are the indices within the pooling window, and stride is the step size, then the corresponding formula is:

[0174] P(ij)=max m,n I in (i·stride+m,j·stride+n) (10);

[0175] After the maximum pooling layer, the size of the output feature map becomes H / stride×W / stride, and then the output feature map is added to the structure of the two basic blocks 1. It should be noted that the structures of the two basic blocks 1 are the same. The specific structures are as follows Figure 9 shown.

[0176] In basic block 1, let the input feature map be X ， First, a convolution operation with a kernel size of 3×3 and a stride of 1 is performed, and the calculation formula is the same as formula (1); then, the batch normalization layer is entered, and the same calculation as formula (2) is performed; then, the ReLU layer is entered, and nonlinear calculation is introduced. Let x be the input element of the ReLU layer, and the calculation formula of this layer is:

[0177] Relu(x)=max(0,x) (11);

[0178] Again, repeat the convolution with a kernel size of 3×3 and a stride of 1 and the batch normalization layer calculation to obtain the output feature map extracted by basic block 1, and then superimpose it with the original input feature map of basic block 1 to realize the residual connection of the feature map, thereby alleviating the gradient disappearance, promoting information flow, and further improving training efficiency; finally, perform the ReLU layer calculation and output the corresponding feature map.

[0179] In the ResNet18 model, layers 2-4 perform the same computational process and are composed of a basic block 2 and a basic block 3, which are described in detail below. First, the structure of basic block 2 is very similar to that of basic block 1. The difference between the two is that the number of output channels of the first convolution of basic block 2 is 128, while the number of input channels and output channels of the second convolution are both 128. In addition, the residual connection part of basic block 2 adds a 1×1 convolution layer with a stride of 2, 64 input channels, and 128 output channels. Its specific structure is as follows: Figure 10 As shown; secondly, the structure of basic block 3 is very similar to that of basic block 1. The difference between the two is that the number of input and output channels of the first convolution of basic block 3 is 128, and the number of input and output channels of the second convolution is 128. The specific structure is as follows Figure 11 shown.

[0180] On the basis of the ResNet18 model, two independent classic LSTM models need to be added to process the coagulation floc feature map sequence and the sensor acquisition parameter sequence respectively, so as to realize the temporal feature extraction of multimodal data in a single round of coagulation experiment.

[0181] Model training: For 80% of the training set data, features are extracted from the image tensor data through the convolution layer, batch normalization layer, activation function, pooling layer, maximum pooling layer, etc. of ResNet18 in training mode, and the final result is flattened into one-dimensional vector data. All one-dimensional vector data are then input into the first LSTM model. At the same time, the normalized sensor acquisition parameter tensor data is input into the second LSTM model, and the forget gate, input gate, and output gate are used to extract the temporal characteristics of continuous multimodal data. During this process, the improved deep learning model uses the Adam optimizer and the mean square error loss function to update the various parameters in the model until the overall model reaches the predetermined number of iterations or the loss function value is lower than the threshold of 10 -5 .

[0182] Model evaluation: For 20% of the validation set data, features are extracted from the image tensor data through the convolutional layer, batch normalization layer, activation function, pooling layer, maximum pooling layer, etc. of ResNet18 in verification mode, and the final result is flattened into one-dimensional vector data. All one-dimensional vector data are then input into the first LSTM model in verification mode. At the same time, the normalized sensor acquisition parameter tensor data is input into the second LSTM model in verification mode. The forget gate, input gate, and output gate are used to extract the temporal characteristics of the continuous multimodal data. Then, the difference between the model prediction results and the dataset labels is used to calculate the accuracy, mean square error (MSE), and mean absolute error (MAE) to comprehensively evaluate the accuracy and reliability of the model.

[0183] (II) Example 2

[0184] Data Preparation: A multimodal dataset consisting of multiple sets of continuous coagulation floc images and sensor acquisition parameters was constructed and divided into training and validation sets in a 4:1 ratio. Each sample, regardless of training or validation, consisted of 125 continuously captured floc images (3x224x224 pixels) and the corresponding sensor acquisition parameters. The sample's label data consisted of coagulation effect parameters (tensor data sequences consisting of pH, temperature, turbidity (NTU), COD, A275, A220, and A520).

[0185] Data preprocessing: Using PyTorch's transforms module, the floc images and sensor acquisition parameters are converted into tensors and then standardized and normalized. The standardization parameters (mean and standard deviation) are set based on the parameters of the pre-trained model, while the specific normalization parameters are determined by the acquisition parameter data itself.

[0186] Model architecture: The ResNet model uses the ResNet50 network, which is a deeper network with 50 layers. It begins to use a variety of bottleneck blocks, which are suitable for networks with more layers and can reduce the amount of computation. The structure of ResNet50 includes an initial 7×7 convolution layer, followed by a maximum pooling layer, then four residual block layers, and finally a global average pooling layer and a fully connected layer. The layer configuration of ResNet50 is: the first layer has 3 bottleneck blocks, the second layer has 4 bottleneck blocks, the third layer has 6 bottleneck blocks, and the fourth layer has 3 bottleneck blocks. Its specific structure is as follows: Figure 12 shown.

[0187] In the ResNet50 model, the input layer is the same as that of the ResNet18 model, so I will not go into details here. The first layer of the ResNet50 model consists of a maximum pooling layer, a bottleneck block 1, and two bottleneck blocks 2. The specific description is as follows: First, the maximum pooling layer structure is the same as the maximum pooling layer structure of the ResNet18 model, so I will not go into details here; Second, the bottleneck block 1 mainly contains 1×1 convolution calculations and 3×3 convolution calculations. The specific calculation process is as follows Figure 13 shown.

[0188] It should be noted that, except for the different parameters of the convolution kernel size, stride, input channels, and output channels, the convolution operation, batch normalization layer, and ReLU layer in bottleneck block 1 are calculated in the same way as those in basic block 1 in the ResNet18 model, so they will not be repeated here. Third, bottleneck block 2 also has a very similar structure to bottleneck block 1, and its detailed description is shown in Figure 14 .

[0189] In the ResNet50 model, the second layer mainly consists of 1 bottleneck block 3 and 3 bottleneck blocks 4, the third layer consists of 1 bottleneck block 3 and 5 bottleneck blocks 4, and the fourth layer consists of 1 bottleneck block 3 and 2 bottleneck blocks 4. The internal structures of bottleneck block 3 and bottleneck block 4 are as follows: Figure 15 and Figure 16 shown.

[0190] As can be seen from the above figures, the internal structure of bottleneck block 3 is very similar to that of bottleneck block 1. The only difference between the two is the setting of parameters such as the step size of the convolution operation, input channels, and output channels. The operation process is otherwise the same. The internal structure of bottleneck block 4 is very similar to that of bottleneck block 2. The only difference between the two is the setting of parameters such as the step size of the convolution operation, input channels, and output channels.

[0191] On the basis of the ResNet50 model, two independent classic LSTM models need to be added to process the coagulation floc feature map sequence and the sensor acquisition parameter sequence respectively, so as to realize the temporal feature extraction of multimodal data in a single round of coagulation experiment.

[0192] Model training: For 80% of the training set data, features are extracted from the image tensor data through the convolution layer, batch normalization layer, activation function, pooling layer, maximum pooling layer, etc. of the ResNet50 model in training mode, and the final result is flattened into one-dimensional vector data. All one-dimensional vector data are then input into the first LSTM model in training mode. At the same time, the normalized sensor acquisition parameter tensor data is input into the second LSTM model in training mode. The forget gate, input gate, and output gate are used to extract the temporal characteristics of the multimodal data at continuous moments. In this process, the improved deep learning model uses the Adam optimizer and the mean square error loss function to update the various parameters in the model until the improved deep learning model reaches the predetermined number of iterations or the loss function value is lower than the threshold of 10 -5 .

[0193] Model evaluation: For 20% of the training set data, features are extracted from the image tensor data through the convolutional layer, batch normalization layer, activation function, pooling layer, maximum pooling layer, etc. of the ResNet50 model in verification mode. The final result is flattened into one-dimensional vector data, and then all one-dimensional vector data is input into the first LSTM model in verification mode. At the same time, the normalized sensor acquisition parameter tensor data is input into the second LSTM model in verification mode. The forget gate, input gate, and output gate are used to extract the temporal characteristics of the multimodal data at continuous moments. Then, the difference between the model prediction results and the dataset labels is used to calculate the accuracy, mean square error (MSE), and mean absolute error (MAE) to comprehensively evaluate the accuracy and reliability of the model.

[0194] In the specific experimental simulation process, the cup coagulation experiment prediction method is used to quickly and accurately predict the turbidity after cup coagulation.

[0195] The present invention combines the water quality coagulation effect prediction technology of the ResNet and LSTM models, and through the deep learning method, brings significant benefits to the field of water quality management. First, through the deep structure and residual learning mechanism of the ResNet model, this technology effectively extracts key features from continuously captured floc images. These features are crucial for understanding the subtle changes of flocs during the coagulation process; secondly, based on the continuous floc image data, by introducing the real-time acquisition parameters of the sensor, the overall water quality changes during the coagulation process are obtained. Thirdly, on this basis, two independent LSTM models are used to extract the temporal characteristics of the water quality coagulation experiment from the image data and sensor data respectively, so as to comprehensively learn the temporal patterns in the water quality coagulation multimodal data sequence and better understand the evolution of the water quality coagulation process.

[0196] Compared to deep learning models based on unimodal data, the improved deep learning model based on multimodal data significantly improves both prediction accuracy and robustness. This is primarily due to the image feature extraction capabilities of ResNet, the time series analysis capabilities of LSTM, and the flexible adjustment of learnable weight parameters. By splitting the dataset into training and validation sets in a 4:1 ratio, the model's generalization ability is enhanced, enabling it to make accurate predictions even on unseen data. Furthermore, this technology can be integrated into water quality monitoring systems to enable real-time monitoring of the coagulation process and provide decision support for water treatment operations, enabling timely adjustments to coagulant dosage.

[0197] Increased automation reduces reliance on manual monitoring and evaluation, lowering labor costs. Furthermore, by optimizing the coagulation process, the use of coagulants can be reduced, achieving a more environmentally friendly water treatment method. Data-driven optimization methods allow the model to continuously improve through the collection of new data, achieving continuous optimization of the coagulation process. The model's flexibility also enables parameter adjustments based on specific application scenarios to accommodate different prediction tasks.

[0198] This technology is not only of great value in scientific research and industrial applications, but also has broad application prospects in industrial water treatment, helping to promote the advancement of water treatment technology. In summary, the ResNet-LSTM-based water coagulation effect prediction technology provides an efficient, accurate, and automated solution for water quality management, with significant practical value and promotional potential.

[0199] It should be noted that, in the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0200] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0201] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A method for predicting water quality coagulation effect based on multimodal data and deep learning model, characterized in that: include: Acquire a set of multimodal data sets of target water quality to be measured, including a set of floc images of the target water quality to be measured collected at multiple times and sensor acquisition parameters at each time; The sensor acquisition parameters include water quality parameters that characterize the coagulation effect; The group of floc images to be tested and the sensor acquisition parameters at each moment are respectively input into the first branch and the second branch of the pre-trained improved deep learning model, and the first coagulation effect prediction result and the second coagulation effect prediction result are obtained accordingly. The first coagulation effect prediction result and the second coagulation effect prediction result are fused to obtain the fused coagulation effect prediction result of the target water quality; wherein, the first branch includes a ResNet model and a first LSTM model connected in sequence, and the second branch includes a second LSTM model; the improved deep learning model is trained based on multiple groups of sample multimodal data sets of water quality samples; the coagulation effect prediction result includes coagulation effect grade and / or coagulation effect parameters; the coagulation effect grade is used to characterize different levels of coagulation effects, and the coagulation effect parameters include water quality parameters characterizing the final coagulation effect.

2. The method for predicting water quality coagulation effect based on multimodal data and deep learning model according to claim 1, characterized in that: The coagulation effect grades include perfect coagulation, very good coagulation, good coagulation, better coagulation, appropriate coagulation, general coagulation, slightly poor coagulation, significantly poor coagulation, very poor coagulation, and no coagulation, a total of 10 grades, indicating that the coagulation effect becomes worse in order; The coagulation effect parameters include pH value, temperature, turbidity NTU, chemical oxygen demand COD, absorbance at A275, absorbance at A220 and absorbance at A520.

3. The method for predicting water quality coagulation effect based on multimodal data and deep learning model according to claim 1, characterized in that: After obtaining the prediction result of the fusion coagulation effect of the target water quality, the method further includes: The coagulant dosage of the target water quality is determined according to the prediction result of the fusion coagulation effect of the target water quality and the pre-established correspondence between the water quality coagulation effect prediction result and the coagulant dosage.

4. The method for predicting water quality coagulation effect based on multimodal data and deep learning model according to claim 1, characterized in that: The training process of the improved deep learning model includes: By collecting floc images and sensor acquisition parameters at each moment during a water quality coagulation experiment with multiple water quality samples, multiple groups of sample multimodal data sets are obtained to form a data set; wherein, one sample in the data set is a group of sample multimodal data sets; each group of sample multimodal data sets corresponds to a single water quality coagulation experiment, including a group of sample floc images and sensor acquisition parameters at multiple moments, wherein the group of sample floc images is floc images collected at multiple moments during the water quality coagulation experiment with the same water quality sample, and the acquisition moments of the sensor acquisition parameters at the multiple moments correspond one-to-one to the acquisition moments of the floc images; Labeling the samples in the data set; wherein the label data of each sample includes a coagulation effect level and / or a coagulation effect parameter; the coagulation effect parameter used as the label data is a sensor acquisition parameter at the last moment of the water quality coagulation experiment; Perform data preprocessing on the dataset after labeling the labeled data, and divide the training set from the dataset after data preprocessing; In each iterative training, the samples of the training set are input into the improved deep learning model, and a group of sample floc images in the sample are processed by the first branch to output a first coagulation effect prediction result; the sensor acquisition parameters at multiple moments in the sample are processed by the second branch to output a second coagulation effect prediction result, and the first coagulation effect prediction result and the second coagulation effect prediction result are fused by weighted fusion to obtain a fused coagulation effect prediction result; The loss function is calculated using the fused coagulation effect prediction results of the input sample and the corresponding label data, and the model parameters are adjusted by backpropagation; wherein the model parameters include the weight parameters corresponding to the first coagulation effect prediction result and the second coagulation effect prediction result in the weighted fusion method; Repeat multiple iterations of training until an improved deep learning model is obtained.

5. The method for predicting water quality coagulation effect based on multimodal data and deep learning model according to claim 4 is characterized in that: In each iterative training, the samples of the training set are input into the improved deep learning model, and a group of sample floc images in the sample are processed using the first branch to output a first coagulation effect prediction result, including: In each iterative training, the samples of the training set are input into the first branch of the improved deep learning model, and the image features are extracted using the ResNet model to obtain the image features of each floc image in the sample; The image features of each floc image in the sample are combined into an image feature time series sequence according to the time sequence of the floc images, and the image features are input into a first LSTM model for time series feature extraction, and a first coagulation effect prediction result corresponding to the sample is output.

6. The method for predicting water quality coagulation effect based on multimodal data and deep learning model according to claim 4, characterized in that: The second branch is used to process the sensor acquisition parameters at multiple moments in the sample and output a second coagulation effect prediction result, including: The sensor acquisition parameters at multiple moments in the sample are arranged in time sequence to form a sensor parameter feature time series sequence, which is input into the second LSTM model for time series feature extraction, and the second coagulation effect prediction result corresponding to the sample is output.

7. The method for predicting water quality coagulation effect based on multimodal data and deep learning model according to claim 4, characterized in that: Data preprocessing for floc images includes: Image culling, image cropping, and image enhancement.

8. The method for predicting water quality coagulation effect based on multimodal data and deep learning model according to claim 7, characterized in that: Data preprocessing for sensor acquisition parameters includes: Data cleaning, handling of missing values ​​and outliers.

9. The method for predicting water quality coagulation effect based on multimodal data and deep learning model according to claim 1, characterized in that: The ResNet model includes a ResNet18 network or a ResNet50 network.

10. The method for predicting water quality coagulation effect based on multimodal data and deep learning model according to claim 4, characterized in that: The method further comprises: A training set is divided from the data set after data preprocessing, and a validation set is divided at the same time. The validation set is used to evaluate the model performance after the training is completed.

Images