Sewage treatment data exception restoration method based on GAN and LSTM

Abstract

The invention relates to the technical field of data processing and intelligent monitoring, and discloses a GAN and LSTM-based sewage treatment data exception restoration method, which comprises the following steps: collecting and processing multi-source data in a sewage treatment plant in an industrial park; constructing a time sequence generative adversarial network model, taking a CNN, an LSTM network and a full connection layer as a framework of a generator, and constructing a discriminator through the full connection layer; performing time sequence generative adversarial network model training and testing on the preprocessed normal data according to a pre-delimited training set and test set proportion, wherein the training process is alternate training of a generator and a discriminator; performing exception repair on the test set data based on the trained time sequence generative adversarial network model; and integrating the time sequence generative adversarial network model into the industrial park sewage treatment plant datamation management control system for real-time detection. According to the method, the abnormal points in the sewage treatment data can be automatically detected, the repair data conforming to the actual rule is generated, and the data quality and the system stability are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing and intelligent monitoring, and in particular to a method for repairing sewage treatment data anomaly based on GAN and LSTM. Background Art

[0002] Industrial park sewage treatment plants play an important role in preventing environmental pollution, maintaining ecological balance and promoting sustainable economic development. Enterprises in industrial parks often involve multiple industries such as chemical, metallurgical, textile, and electronics, and the discharged sewage often contains high concentrations of organic matter, inorganic salts and toxic and hazardous substances. And because the production processes and discharge rules of each enterprise are different, the water quality and water volume of sewage have large fluctuations, making the sewage treatment process highly complex and uncertain.

[0003] In order to meet this challenge, most sewage treatment plants are equipped with online water quality monitoring equipment and corresponding sewage treatment process control systems, which can collect and process a large amount of sewage treatment process data in real time. However, in actual applications, due to various reasons such as sensor failure, communication network interruption, environmental interference, etc., these data often have problems such as missing, abnormal or noisy, resulting in low data quality and unable to meet the needs of subsequent analysis and processing.

[0004] Traditional data repair methods, such as statistical models or simple interpolation algorithms, can alleviate the problem of missing and abnormal data to a certain extent, but these methods are limited in effectiveness for sewage treatment time series data with complex biochemical reaction processes. They often cannot accurately capture the complex patterns and changing trends in the data, resulting in certain errors and uncertainties in the repaired data.

[0005] With the continuous development of artificial intelligence (AI) technology, AI+wastewater treatment has become a development trend. However, the driving of AI models requires a large amount of high-quality data as support. When there are problems with data quality, the application of AI models in the field of sewage treatment is greatly limited. Therefore, how to effectively repair the anomalies and missing problems in sewage treatment data and improve the quality and availability of data has become a key issue that needs to be solved urgently. Summary of the invention

[0006] The present invention aims to provide a sewage treatment data anomaly repair method based on GAN and LSTM, which can automatically detect anomalies in sewage treatment data, and use a deep learning model to generate repair data that conforms to actual laws, thereby improving data quality and system stability, so as to improve the effectiveness and reliability of data in the sewage treatment process and ensure the stable operation of the sewage treatment system.

[0007] In order to achieve the above object, the present invention adopts the following technical scheme:

[0008] A sewage treatment data anomaly repair method based on GAN and LSTM, comprising:

[0009] Data preprocessing step: collecting multi-source data from the industrial park sewage treatment plant and preprocessing the data;

[0010] Model construction steps: construct a temporal generative adversarial network model. The temporal generative adversarial network model includes a generator and a discriminator. CNN, LSTM network and fully connected layer are used as the architecture of the generator, and the discriminator is constructed with a fully connected layer;

[0011] In the model training step, the preprocessed non-missing data is used to train and test the temporal generative adversarial network model according to the pre-defined training set and test set ratios. The training process is alternating training of the generator and the discriminator.

[0012] The data anomaly repair and evaluation step repairs the anomaly of the test set data based on the trained time series generative adversarial network model;

[0013] In the system integration and deployment steps, the time series generative adversarial network model is integrated into the data management and control system of the industrial park sewage treatment plant for real-time detection.

[0014] The principles and advantages of this solution are as follows: in actual application, multi-source data in the industrial park sewage treatment plant is collected and preprocessed to ensure data availability; a time series generative adversarial network model is constructed, with CNN, LSTM network and fully connected layer as the architecture of the generator, and a discriminator is constructed with a fully connected layer to improve the learning ability of the generator. The preprocessed normal data is used for time series generative adversarial network model training and testing according to the pre-defined training set and test set ratios. The training process is alternating training of the generator and the discriminator to improve the discriminator's ability to distinguish between real data and generated data; based on the trained time series generative adversarial network model, the test set data is repaired for anomalies to improve the repair accuracy and generalization ability; finally, the time series generative adversarial network model is integrated into the digital management and control system of the industrial park sewage treatment plant for real-time detection to effectively repair anomalies and omissions in sewage treatment data and improve data quality and availability.

[0015] Beneficial effects: Improve data quality: LSTM is good at processing long-term dependencies related to time series. LSTM can capture the correlation between abnormal data and the data at the previous and next time points, so as to more accurately identify and repair abnormal data. GAN is trained by game theory, which can learn complex data distribution and generate data similar to normal data to repair the gaps left by abnormal data, thereby restoring the integrity and consistency of the data. Through the combination of GAN and LSTM, it is possible to effectively identify and repair abnormal points in sewage treatment data, and improve the accuracy and integrity of the data.

[0016] Enhance system stability: Accurate sewage treatment data provides strong support for system monitoring and decision-making, helps to timely discover and deal with potential problems, and ensures stable system operation.

[0017] Improve the level of intelligence: realize the automation and intelligence of data processing in industrial park sewage treatment plants, and reduce manual intervention and costs.

[0018] Preferably, as an improvement, the preprocessing includes data cleaning, feature selection, data standardization, data loading and data partitioning.

[0019] Technical effect: easy to improve data reliability.

[0020] Preferably, as an improvement, the data loading includes: clarifying the location and amount of missing data, repairing the data with fewer missing columns according to the information gain theory, and classifying all data without missing data as independent variables and data with missing data as target variables on the existing complete data set, so that the independent variables and target variables form a data set.

[0021] Technical effect: It is easy to improve the accuracy of the data set.

[0022] Preferably, as an improvement, the architecture using CNN, LSTM network and fully connected layer as generator includes: using one-dimensional convolutional network to convert three-dimensional data into one-dimensional form, and then capturing the temporal dependencies in sewage treatment data by introducing LSTM network model with gating mechanism and memory unit, learning the distribution of normal sewage treatment data, extracting long-term and short-term features in the data, and guiding the generator to generate data samples that conform to reality.

[0023] Technical effect: It is easy to improve the learning ability of the generator.

[0024] Preferably, as an improvement, the model training step includes:

[0025] In the discriminator training sub-step, the generator weight parameters are randomly initialized, a set of random noise with the same dimension as the training data is input into the generator, a batch of sample data is generated, the batch of sample data is labeled as generated data, and a batch of real data is extracted from the training data set and labeled as real data;

[0026] In the generator training sub-step, the discriminator weight parameters are fixed, the generator is optimized using the discriminator, the training data is input into the generator to obtain the generated data, the generated data is labeled as real data and input into the discriminator, and the generator parameters are optimized through the loss returned by the discriminator.

[0027] Technical effect: It is easy to train the discriminator to distinguish between real data and generated data.

[0028] Preferably, as an improvement, in the model training step, the loss function selects a binary cross entropy loss function, and the model weights are adjusted by a back propagation algorithm.

[0029] Technical effect: It is easy to improve the robustness of the model.

[0030] Preferably, as an improvement, the loss function of the generator is:

[0031]

[0032] Among them, z represents the noise The random noise sampled from the distribution, g(z) represents the data generated by the generator based on the noise z, and d(g(z)) represents the judgment result of the discriminator on the generated data.

[0033] Technical effect: It is easy to reduce the generator loss and improve the robustness of the model.

[0034] Preferably, as an improvement, the loss function of the discriminator is:

[0035]

[0036] Among them, x represents the real data set X train The data sampled from X noise The random noise sampled from the distribution, g(z) represents the data generated by the generator based on the noise z, d(x) and d(g(z)) represent the judgment results of the discriminator on the real data and the generated data, respectively.

[0037] Technical effect: It is easy to reduce the discriminator loss and improve the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a technical route diagram of a sewage treatment data anomaly repair method based on GAN and LSTM;

[0039] Figure 2 Schematic diagram of the TSGAN architecture according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following is further described in detail through specific implementation methods:

[0041] The embodiment is basically as shown in the attached Figure 1 , Attachment Figure 2 As shown:

[0042] A sewage treatment data anomaly repair method based on GAN and LSTM, comprising:

[0043] The data preprocessing step collects multi-source data from the industrial park sewage treatment plant and preprocesses the data, which includes data cleaning, feature selection, data standardization, data loading and data partitioning.

[0044] The multi-source data include water inlet flow, water level, conventional water quality parameters (such as pH value, COD, ammonia nitrogen, dissolved oxygen, etc.), blower working status, pump station operating status, etc.

[0045] The data cleaning includes: processing abnormal values ​​(such as values ​​that obviously exceed the physical range) and duplicate data.

[0046] The feature selection includes: selecting key features according to business needs. For example, if the goal is to reduce energy consumption, it is first necessary to clarify the key indicators of the sewage treatment plant in terms of treatment efficiency, effluent water quality, energy consumption, etc., and determine the five conventional (COD, BOD5, TSS, ammonia nitrogen, pH) inlet and outlet water quality parameters, aeration energy consumption, and total energy consumption data as key features based on relevant knowledge, and then use the key features for model training and prediction.

[0047] The data standardization includes: standardizing key feature data and eliminating the influence of different dimensions on model training so that the model can learn better.

[0048] The data loading includes: clarifying the location and quantity of missing key feature data, and according to the information gain theory, that is, the amount of uncertainty reduction of the data set under the condition of knowing a certain feature, the calculation formula is as follows:

[0049] IG(D,A)=H(D)-H(D|A)

[0050] In the formula, H(D) is the entropy of the data set D, which indicates the degree of disorder of the data set; H(D|A) indicates the conditional entropy of the data set D under the condition of given feature A, that is, the degree of disorder of the data set when feature A is known, indicating how much the uncertainty of the data set D is reduced by obtaining the information of feature A. If the information gain is negative, it means that feature A can bring information gain; if the information gain is zero, it means that feature A has been fully utilized and can no longer provide additional information. When dealing with missing data, missing data will increase the uncertainty of the data set, and information gain is used to measure the reduction in uncertainty. When repairing missing multi-feature data, first repair the data with fewer missing columns of key feature data, which can reduce the uncertainty of the data set more quickly and improve the accuracy of subsequent data analysis and model building. On the existing complete data set, all key feature data without missing columns are classified as independent variables X, and key feature data with missing columns are classified as target variables Y to form a data set (X, Y).

[0051] The data division includes: dividing all key feature data (X, Y) in the data set into a training set (X train ,Y train ) and the test set (X test ,Y test ), which are used for model training and testing respectively. The ratio division refers to the common settings in the field of machine learning and data science, which can balance training and evaluation, improve model performance and avoid overfitting. In this embodiment, the preset ratio of training set: test set is 8:2.

[0052] Model construction steps: construct a temporal generative adversarial network (GAN) model. The temporal generative adversarial network model includes a generator and a discriminator. The CNN, LSTM network and fully connected layer are used as the architecture of the generator. First, a one-dimensional convolutional network is used to convert the three-dimensional data into a one-dimensional form, and then the temporal dependency in the sewage treatment data is captured by introducing a gating mechanism and an LSTM network model with a memory unit. Specifically, the gating mechanism discards some non- / low-correlated information through the forgetting gate, the input gate selectively accepts new information and combines it with the old information, the output gate determines which information is output from the cell state, and the memory unit is responsible for updating and storing the cell state. Through the control of the forgetting gate and the input gate, the memory unit can continuously update its internal parameter weights to reflect the long-term dependency of the input data. Through the continuous adversarial training of the generator and the discriminator in GAN, the capabilities of both will gradually improve. According to the goal of minimizing the loss function, the Nash equilibrium is finally reached. The model learns the distribution of the non-missing sewage treatment data, making the distribution of the generated data close to the real data distribution (g(z)→X train). The gating mechanism and memory unit in LSTM are used to extract long-term and short-term features in the data, guiding the generator to generate data samples that conform to reality. Long-term features include water quality stability (biochemical stability, long-term trend of dissolved oxygen, COD and other indicators), changes in pollutant components (affected by seasons, weather, etc.), and short-term features include daily fluctuations (such as changes in domestic sewage discharge within a day are usually related to people’s daily living habits), sudden pollution events, and equipment operation status (changes in treatment equipment in a short period of time, such as the flow of water pumps and the oxygen supply of aeration equipment, will affect water quality). The LSTM network is followed by a fully connected layer, which further transforms the features through the construction of multiple hidden layers, deepens the network depth, strengthens the nonlinear expression of the model, and improves the learning ability of the generator. The discriminator is constructed with a fully connected layer to distinguish whether the input data is real data or data generated by the generator.

[0053] The model training step is to train and test the time series generative adversarial network model using the preprocessed non-missing data according to the pre-defined training set and test set ratios. The training process is alternating training of the generator and the discriminator. The model training step includes:

[0054] The discriminator training substep randomly initializes the generator weight parameters and the generator inputs a set of training data X train Random noise X of the same dimension noise , generate a batch of sample data (X noise ,Y noise ), label the batch of sample data as generated data (marked as 0), and extract a batch of real data (X train ,Y train ), label it as real data (marked as 1); generate sample data (X noise ,Y noise ) )(marked as 0) and the real data (X train ,Y train ) (marked as 1) is input into the discriminator, and the weight parameters in the discriminator are updated through the Adam optimizer with the goal of minimizing the loss function, so as to train the discriminator to distinguish between real data and generated data.

[0055] Generator training sub-step, fix the discriminator weight parameters, use the discriminator to optimize the generator, and train the data X train Input generator to get generated data Will generate data The labeled real data (marked as 1) is input into the discriminator, and the loss function is calculated by the value returned by the discriminator. The goal is to minimize the loss function and optimize the generator parameters through Adam update.

[0056] Among them, the loss function selects the binary cross entropy (BCE) loss function, and the model weight is adjusted through the Adam optimizer.

[0057] The loss function of the generator is:

[0058]

[0059] Among them, z represents the noise The random noise sampled from the distribution, g(z) represents the data generated by the generator based on the noise z, and d(g(z)) represents the judgment result of the discriminator on the generated data.

[0060] The loss function of the discriminator is:

[0061]

[0062] Among them, x represents the real data set X train The data sampled from X noise The random noise sampled from the distribution, g(z) represents the data generated by the generator based on the noise z, d(x) and d(g(z)) represent the judgment results of the discriminator on the real data and the generated data, respectively.

[0063] In the data anomaly repair and evaluation step, the test set data is repaired based on the trained time series generative adversarial network model; specifically, the generator extracts time series features to generate repaired data samples, and the discriminator makes a comprehensive judgment on the repair results, so that the overall data is close to the distribution of normal data. The repaired data is evaluated for quality and compared with the original data to verify the repair effect. According to the evaluation results, the model parameters are adjusted or the model structure is optimized to improve the repair accuracy and generalization ability, and then the tested model is applied to the abnormal data for repair.

[0064] In the system integration and deployment steps, the time-series generative adversarial network model is integrated into the data management and control system of the industrial park sewage treatment plant for real-time detection. The system is deployed in the actual production environment to ensure that the data stream can be processed in real time, the system can be monitored in real time, and any abnormal data phenomena can be prompted.

[0065] The above is only an embodiment of the present invention, and the common knowledge such as the known specific technical solutions and / or characteristics in the solution is not described in detail here. It should be pointed out that for those skilled in the art, without departing from the technical solution of the present invention, several modifications and improvements can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A sewage treatment data anomaly repair method based on GAN and LSTM, characterized in that: include: Data preprocessing step: collecting multi-source data from the industrial park sewage treatment plant and preprocessing the data; Model construction steps: construct a temporal generative adversarial network model. The temporal generative adversarial network model includes a generator and a discriminator. CNN, LSTM network and fully connected layer are used as the architecture of the generator, and the discriminator is constructed with a fully connected layer; In the model training step, the preprocessed non-missing data is used to train and test the temporal generative adversarial network model according to the pre-defined training set and test set ratios. The training process is alternating training of the generator and the discriminator. The data anomaly repair and evaluation step repairs the anomaly of the test set data based on the trained time series generative adversarial network model; In the system integration and deployment steps, the time series generative adversarial network model is integrated into the data management and control system of the industrial park sewage treatment plant for real-time detection.

2. According to claim 1, a sewage treatment data anomaly repair method based on GAN and LSTM is characterized in that: The preprocessing includes data cleaning, feature selection, data standardization, data loading and data partitioning.

3. According to claim 2, a sewage treatment data anomaly repair method based on GAN and LSTM is characterized in that: The data loading includes: clarifying the location and amount of missing data, repairing the data with fewer missing columns according to the information gain theory, and classifying all data without missing data as independent variables and data with missing data as target variables on the existing complete data set, so that the independent variables and target variables form a data set.

4. According to claim 1, a sewage treatment data anomaly repair method based on GAN and LSTM is characterized in that: A one-dimensional convolutional network is used to transform three-dimensional data into one-dimensional form. Then, an LSTM network model with a gating mechanism and memory units is introduced to capture the temporal dependencies in the sewage treatment data, learn the distribution of normal sewage treatment data, extract long-term and short-term features in the data, and guide the generator to generate data samples that conform to reality.

5. According to claim 1, a sewage treatment data anomaly repair method based on GAN and LSTM is characterized in that: The model training step includes: In the discriminator training sub-step, the generator weight parameters are randomly initialized, a set of random noise with the same dimension as the training data is input into the generator, a batch of sample data is generated, the batch of sample data is labeled as generated data, and a batch of real data is extracted from the training data set and labeled as real data; In the generator training sub-step, the discriminator weight parameters are fixed, the generator is optimized using the discriminator, the training data is input into the generator to obtain the generated data, the generated data is labeled as real data and input into the discriminator, and the generator parameters are optimized through the loss returned by the discriminator.

6. According to claim 1, a sewage treatment data anomaly repair method based on GAN and LSTM is characterized in that: In the model training step, the loss function selects a binary cross entropy loss function, and the model weights are adjusted through a back propagation algorithm.

7. According to claim 6, a sewage treatment data anomaly repair method based on GAN and LSTM is characterized in that: The loss function of the generator is: Among them, z represents the noise The random noise sampled from the distribution, g(z) represents the data generated by the generator based on the noise z, and d(g(z)) represents the judgment result of the discriminator on the generated data.

8. According to claim 6, a sewage treatment data anomaly repair method based on GAN and LSTM is characterized in that: The loss function of the discriminator is: Among them, x represents the real data set X train The data sampled from X noise The random noise sampled from the distribution, g(z) represents the data generated by the generator based on the noise z, and d(x) and d(g(z)) represent the judgment results of the discriminator on the real data and the generated data, respectively.

Images