Intelligent detection and dynamic early warning system based on machine learning and multi-source data fusion

Abstract

The invention discloses an intelligent detection and dynamic early warning system based on machine learning and multi-source data fusion, which relates to the technical field of membrane separation process monitoring and consists of a data acquisition and preprocessing part, a physical model and data driving model part, a dynamic weight fusion and self-adaptive mechanism part and a man-machine interaction and alarm part. According to the system, parameters such as membrane flux, transmembrane pressure difference, solution concentration, temperature, water inlet flow and operation pressure are collected, an improved XGBoost machine learning model is combined, and a comprehensive pollution index is constructed to quantify the membrane pollution degree; the model realizes double analysis of a pollution mechanism and a data rule through feature engineering. The system supports full-closed-loop management from minute-level early warning to hour-level cleaning decision making, the threshold value is dynamically adjusted through Bayesian optimization, the generalization and anti-interference capacity of the system are remarkably improved, and the system is suitable for intelligent operation and maintenance of membrane separation processes such as reverse osmosis and ultrafiltration.

Description

Technical Field

[0001] The present invention relates to the technical field of membrane separation process monitoring. Specifically, it relates to an intelligent detection and dynamic early warning system based on machine learning and multi-source data fusion. Background Art

[0002] Traditional membrane fouling monitoring systems face multiple technical challenges: The reliance on a single parameter leads to a high risk of misjudgment. For example, flow rate fluctuations can easily cause an instantaneous increase in transmembrane pressure difference (ΔP). Traditional models often misjudge this as an exacerbation of fouling, resulting in a warning delay of 2 - 4 hours. The simplified physical model based on Darcy's law ignores complex operating conditions such as concentration polarization and membrane pore blockage. In low-flow, high-concentration environments, the prediction error exceeds ±15%, making it difficult to meet the requirements of refined scenarios such as industrial wastewater treatment and seawater desalination. Although data-driven models can capture non-linear relationships, they have the limitation of relying on a large amount of historical data. When encountering water quality mutations such as organic matter shocks, the accuracy drops below 60% immediately, and the model reuse rate across different processes (such as reverse osmosis and ultrafiltration) is insufficient, significantly increasing the R & D cost. The traditional fixed-threshold alarm mechanism does not consider the parameter sensitivity drift caused by membrane aging. The same pollution index (PI) may correspond to an actual pollution difference of more than 3 times, leading to problems such as over-cleaning (membrane life is shortened by 20% - 30%) or untimely cleaning of the water production quality, systematically reducing the monitoring efficiency.

[0003] The technical bottlenecks in the industry are further highlighted in three aspects: The phenomenon of multi-source data islands is common. Sensor parameters (pressure, flow rate, concentration) and operation and maintenance records (cleaning cycle, membrane material parameters) are stored separately, making it difficult to mine the time-series coupling relationship between parameters such as temperature and turbidity, restricting the global analysis ability of complex operating conditions. The static weight allocation model performs poorly under dynamic operating conditions such as the rainy season. For example, when the influent turbidity suddenly increases by 20%, the traditional moving window average method results in a calculation error of the pollution index (PI) exceeding 20%. Edge scenarios (such as remote water plants) face the challenge of data scarcity. When the amount of historical data is less than 1000, the accuracy of machine learning models drops below 70% immediately, seriously hindering the process of intelligent transformation. Summary of the Invention

[0004] In order to solve the above problems, the purpose of the present invention is to provide a membrane fouling intelligent detection and dynamic early warning system based on machine learning and multi-source data fusion, aiming to solve the problems of single-parameter dependence, early warning lag, and extensive control strategies in traditional membrane separation processes.

[0005] To achieve the above technical objectives, the present application provides an intelligent detection and dynamic early warning system based on machine learning and multi-source data fusion, including: A data acquisition and preprocessing module, which is used to collect membrane flux, transmembrane pressure difference, solution concentration, temperature, influent flow rate, and operating pressure during the membrane separation process, and integrate historical operation data to form a data set; A data analysis module, which is used to simulate the pollution trend based on the membrane pollution kinetics theory by setting a physical model according to a data set, and use a data-driven model to learn the non-linear relationship from historical data through machine learning; An adaptive adjustment module, which is used to allocate dynamic weight coefficients to the physical model and the data-driven model through Bayesian optimization or SHAP value analysis, and adjust the weights in real time according to membrane aging or water quality mutation; An interactive warning module, which is used to display the real-time PI value, pollution trend and weight change, and trigger an alarm for exceeding the threshold through a message queue.

[0006] Preferably, the data analysis module obtains a physical driving index according to the physical model to quantify the direct impact of flow fluctuation and pollution accumulation; and obtains a data-driven index according to the data-driven model to capture non-explicit rules through machine learning, and converts the probability output into an exponential form to enhance sensitivity.

[0007] Preferably, the data analysis module obtains a comprehensive pollution index based on the physical driving index and the data-driven index, where the comprehensive pollution index is expressed as: In the formula, : The physical driving index The weight of, reflecting the contribution ratio of the physical mechanism; : The data-driven index The weight of, reflecting the contribution ratio of the experience learned by the machine learning model from historical data, and PI represents the comprehensive pollution index.

[0008] Preferably, when the data analysis module obtains the physical driving index, the physical driving index is expressed as: In the formula, : The weight coefficient; : The transmembrane pressure difference at the t time point; : The time window; : The membrane flux at the t time point; : The flow rate of the influent water; : The solution concentration at the t time point; : The initial concentration of the influent solution; where, Quantifies the time change rate of the transmembrane pressure difference under unit flow rate, reflecting the cumulative effect of flow resistance caused by membrane pollution; Normalizes the actual concentration by the reference concentration to measure the negative correlation between the pollutant interception efficiency and the degree of membrane pollution.

[0009] Preferably, when the data analysis module obtains the data-driven index, the data-driven index is expressed as: In the formula, is the sensitivity parameter; is the probability of membrane fouling degree.

[0010] Preferably, the adaptive adjustment module optimizes the physical index parameters through XGBoost grid search , ; and adopts Bayesian optimization to optimize the data index parameters with F1-score as the evaluation index .

[0011] Preferably, the interactive warning module defines the hierarchical warning and response trigger of the comprehensive pollution index according to the membrane material characteristics and working conditions: Normal operation: When , maintain the current working condition and continuously monitor; Mild pollution warning: When , trigger an hourly warning: push optimization suggestions, increase the flow rate or reduce the influent concentration; Moderate pollution warning: When , trigger a minute-level warning: perform local pulse cleaning, close the membrane module in the faulty area and switch to the standby pool for operation; Severe pollution warning: When , trigger an emergency cleaning instruction: start the chemical cleaning program, empty the membrane pool and perform manual maintenance.

[0012] Preferably, the adaptive adjustment module dynamically updates the warning threshold through a Bayesian network based on historical operation data and SHAP value analysis to adapt to water quality fluctuations or membrane material aging..

[0013] The present invention discloses the following technical effects: The present invention supports full closed-loop management from minute-level warning to hourly cleaning decision-making, and dynamically adjusts the threshold through Bayesian optimization, significantly improving the generalization and anti-interference capabilities of the system, and is applicable to the intelligent operation and maintenance of membrane separation processes such as reverse osmosis and ultrafiltration. Brief Description of the Drawings

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0015] Figure 1 is the system module interaction flow chart of the present invention. Detailed Embodiments

[0016] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part rather than all of the embodiments of this application. Components of the embodiments of this application usually described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative efforts belong to the scope of protection of this application.

[0017] As Figure 1 shown, the present invention provides an intelligent membrane fouling detection and dynamic warning system based on machine learning and multi-source data fusion. The system collects parameters such as membrane flux, transmembrane pressure difference, solution concentration, temperature, influent flow rate, and operating pressure in real time through a multi-sensor network, and constructs a comprehensive fouling index to quantify the degree of membrane fouling in combination with an improved XGBoost machine learning model. The model realizes the dual analysis of fouling mechanism and data law through feature engineering. Specifically, the system includes four core parts: data collection and preprocessing, physical model and data-driven model, dynamic weight fusion and adaptive mechanism, and human-computer interaction and alarm: Data collection and preprocessing module (data collection and preprocessing part): As the data foundation of the system, this module is responsible for collecting key parameters such as transmembrane pressure difference, membrane flux, and influent concentration in real time, and integrating historical operation data. After obtaining multi-source data through the sensor network, cleaning and feature engineering are performed to provide high-quality input for subsequent modeling.

[0018] Data analysis module (physical model and data-driven model part), which respectively mines fouling laws. The physical model is based on the membrane fouling kinetics theory, combined with the calibration of parameters such as viscosity and concentration gradient, to simulate the fouling trend; the data-driven model uses machine learning to learn the non-linear relationship from historical data to enhance the adaptability to complex working conditions. The outputs of both are fused through dynamic weights to generate the final fouling index.

[0019] Adaptive adjustment module (dynamic weight fusion and adaptive mechanism part), through Bayesian optimization or SHAP value analysis, the system assigns dynamic weight coefficients to the physical model and the data-driven model

[0020] The interactive warning module (human-machine interaction and alarm part) provides a visual dashboard to display real-time PI values, pollution trends, and weight changes, and triggers an alarm when the threshold is exceeded through a message queue. Operation and maintenance personnel can formulate scientific maintenance strategies based on the cleaning cycle suggestions generated by the historical data analysis module.

[0021] The steps for the system to perform detection are as follows: S1. The model collects and processes the following parameters in real time through a multi-sensor network for training: membrane flux ; transmembrane pressure difference , solution concentration , temperature T, influent flow rate , operating pressure P, and time series features ; Analyze the data law through feature engineering, including non-linear combinations of each parameter and dynamic weight allocation.

[0022] S2. The comprehensive pollution index adopts a weighted superposition model, combining physical driving factors and data-driven features: Where : weight of the physical driving index , reflecting the contribution ratio of the physical mechanism; : weight of the data-driven index , reflecting the contribution ratio of the experience learned by the machine learning model from historical data.

[0023] S3. The physical driving index is based on the core physical equation of the membrane separation process, quantifying the direct impact of flow rate fluctuations and pollution accumulation: Among them, : weight coefficient; : transmembrane pressure difference at the t time point; : time window; : membrane flux at the t time point; : influent flow rate; : solution concentration at the t time point; : initial concentration of the influent solution; Quantify the time change rate of the transmembrane pressure difference per unit flow rate, reflecting the cumulative effect of flow resistance caused by membrane pollution; Normalize the actual concentration by the reference concentration to measure the negative correlation between pollutant rejection efficiency and membrane pollution degree.

[0024] S4. The data-driven index Capture implicit patterns through machine learning, convert probability outputs into exponential form to enhance sensitivity: Among them, : Sensitivity parameter; : Probability of membrane fouling degree (range 0-1).

[0025] S5. Achieve adaptive fusion of physical and data-driven exponents through double-layer XGBoost optimization: The first layer: The input features include flow rate volatility, pressure difference change rate, concentration gradient, etc. Through the following formula, optimize the weight parameters and output , : Among them, λ represents the soft constraint penalty coefficient, which is used to balance the weight distribution deviation when the hard constraint conditions cannot be fully met.

[0026] The second layer: Optimize the physical exponent parameters , through XGBoost grid search; Use Bayesian optimization to optimize the sensitivity parameter with F1-score as the evaluation index.

[0027] Machine learning reconstructs the fouling index formula through a double-layer optimization framework to achieve dynamic coordination of physical parameters and data features: Train the XGBoost model based on historical data to automatically optimize the physical driving coefficients , and the time window . For example, in a reverse osmosis system, when the model determines through grid search that = 0.7, = 0.3, the calculation error of ∈[0,1] is mapped into exponential form , and the k value is determined through Bayesian optimization to avoid the risk of exponential explosion.

[0028] Define the hierarchical warning and response trigger of the comprehensive fouling index according to the membrane material characteristics and working conditions: Normal operation: When , maintain the current working condition and continuously monitor; Mild fouling warning: When , trigger an hourly warning: Push optimization suggestions to increase the flow rate or reduce the inlet concentration; Moderate fouling warning: When When [condition], trigger a minute-level warning: perform local pulsed cleaning, close the membrane modules in the fault area, and switch to operation in the standby pool; Severe pollution warning: When [condition], trigger an emergency cleaning instruction: start the chemical cleaning program, empty the membrane pool, and perform manual maintenance.

[0029] Based on historical operation data and SHAP value analysis, dynamically update the warning threshold through a Bayesian network to adapt to water quality fluctuations or membrane material aging.

[0030] Example 1: Install a reverse osmosis membrane module, where the membrane material is a polyamide composite membrane with an effective area of 0.5 , equipped with a pressure sensor, a flow meter, and an online COD monitor. Run the device to obtain a dataset: = 1 - 3 bar, = 5 - 15 , = 50 - 300 , = 20 - 35 °C.

[0031] Determine the optimal parameter combination through grid search to obtain , (validation set MAE = 3.2%); Design two groups of comparative experiments to output the validation set: , corresponding to MAE = 5.8%, at this time the pressure term is underfitted; , corresponding to MAE = 6.5%, at this time the concentration term dominates the deviation.

[0032] The XGBoost feature importance ranking is flow rate volatility (32%), pressure difference change rate (28%), turbidity - TDS correlation (22%). Determine corresponding to F1-score = 0.89 through Bayesian optimization.

[0033] After inputting the three groups of parameters into the system, output the experimental results: parameter combination , outputs a pollution warning accuracy rate of 92.5%, parameter combination , outputs a pollution warning accuracy rate of 84.3%, parameter combination outputs a pollution warning accuracy rate of 91.2%. Comparing the data, the conclusion can be drawn that the physical parameters , need to be dynamically adjusted according to the working conditions, and the data-driven parameter k can adapt to water quality fluctuations through Bayesian optimization.

[0034] Example 2: The device is the same as that in Embodiment 1, with a noise generator and a flux attenuation controller added. Interference factors are added: the flow rate is suddenly increased from 5 m³ / h to 10 m³ / h for 30 minutes; sensor noise is injected, and a ±10% Gaussian noise is superimposed on the measured value of ΔP; membrane aging is simulated by increasing the flux attenuation rate from 0.5% / h to 1.2% / h.

[0035] After processing the measured data and inputting it into the system, the comprehensive pollution index in the flow rate mutation scenario is calculated to obtain: (Trigger a mild alarm) In the scenario of injecting sensor noise, the ΔP error after filtering is controlled within ±1.5%.

[0036] The experimental results are output. In the flow rate step scenario, MAE = 2.8%, the false alarm rate is 4.3%, and the missed alarm rate is 3.1%; in the sensor noise scenario, MAE = 1.2%, the false alarm rate is 2.8%, and the missed alarm rate is 1.9%; in the membrane aging scenario, MAE = 3.5%, the false alarm rate is 5.6%, and the missed alarm rate is 4.7%. It can be analyzed that the system maintains high precision (MAE < 3%) in the flow rate mutation scenario; through online filtering and dynamic weight adjustment, the false alarm rate < 5%.

[0037] Embodiment 3: To verify the applicability of the system in different membrane processes such as reverse osmosis (RO) and ultrafiltration (UF). Two sets of systems are set up: an RO system and a UF system. The corresponding data are measured respectively: RO system - data of a seawater desalination plant (ΔP = 50 - 80 bar, COD = 300 - 600 mg / L); UF system - data of a municipal sewage treatment plant (ΔP = 0.5 - 5 bar, COD = 50 - 150 mg / L). The known migration method is , Through the migration method, the data of the RO system is migrated to the UF system, and the characteristic weights of the source domain (RO) are obtained as the flow rate volatility is 0.4 and the pressure change rate is 0.6; the fine-tuned weights of the target domain (UF) are the flow rate volatility is 0.3 and the pressure change rate is 0.7. The data of the UF system is migrated to the RO system, and the characteristic weights of the source domain (UF) are obtained as the flow rate volatility is 0.2 and the pressure change rate is 0.8; the fine-tuned weights of the target domain (RO) are the flow rate volatility is 0.4 and the pressure change rate is 0.6.

[0038] According to the above steps, the experimental results show that for the RO→UF scenario, the accuracy of the original model is 78.6%, and after fine-tuning, the accuracy is 89.2%, with the generalization error reduced by 11.4%; for the UF→RO scenario, the accuracy of the original model is 81.4%, and after fine-tuning, the accuracy is 90.7%, with the generalization error reduced by 9.3%. Based on this, it is concluded that through the dynamic weight transfer mechanism, the system maintains high precision (accuracy > 89%) in cross-process scenarios, and since the system fine-tuning time < 2 hours, the secondary development cost of the model is significantly reduced.

[0039] In summary, the present invention proposes four major core technological breakthroughs: First, by constructing a multi-source data fusion platform, integrating real-time sensor data (ΔP, membrane flux, solution concentration) and operation and maintenance historical data (cleaning records, membrane material parameters), establishing a unified database, and using feature engineering to extract high-order interaction features such as "turbidity × flow rate" and "pH × temperature", breaking through the limitations of traditional single-parameter analysis; Second, innovating a double-layer dynamic weight optimization mechanism, using the XGBoost model to realize the dynamic weight allocation of the physical model ( ), and the data model ( ), combining the Bayesian optimization algorithm to calibrate the physical parameters ( , ), and the data sensitivity parameters ( , ) in real time, significantly improving the adaptability of the model to water quality mutations (such as organic matter shock); ) Third, developing an intelligent early warning and adaptive control system, establishing dynamic thresholds for pollution levels (normal → mild → moderate → severe) based on SHAP value analysis, realizing threshold self-update through a Bayesian network, integrating an ultrasonic imaging module to achieve centimeter-level positioning of pollution hotspots (accuracy ±2 cm), and accurately guiding cleaning decisions; Finally, adopting an edge-cloud collaborative architecture, deploying a lightweight model (TensorFlowLite) at the edge to ensure minute-level response, and relying on the Transformer and PPO algorithms to train a high-precision model and regularly push policy updates at the cloud, forming a closed-loop system covering data collection, model training, and decision execution. Through experimental verification, the system controls the calculation error of the pollution index within ±1.5% - 3%, improves the early warning timeliness by 300%, and extends the membrane life by more than 50%, providing an intelligent solution for the full life cycle of membrane separation processes under complex working conditions.

[0040] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to produce a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices produce means for implementing the functions specified in one flow Figure 1 one flow or more flows and / or blocks Figure 1 or means for implementing the functions specified in one block or more blocks.

[0041] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality of" means two or more unless otherwise specifically defined.

[0042] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. An intelligent detection and dynamic early warning system based on machine learning and multi-source data fusion, characterized in that, Including: A data acquisition and preprocessing module, which is used to collect the membrane flux, transmembrane pressure difference, solution concentration, temperature, influent flow rate and operating pressure during the membrane separation process, and integrate historical operation data to form a data set; A data analysis module, which is used to simulate the pollution trend according to the membrane fouling kinetics theory by setting a physical model based on the data set, and use a data-driven model to learn the non-linear relationship from historical data through machine learning; An adaptive adjustment module, which is used to assign dynamic weight coefficients to the physical model and the data-driven model through Bayesian optimization or SHAP value analysis, and adjust the weights in real time according to membrane aging or water quality mutation; An interactive warning module, which is used to display real-time PI values, pollution trends and weight changes, and trigger an alarm when the threshold is exceeded through a message queue.

2. The intelligent detection and dynamic warning system based on machine learning and multi-source data fusion according to claim 1, characterized in that: The data analysis module obtains a physical driving index according to the physical model, which is used to quantify the direct impact of flow rate fluctuations and pollution accumulation; according to the data-driven model, a data-driven index is obtained, which is used to capture non-explicit laws through machine learning, and convert the probability output into an exponential form to enhance sensitivity.

3. The intelligent detection and dynamic warning system based on machine learning and multi-source data fusion according to claim 2, characterized in that: The data analysis module obtains a comprehensive pollution index according to the physical driving index and the data-driven index, where the comprehensive pollution index is expressed as: In the formula, : the weight of the physical driving index , reflecting the contribution ratio of the physical mechanism; : the weight of the data-driven index , reflecting the contribution ratio of the experience learned by the machine learning model from historical data, and PI represents the comprehensive pollution index.

4. The intelligent detection and dynamic warning system based on machine learning and multi-source data fusion according to claim 3, characterized in that: When the data analysis module obtains the physical driving index, the physical driving index is expressed as: Wherein, : Weight coefficient; : Transmembrane pressure difference at time point t; : Time window; : Membrane flux at time point t; : Flow rate of influent water; : Solution concentration at time point t; : Initial concentration of influent solution; wherein, Quantifies the time change rate of transmembrane pressure difference under unit flow rate, reflecting the cumulative effect of flow resistance caused by membrane fouling; Normalizes the actual concentration by the reference concentration to measure the negative correlation between pollutant retention efficiency and membrane fouling degree.

5. The intelligent detection and dynamic warning system based on machine learning and multi-source data fusion according to claim 4, characterized in that: When the data analysis module obtains the data-driven index, the data-driven index is expressed as: In the formula, is the sensitivity parameter; is the probability of membrane fouling degree.

6. The intelligent detection and dynamic warning system based on machine learning and multi-source data fusion according to claim 5, characterized in that: The adaptive adjustment module optimizes the physical index parameters through XGBoost grid search 、 ; adopts Bayesian optimization to optimize the data index parameters with F1-score as the evaluation index .

7. The intelligent detection and dynamic warning system based on machine learning and multi-source data fusion according to claim 6, characterized in that: The interactive warning module defines hierarchical warnings and response triggers for the comprehensive pollution index according to the membrane material characteristics and working conditions: Normal operation: When occurs, maintain the current working condition and continuously monitor; Minor pollution warning: When is reached, an hourly warning is triggered: Push optimization suggestions to increase the flow rate or reduce the influent concentration; Medium pollution warning: When is reached, a minute-level warning is triggered: perform local pulse cleaning, close the membrane modules in the fault area and switch to the standby pool for operation; Severe pollution warning: When is reached, an emergency cleaning instruction is triggered: start the chemical cleaning procedure, empty the membrane tank and perform manual maintenance.

8. The intelligent detection and dynamic warning system based on machine learning and multi-source data fusion according to claim 7, characterized in that: The adaptive adjustment module dynamically updates the warning threshold through a Bayesian network based on historical operation data and SHAP value analysis to adapt to water quality fluctuations or membrane material aging.

Images