Intelligent detection and dynamic early warning system based on machine learning and multi-source data fusion

Abstract

The present invention discloses an intelligent detection and dynamic early warning system based on machine learning and multi-source data fusion, which relates to the field of membrane separation process monitoring technology. It consists of a data acquisition and preprocessing part, a physical model and data-driven model part, a dynamic weight fusion and adaptive mechanism part, and a human-computer interaction and alarm part. The system collects parameters such as membrane flux, transmembrane pressure difference, solution concentration, temperature, water flow rate and operating pressure, and combines them with an improved XGBoost machine learning model to construct a comprehensive pollution index to quantify the degree of membrane pollution. The model uses feature engineering to achieve dual analysis of pollution mechanisms and data patterns. The system supports full closed-loop management from minute-level early warnings to hour-level cleaning decisions, and dynamically adjusts thresholds through Bayesian optimization, significantly improving the system's generalization and anti-interference capabilities. It is suitable for the intelligent operation and maintenance of membrane separation processes such as reverse osmosis and ultrafiltration.

Description

Technical Field

[0001] The present invention relates to the technical field of membrane separation process monitoring, and in particular to an intelligent detection and dynamic early warning system based on machine learning and multi-source data fusion. Background Art

[0002] Traditional membrane fouling monitoring systems face multiple technical challenges. Their reliance on a single parameter leads to a high risk of misjudgment. For example, flow rate fluctuations can easily trigger a transient increase in the transmembrane pressure differential (ΔP), which traditional models often misinterpret as increased contamination, resulting in warning delays of 2-4 hours. Simplified physical models based on Darcy's law ignore complex operating conditions such as concentration polarization and membrane pore blockage. Prediction errors exceed ±15% in low-flow, high-concentration environments, making them difficult to meet the demands of sophisticated scenarios such as industrial wastewater and seawater desalination. While data-driven models can capture nonlinear relationships, they are limited by their reliance on massive amounts of historical data. Accuracy plummets to below 60% when encountering sudden changes in water quality, such as organic impact. Furthermore, model reuse across processes (such as reverse osmosis and ultrafiltration) is insufficient, significantly increasing R&D costs. Traditional fixed-threshold alarm mechanisms fail to account for parameter sensitivity drift caused by membrane aging. The same contamination index (PI) can correspond to actual contamination differences of more than three times, leading to water quality issues due to over-cleaning (shortening membrane life by 20%-30%) or untimely cleaning, systematically reducing monitoring efficiency.

[0003] The industry's technical bottlenecks are further highlighted as three difficulties: the phenomenon of multi-source data silos is widespread, sensor parameters (pressure, flow, concentration) and operation and maintenance records (cleaning cycles, membrane material parameters) are stored in a decentralized manner, making it difficult to explore the temporal coupling relationship between parameters such as temperature and turbidity, which restricts the global analysis capabilities of complex working conditions; the static weight distribution model performs poorly under dynamic working conditions such as the rainy season. For example, when the turbidity of the inlet water suddenly increases by 20%, the traditional sliding window averaging method causes the pollution index (PI) calculation error to exceed 20%; edge scenarios (such as remote water plants) face the challenge of data scarcity. When the amount of historical data is less than 1,000, the accuracy of the machine learning model drops sharply to below 70%, seriously hindering the process of intelligent transformation. Summary of the Invention

[0004] In order to solve the above problems, the purpose of the present invention is to provide an intelligent membrane fouling detection and dynamic early warning system based on machine learning and multi-source data fusion, aiming to solve the problems of pollution monitoring relying on a single parameter, early warning lag and extensive control strategy in traditional membrane separation processes.

[0005] To achieve the above technical objectives, this application provides an intelligent detection and dynamic early warning system based on machine learning and multi-source data fusion, including:

[0006] The data acquisition and preprocessing module is used to collect membrane flux, transmembrane pressure difference, solution concentration, temperature, inlet flow rate and operating pressure during the membrane separation process, and integrate historical operation data to form a data set;

[0007] The data analysis module is used to simulate the pollution trend based on the data set by setting up a physical model according to the membrane fouling dynamics theory, and use the data-driven model to learn nonlinear relationships from historical data through machine learning;

[0008] Adaptive adjustment module, which is used to assign dynamic weight coefficients to physical models and data-driven models through Bayesian optimization or SHAP value analysis, and adjust weights in real time according to membrane aging or water quality mutations;

[0009] The interactive early warning module is used to display real-time PI values, pollution trends and weight changes, and trigger threshold-exceeding alarms through the message queue.

[0010] Preferably, the data analysis module obtains a physical driving index based on a physical model to quantify the direct impact of flow fluctuations and pollution accumulation; and obtains a data-driven index based on a data-driven model to capture non-explicit rules through machine learning and convert the probability output into an exponential form to enhance sensitivity.

[0011] Preferably, the data analysis module obtains a comprehensive pollution index based on the physical driving index and the data driving index, wherein the comprehensive pollution index is expressed as:

[0012]

[0013] Where, : Physical Drive Index The weight reflects the contribution ratio of the physical mechanism; : Data-Driven Index The weight reflects the contribution ratio of the experience learned by the machine learning model from historical data, and PI represents the comprehensive pollution index.

[0014] Preferably, when the data analysis module obtains the physical driving index, the physical driving index is expressed as:

[0015]

[0016] Where, : weight coefficient; : Transmembrane pressure difference corresponding to time point t; : time window; : membrane flux corresponding to time point t; : water inlet flow rate; : corresponds to the solution concentration at time point t; : Initial concentration of the influent solution; where, Quantify the time rate of change of transmembrane pressure difference under unit flow rate, reflecting the cumulative effect of flow resistance caused by membrane fouling; The actual concentration was normalized by the reference concentration to measure the negative correlation between the pollutant retention efficiency and the degree of membrane fouling.

[0017] Preferably, when the data analysis module obtains the data-driven index, the data-driven index is expressed as:

[0018]

[0019] Where, is the sensitivity parameter; is the probability of membrane fouling degree.

[0020] Preferably, the adaptive adjustment module optimizes the physical index parameters through XGBoost grid search 、 ; Using Bayesian optimization, optimize the data index parameters with F1-score as the evaluation index .

[0021] Preferably, the interactive warning module defines graded warning and response triggering of the comprehensive pollution index according to membrane material properties and working conditions:

[0022] Normal operation: When When the system is running, maintain the current working conditions and continue to monitor;

[0023] Light pollution warning: When the hourly warning is triggered, optimization suggestions will be pushed to increase the flow rate or reduce the inlet water concentration;

[0024] Moderate pollution warning: When a fault occurs, a minute-level warning is triggered: local pulse cleaning is performed, the membrane components in the faulty area are shut down and the operation is switched to the backup tank;

[0025] Severe pollution warning: When the membrane tank is damaged, the emergency cleaning instruction is triggered: the chemical cleaning program is started, the membrane tank is drained and manual maintenance is performed.

[0026] Preferably, the adaptive adjustment module dynamically updates the warning threshold through a Bayesian network based on historical operating data and SHAP value analysis to adapt to water quality fluctuations or membrane material aging.

[0027] The present invention discloses the following technical effects:

[0028] The present invention supports full closed-loop management from minute-level early warning to hourly cleaning decision-making, and dynamically adjusts thresholds through Bayesian optimization, significantly improving the system's generalization and anti-interference capabilities. It is suitable for intelligent operation and maintenance of membrane separation processes such as reverse osmosis and ultrafiltration. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 It is a flow chart of the interaction of system modules described in the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0032] like Figure 1 As shown, the present invention provides an intelligent membrane fouling detection and dynamic early warning system based on machine learning and multi-source data fusion. The system uses a multi-sensor network to collect parameters such as membrane flux, transmembrane pressure difference, solution concentration, temperature, inlet flow rate, and operating pressure in real time. Combined with an improved XGBoost machine learning model, it constructs a comprehensive pollution index to quantify the degree of membrane fouling. The model uses feature engineering to achieve dual analysis of pollution mechanisms and data patterns. Specifically, the system includes four core components: data acquisition and preprocessing, physical models and data-driven models, dynamic weight fusion and adaptive mechanisms, and human-computer interaction and alarms.

[0033] Data Acquisition and Preprocessing Module (Data Acquisition and Preprocessing): Serving as the system's data foundation, this module is responsible for real-time acquisition of key parameters such as transmembrane pressure, membrane flux, and inlet concentration, and integrating historical operational data. After acquiring multi-source data through the sensor network, it undergoes cleaning and feature engineering to provide high-quality input for subsequent modeling.

[0034] The data analysis module (physical model and data-driven model) separately explores pollution patterns. The physical model is based on membrane fouling kinetics theory and incorporates calibration parameters such as viscosity and concentration gradient to simulate pollution trends. The data-driven model uses machine learning to learn nonlinear relationships from historical data, enhancing adaptability to complex operating conditions. The outputs of both are dynamically weighted to generate the final pollution index.

[0035] Adaptive adjustment module (dynamic weight fusion and adaptive mechanism part), through Bayesian optimization or SHAP value analysis, the system is a physical model and data-driven models Assign dynamic weight coefficients and , ensuring the combination of the robustness of physical mechanisms and the flexibility of data-driven. At the same time, online learning and anomaly detection technologies are introduced to adjust weights in real time according to membrane aging or water quality mutations to avoid model failure.

[0036] The interactive early warning module (human-computer interaction and alarm section) provides a visual dashboard displaying real-time PI values, pollution trends, and weight changes, and triggers threshold-exceeding alarms through a message queue. Operations and maintenance personnel can develop scientific maintenance strategies based on cleaning cycle recommendations generated by the historical data analysis module.

[0037] The system performs the following detection steps:

[0038] S1. The model collects and processes the following parameters in real time through a multi-sensor network for training: membrane flux ; Transmembrane pressure difference , solution concentration , temperature T, water flow , operating pressure P and time series characteristics ; Analyze data patterns through feature engineering, including nonlinear combination of parameters and dynamic weight allocation.

[0039] S2. Comprehensive pollution index A weighted overlay model is used to combine physical drivers with data-driven features:

[0040]

[0041] Where, : Physical Drive Index The weight reflects the contribution ratio of the physical mechanism; : Data-Driven Index The weight reflects the contribution ratio of the experience learned by the machine learning model from historical data.

[0042] S3, Physical Drive Index It is based on the core physical equations of the membrane separation process and quantifies the direct impact of flow fluctuations and fouling accumulation:

[0043]

[0044] in, : weight coefficient; : Transmembrane pressure difference corresponding to time point t; : time window; : membrane flux corresponding to time point t; : water inlet flow rate; : corresponds to the solution concentration at time point t; : initial concentration of influent solution; Quantify the time rate of change of transmembrane pressure difference under unit flow rate, reflecting the cumulative effect of flow resistance caused by membrane fouling; The actual concentration was normalized by the reference concentration to measure the negative correlation between the pollutant retention efficiency and the degree of membrane fouling.

[0045] S4. Data-driven index It captures non-explicit rules through machine learning and converts probability output into exponential form to enhance sensitivity:

[0046]

[0047] in, : sensitivity parameter; : Probability of membrane fouling degree (range 0 to 1).

[0048] S5. Adaptive fusion of physical and data-driven indices through dual-layer XGBoost optimization:

[0049] First layer: Input features include flow fluctuation rate, pressure difference change rate, concentration gradient, etc. The weight parameters are optimized through the following formula to output , :

[0050]

[0051] Among them, λ represents the soft constraint penalty coefficient, which is used to Balance weight distribution bias when it cannot be fully satisfied.

[0052] Second layer: Optimizing physical index parameters through XGBoost grid search 、 ; Using Bayesian optimization, the sensitivity parameters are optimized with F1-score as the evaluation index .

[0053] Machine learning reconstructs the pollution index formula through a two-layer optimization framework to achieve dynamic coordination between physical parameters and data features: training the XGBoost model based on historical data to automatically optimize the physical driving coefficients in the formula 、 and time window For example, in a reverse osmosis system, the model is determined by grid search =0.7, =0.3, The calculation error is reduced from ±12% to ±3%. The lightweight Transformer model is introduced to convert the contamination probability output by XGBoost into ∈[0,1] is mapped to exponential form , the k value is determined by Bayesian optimization to avoid the risk of exponential explosion.

[0054] Define the graded warning and response trigger of the comprehensive pollution index based on membrane material characteristics and operating conditions:

[0055] Normal operation: When When the system is running, maintain the current working conditions and continue to monitor;

[0056] Light pollution warning: When the hourly warning is triggered, optimization suggestions will be pushed to increase the flow rate or reduce the inlet water concentration;

[0057] Moderate pollution warning: When a fault occurs, a minute-level warning is triggered: local pulse cleaning is performed, the membrane components in the faulty area are shut down and the operation is switched to the backup tank;

[0058] Severe pollution warning: When the membrane tank is damaged, the emergency cleaning instruction is triggered: the chemical cleaning program is started, the membrane tank is drained and manual maintenance is performed.

[0059] Based on historical operating data and SHAP value analysis, the warning threshold is dynamically updated through the Bayesian network to adapt to water quality fluctuations or membrane material aging.

[0060] Example 1: Installing a reverse osmosis membrane assembly, wherein the membrane material is an effective area of 0.5 The polyamide composite membrane is equipped with a pressure sensor, flow meter, and COD online monitor. The data set is obtained by running the device: =1~3bar, =5~15 , =50~300 , =20~35℃.

[0061] The optimal parameter combination is determined by grid search. , (MAE of validation set = 3.2%); design two sets of comparative experiments to output validation sets: , The corresponding MAE is 5.8%, and the pressure term is underfitting; , This corresponds to MAE=6.5%, where the concentration term dominates the deviation.

[0062] The XGBoost feature importance ranking is flow rate volatility (32%), pressure difference change rate (28%), and turbidity-TDS correlation (22%), determined by Bayesian optimization. Corresponding F1-score=0.89.

[0063] Input three sets of parameters into the system and output the experimental results: parameter combination , The accuracy of output pollution warning is 92.5%, and the parameter combination , The accuracy of the output pollution warning is 84.3%, and the parameter combination The accuracy rate of output pollution warning is 91.2%. By comparing the data, we can conclude that: physical parameters 、 It needs to be dynamically adjusted according to the working conditions, and the data-driven parameter k can adapt to water quality fluctuations through Bayesian optimization.

[0064] Example 2:

[0065] The apparatus was the same as in Example 1, with the addition of a noise generator and flux decay controller. Interference factors were added: the flow rate was suddenly increased from 5 m³ / h to 10 m³ / h for 30 minutes; sensor noise was injected, and ±10% Gaussian noise was superimposed on the ΔP measurement; membrane aging was simulated by increasing the flux decay rate from 0.5% / h to 1.2% / h.

[0066] The measured data is processed and input into the system to calculate the comprehensive pollution index under the traffic mutation scenario: (Triggers a mild alarm)

[0067] In the scenario where sensor noise is injected, the ΔP error after filtering is controlled within ±1.5%.

[0068] The output experimental results show that in the flow step scenario, the MAE is 2.8%, the false alarm rate is 4.3%, and the missed alarm rate is 3.1%; in the sensor noise scenario, the MAE is 1.2%, the false alarm rate is 2.8%, and the missed alarm rate is 1.9%; in the membrane aging scenario, the MAE is 3.5%, the false alarm rate is 5.6%, and the missed alarm rate is 4.7%. The analysis results show that the system maintains high accuracy (MAE <3%) under sudden flow rate changes; through online filtering and dynamic weight adjustment, the false alarm rate is less than 5%.

[0069] Example 3:

[0070] In order to verify the applicability of the system in different membrane processes such as reverse osmosis (RO) and ultrafiltration (UF), two systems were set up: RO system and UF system. The corresponding data were measured: RO system - seawater desalination plant data (ΔP = 50-80 bar, COD = 300-600 mg / L); UF system - municipal sewage treatment plant data (ΔP = 0.5-5 bar, COD = 50-150 mg / L). The migration method is known to be

[0071] ,

[0072] Using the migration method, the RO system data was migrated to the UF system, resulting in feature weights of the source domain (RO) of 0.4 for flow fluctuation and 0.6 for pressure change. The fine-tuned weights in the target domain (UF) were 0.3 for flow fluctuation and 0.7 for pressure change. Migrating the UF system data to the RO system resulted in feature weights of 0.2 for flow fluctuation and 0.8 for pressure change. The fine-tuned weights in the target domain (RO) were 0.4 for flow fluctuation and 0.6 for pressure change.

[0073] The experimental results from the above steps show that the original model accuracy under RO→UF was 78.6%, and after fine-tuning, it reached 89.2%, with a generalization error reduction of 11.4%. The original model accuracy under UF→RO was 81.4%, and after fine-tuning, it reached 90.7%, with a generalization error reduction of 9.3%. This concludes that the dynamic weight migration mechanism enables the system to maintain high accuracy (accuracy >89%) in cross-process scenarios. Furthermore, since the system fine-tuning time is less than 2 hours, the cost of secondary model development is significantly reduced.

[0074] In summary, the present invention proposes four core technological breakthroughs:

[0075] First, by building a multi-source data fusion platform, we integrated real-time sensor data (ΔP, membrane flux, solution concentration) with historical operation and maintenance data (cleaning records, membrane material parameters), established a unified database, and used feature engineering to extract high-order interactive features such as "turbidity × flow rate" and "pH × temperature", breaking through the limitations of traditional single-parameter analysis.

[0076] The second is to innovate the two-layer dynamic weight optimization mechanism and use the XGBoost model to realize the physical model ( ) and the data model ( )’s weights are dynamically allocated ( 、 ), combined with Bayesian optimization algorithm to calibrate physical parameters in real time ( 、 ) and the data sensitivity parameter ( ), significantly improving the model's adaptability to sudden changes in water quality (such as organic matter impact);

[0077] The third is to develop an intelligent early warning and adaptive control system. Based on SHAP value analysis, dynamic thresholds for pollution levels (normal → mild → moderate → severe) are established. The thresholds are automatically updated through a Bayesian network. An integrated ultrasonic imaging module is used to locate pollution hotspots at the centimeter level (with an accuracy of ±2cm), accurately guiding cleaning decisions.

[0078] Finally, a collaborative edge-cloud architecture was employed. A lightweight model (TensorFlow Lite) was deployed on the edge to ensure minute-level response times. The cloud leveraged the Transformer and PPO algorithms to train high-precision models and regularly push policy updates, forming a closed-loop system encompassing data collection, model training, and decision execution. Experimental verification demonstrated that this system controlled the error in pollution index calculation to within ±1.5%-3%, improved early warning timeliness by 300%, and extended membrane life by over 50%, providing a full-lifecycle intelligent solution for membrane separation processes under complex operating conditions.

[0079] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0081] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. Intelligent detection and dynamic early warning system based on machine learning and multi-source data fusion, characterized by: include: The data acquisition and preprocessing module is used to collect membrane flux, transmembrane pressure difference, solution concentration, temperature, inlet flow rate and operating pressure during the membrane separation process, and integrate historical operation data to form a data set; A data analysis module is used to simulate the fouling trend based on the data set by setting a physical model according to the membrane fouling kinetics theory, and to learn nonlinear relationships from historical data through machine learning using a data-driven model; An adaptive adjustment module, configured to assign dynamic weight coefficients to the physical model and the data-driven model through Bayesian optimization or SHAP value analysis, and to adjust the weights in real time according to membrane aging or water quality mutation; Interactive early warning module, used to display real-time PI values, pollution trends and weight changes, and trigger threshold exceeding alarms through message queues; The data analysis module obtains a physical driving index based on the physical model to quantify the direct impact of flow fluctuations and pollution accumulation; obtains a data-driven index based on the data-driven model to capture non-explicit regularities through machine learning and convert the probability output into an exponential form to enhance sensitivity; The data analysis module obtains a comprehensive pollution index based on the physical driving index and the data driving index, wherein the comprehensive pollution index is expressed as: Where, : Physical Drive Index The weight reflects the contribution ratio of the physical mechanism; : Data-Driven Index The weight reflects the contribution ratio of the experience learned by the machine learning model from historical data. PI represents the comprehensive pollution index. When the data analysis module obtains the physical driving index, the physical driving index is expressed as: Where, : weight coefficient; : Transmembrane pressure difference corresponding to time point t; : time window; : membrane flux corresponding to time point t; : water inlet flow rate; : corresponds to the solution concentration at time point t; : Initial concentration of the influent solution; where, Quantify the time rate of change of transmembrane pressure difference under unit flow rate, reflecting the cumulative effect of flow resistance caused by membrane fouling; The actual concentration was normalized by the reference concentration to measure the negative correlation between the pollutant retention efficiency and the degree of membrane fouling.

2. The intelligent detection and dynamic early warning system based on machine learning and multi-source data fusion according to claim 1 is characterized by: When the data analysis module obtains the data-driven index, the data-driven index is expressed as: Where, is the sensitivity parameter; is the probability of membrane fouling degree.

3. The intelligent detection and dynamic early warning system based on machine learning and multi-source data fusion according to claim 2 is characterized by: The adaptive adjustment module optimizes the physical index parameters through XGBoost grid search 、 ; Using Bayesian optimization, optimize the data index parameters with F1-score as the evaluation index .

4. The intelligent detection and dynamic early warning system based on machine learning and multi-source data fusion according to claim 3 is characterized by: The interactive warning module defines the hierarchical warning and response triggering of the comprehensive pollution index according to the membrane material characteristics and working conditions: Normal operation: When When the system is running, maintain the current working conditions and continue to monitor; Light pollution warning: When the hourly warning is triggered, optimization suggestions will be pushed to increase the flow rate or reduce the inlet water concentration; Moderate pollution warning: When a fault occurs, a minute-level warning is triggered: local pulse cleaning is performed, the membrane components in the faulty area are shut down and the operation is switched to the backup tank; Severe pollution warning: When the membrane tank is damaged, the emergency cleaning instruction is triggered: the chemical cleaning program is started, the membrane tank is drained and manual maintenance is performed.

5. The intelligent detection and dynamic early warning system based on machine learning and multi-source data fusion according to claim 4 is characterized by: The adaptive adjustment module dynamically updates the warning threshold through the Bayesian network based on historical operating data and SHAP value analysis to adapt to water quality fluctuations or membrane material aging.

Images