Multi-modal time series data prediction system and use method thereof

Abstract

The invention relates to the technical field of pH value prediction systems, and particularly discloses a prediction system based on multi-modal time sequence data and a use method of the prediction system. The edge end equipment comprises a multi-modal data acquisition module, a data preprocessing module and a prediction model module; the edge end equipment comprises a multi-modal data acquisition module and a data preprocessing module; the multi-modal data acquisition module is used for acquiring multi-modal data; the data preprocessing module is used for eliminating abnormal values and standardizing multi-source data to obtain historical sequence data; the prediction model module comprises an input layer, an LSTM (Long Short Term Memory) layer, a Transform layer and a fusion layer. According to the method, the defects of single data dimension, obvious real-time bottleneck, mechanism fusion and insufficient anti-interference capability are effectively overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of pH value prediction systems, and in particular to a prediction system based on multimodal time series data and a method for using the same. Background Art

[0002] Currently, conventional chemical wastewater pH control relies on a single pH sensor with a hysteresis feedback control system. The core process involves real-time pH measurement using a pH electrode, which transmits the data to a controller for comparison with a preset target value. The controller then calculates the deviation and outputs a signal to control a dosing pump to adjust the dosage of acid and alkali reagents. While this technology is widely used in wastewater treatment applications across industries such as chemical, pharmaceutical, and electroplating, with the core goal of stabilizing wastewater pH within the discharge standard range of 6-9, it has significant drawbacks in practice. A single sensor cannot accurately capture the spatial distribution of pH within a reaction tank (for example, pH deviations between the center and edge of a large reaction tank can reach ±0.8). Furthermore, the hysteresis feedback mechanism (a cumulative delay of 2-5 seconds in sensor response, signal transmission, and execution) struggles to cope with the strong nonlinear characteristics of water quality fluctuations, such as sudden pH changes caused by the buffer system. Control accuracy can drop significantly to within ±0.5 pH, especially under extreme conditions such as ±30% influent flow rate fluctuations and sudden COD concentration changes. This leads to a 15%-25% waste rate.

[0003] In addition, although some studies have attempted to introduce deep learning models such as LSTM for pH prediction, significant technical bottlenecks still exist: the model only models single-modal pH data, ignoring the coupling effects of key influencing factors such as COD, turbidity, and temperature (for example, every 100 mg / L increase in COD can lead to an average decrease in pH of 0.15), and the prediction error is more than 40% higher than that of multi-modal models; the model is deployed in the cloud, resulting in a round-trip data delay of more than 1 second, which cannot meet the sub-second response requirements of industrial sites (for example, the pharmaceutical industry requires a control cycle of ≤500ms), and the instability of network transmission further exacerbates the unreliability of the prediction results. These technical shortcomings make it difficult for traditional control solutions to achieve stable and standard emissions in complex chemical wastewater treatment scenarios. There is an urgent need to break through the existing bottlenecks through multi-modal data fusion and edge intelligent computing technology.

[0004] For example, the patent document "Method and system for controlling pH value based on mathematical AI modeling" with publication number CN118897585A discloses an AI-based drug dosing model. Its core is to use historical pH value, drug dosage and water inflow data, and combine the gated recurrent unit with the fully connected layer training model to realize real-time calculation of drug dosage to adjust the pH value. This patent has three defects: first, it relies on single-modal data. The model input only covers pH value, water inflow and drug dosage. It does not integrate multi-modal data such as temperature, turbidity, and conductivity. It is difficult to capture the nonlinear coupling relationship between complex water quality parameters, such as the dynamic effect of microbial activity on pH; second, there is a lack of edge computing. The model deployment relies on cloud computing. There is no mention of localized reasoning of edge devices. The real-time performance is limited by network delay (>1 second), which cannot meet the sub-second industrial control requirements; third, the physical mechanism is not sufficiently integrated. Although historical data training is introduced, physical constraints such as reaction kinetics equations or microbial metabolic pathways are not embedded, and the model's interpretability and generalization capabilities are limited.

[0005] The paper "Trapezoidal LSTM endpoint pH value model predictive control based on process characteristics of leaching process" (Chen Yu, Liu Xuebin, Lao Jiafeng, 2023, Chemical Automation and Instrumentation, Vol. 50, No. 8, 789-796, 10.11234 / huagong.2023.08.005) discloses the difficulties of strong nonlinearity, large hysteresis and inlet material composition fluctuation in pH value control in zinc smelting leaching process. This study proposes a trapezoidal long short-term memory network-model predictive control (Trapezoidal L The STM-MPC) algorithm achieved technological breakthroughs through deep integration of physical properties, dynamic optimization control strategies, and industrial scenario verification: a trapezoidal LSTM structure was designed to simulate the physical characteristics of a cascade reactor, enhancing the model's ability to characterize and explain process mechanisms and resolving the disconnection between traditional black-box models and physical processes; a model predictive control framework based on a particle swarm algorithm was constructed, and the dosing strategy was continuously optimized to compensate for inlet fluctuations. Simulations showed that under conditions of sudden changes in inlet pH or fluctuations in the solid-liquid ratio, overshoot was reduced by 51.2% and adjustment time was shortened by 57.4%. This paper has three defects: First, the single-modal data has significant limitations. The model mainly relies on process parameters such as pH time series data and reactor flow rate, and does not integrate multi-source sensor data such as dissolved oxygen and metal ion concentration, resulting in insufficient adaptability in multi-factor coupling scenarios such as microbial metabolic interference; First, the single-modal data has significant limitations. The model mainly relies on process parameters such as pH time series data and reactor flow rate, and does not integrate multi-source sensor data such as dissolved oxygen and metal ion concentration, resulting in insufficient adaptability in multi-factor coupling scenarios such as microbial metabolic interference; Second, the real-time performance is difficult to meet industrial needs. The model is deployed on an industrial computer or the cloud, and does not use edge computing chips such as FPGA for lightweight acceleration, making the sub-second response target difficult to achieve; Third, the anti-interference ability is limited. Although MPC optimization control is adopted, there is a lack of dynamic weight distribution or quality perception fusion strategy. The system has poor robustness when facing sudden changes in water quality (such as shock loads).

[0006] In summary, there are three common defects: First, the data dimension is single, the model is overly dependent on single-mode or limited process parameters such as pH value and flow rate, and does not integrate multi-source sensor data such as temperature, turbidity, dissolved oxygen, and metal ion concentration. It is difficult to analyze the nonlinear coupling relationship between complex water quality parameters such as microbial activity and metabolic interference, resulting in insufficient adaptability in multi-factor coupling scenarios; second, there is a significant real-time bottleneck, the model deployment relies on the cloud or industrial computer, and lacks the lightweight acceleration design and localized reasoning capabilities of edge computing chips such as FPGA. The network delay is generally greater than 1 second, which cannot meet the sub-second control requirements of industrial sites; third, the mechanism fusion and anti-interference capabilities are insufficient. The patent does not embed physical constraints such as reaction kinetics equations, and the paper lacks a dynamic weight distribution strategy. Neither of them constructs a data-mechanism collaborative model, resulting in limited model interpretability, generalization ability, and robustness to sudden changes in water quality (such as shock loads). Summary of the Invention

[0007] In response to the above problems, the present invention proposes a prediction system based on multimodal time series data, which solves the defects of the existing data, such as single data dimension, significant real-time bottleneck, insufficient mechanism fusion and anti-interference capabilities.

[0008] The technical solution adopted by the present invention is as follows:

[0009] A prediction system based on multimodal time series data, comprising: an edge device; the edge device comprising a multimodal data acquisition module, a data preprocessing module, and a prediction model module; the edge device comprising a multimodal data acquisition module and a data preprocessing module; the multimodal data acquisition module is used to acquire multimodal data; the data preprocessing module is used to remove outliers and standardize multi-source data to obtain historical sequence data;

[0010] The prediction model module includes an input layer, an LSTM layer, a Transformer layer and a fusion layer; wherein the input layer is used to receive historical sequence data from multiple modal sensors, the LSTM layer is used to analyze the pH historical sequence from both forward and reverse directions simultaneously, the Transformer layer is used to utilize a multi-head self-attention mechanism to capture the global correlation between multimodal data from different angles in parallel, and the fusion layer is used to splice the outputs of the LSTM and Transformer, integrate the feature information extracted by the two models, and then perform feature fusion and dimensionality reduction through a fully connected layer to output the pH prediction value.

[0011] The present invention integrates LSTM and Transformer to construct a closed loop of "multimodal data fusion-deep learning prediction-edge intelligent control", in which LSTM is used to capture the long-term fluctuation law of the pH series to analyze the time series dependency, and Transformer is responsible for analyzing the real-time impact of variables such as COD and flow on pH to explore cross-parameter correlations. The combination of the two realizes in-depth processing of multimodal data; at the same time, it is lightweight deployed to the edge through TensorRT, and the control parameters are dynamically adjusted in combination with the PSO-GA hybrid algorithm to form a collaborative optimization mechanism, ultimately achieving high-precision control of the pH value of ±0.2 units, which is 70% lower than the steady-state error of the traditional PID scheme, effectively solving the problems of strong nonlinearity, large time lag and multivariable coupling in the pH control of chemical wastewater.

[0012] Optionally, the multimodal data acquisition module includes a pH sensor, a COD sensor, a turbidity sensor, a temperature sensor and a flow sensor.

[0013] Optionally, the prediction model module is used to quantize the model using the TensorRT tool to convert the model parameters from FP32 to INT8.

[0014] Optionally, the LSTM layer adopts a multi-layer bidirectional LSTM unit, and each layer has multiple neurons.

[0015] Optionally, a cloud device is also included, which includes a fuzzy PID controller. The cloud device is also used to input the pH prediction result obtained by the prediction model as a feedforward signal into the PSO-GA optimized fuzzy PID controller.

[0016] Optionally, the cloud device includes a storage module, and the storage module has an InfluxDB time series database.

[0017] Optionally, the cloud device is used to push optimized model parameters to the edge end through over-the-air download technology to achieve remote updating of the model.

[0018] Optionally, the cloud device periodically retrains the model using the full amount of historical data to update the hidden state parameters of the LSTM layer.

[0019] Optionally, the cloud device also has a visual interface for displaying prediction accuracy indicators.

[0020] The present invention also discloses a method for using a prediction system based on multimodal time series data, comprising the following steps:

[0021] S1. Collect multimodal data using a multimodal data acquisition module;

[0022] S2. Use a window of several seconds to remove outliers from pH data and standardize multi-source data to obtain historical sequence data;

[0023] S3. Use the LSTM layer to analyze the pH history sequence from both the forward and reverse directions to capture the long-term dependencies of the pH sequence and obtain training and test datasets.

[0024] S4. Using the multi-head self-attention mechanism, we can capture the global correlation between multimodal data from different perspectives in parallel and quantify the influence of different variables on pH.

[0025] S5. Perform an optimization evaluation on the prediction model. If the predicted model is optimal, use the test algorithm to test it, then evaluate the prediction accuracy, and finally output the prediction effect. If the predicted model is not optimal, retrain the model using the training data set and update the hidden state parameter judgment of the LSTM layer.

[0026] The beneficial effects of the present invention include at least:

[0027] 1. In complex scenarios such as industrial wastewater treatment, pH value is affected by multiple factors, and a single data modality is difficult to accurately characterize its changing pattern. The present invention integrates multimodal data such as pH, COD, turbidity, temperature, and flow, and utilizes the powerful feature extraction capabilities of the LSTM / Transformer model to deeply explore the nonlinear coupling relationship between variables. The LSTM model is good at capturing long-term dependency information in time series, while the Transformer model can effectively handle complex correlations between different variables. By combining the two, pH value changes can be accurately predicted. The goal is to reduce the pH prediction error to within ±0.1pH, specifically measured by the root mean square error (RMSE), requiring RMSE≤0.121pH, and achieving a prediction lead of 30 seconds, thereby providing sufficient reaction time for process adjustments in industrial production and avoiding product quality problems or equipment failures due to abnormal pH values.

[0028] 2. The water quality in industrial production is complex and changeable, with interference factors such as large fluctuations in water flow (such as ±50% fluctuations), sudden changes in pH value (such as ±1.5 sudden changes), and drift in sensor performance (such as a 5% annual decrease in the slope of the pH electrode). The present invention addresses these challenges through online data calibration and model adaptive update mechanisms. Online data calibration monitors sensor data in real time, promptly discovers and corrects abnormal data; the model adaptive update mechanism fine-tunes the model based on real-time data, enabling it to quickly adapt to changes in water quality. Through these measures, it is ensured that under various complex working conditions, the long-term prediction error is stable within ±0.15pH, ensuring that the system always maintains high prediction accuracy and reliability during long-term operation.

[0029] 3. The present invention innovatively integrates LSTM and Transformer to construct a closed loop of "multimodal data fusion-deep learning prediction-edge intelligent control", in which LSTM is used to capture the long-term fluctuation pattern of the pH series to analyze the time series dependency, and Transformer is responsible for analyzing the real-time impact of variables such as COD and flow on pH to explore cross-parameter correlations. The combination of the two enables in-depth processing of multimodal data; at the same time, it is lightweight deployed to the edge through TensorRT, and the control parameters are dynamically adjusted in combination with the PSO-GA hybrid algorithm to form a collaborative optimization mechanism, ultimately achieving high-precision control of the pH value of ±0.2 units, which is 70% lower than the steady-state error of the traditional PID scheme, effectively solving the problems of strong nonlinearity, large time lag and multivariable coupling in the pH control of chemical wastewater.

[0030] 4. Compared with the single LSTM model, the prediction determination coefficient (R 2) increased from 0.8 to 0.92, and the root mean square error (RMSE) decreased by 18% (from 0.15pH to 0.12pH). Especially in the water quality multivariate coupling scenario (such as COD fluctuation ±20% + flow mutation ±15%), the prediction accuracy was significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of sample processing and model optimization based on LSTM in the prediction system based on multimodal time series data of the present invention;

[0032] Figure 2 This is a schematic diagram of the LSTM-driven data preprocessing, model training, and prediction process of the prediction system based on multimodal time series data of the present invention.

[0033] Figure 3 This is an analytical diagram of the control system architecture of the prediction system based on multimodal time series data of the present invention; DETAILED DESCRIPTION

[0034] The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0035] The Long Short-Term Memory Network (LSTM) in this paper is a special recurrent neural network (RNN) designed to address the vanishing and exploding gradient problems faced by traditional RNNs when processing long sequences of data. Its core structure comprises a memory unit consisting of an input gate, a forget gate, and an output gate. It selectively stores and updates long-term dependency information, effectively capturing long-term correlation features in time series.

[0036] The Transformer in this invention is a deep learning architecture based on the self-attention mechanism (Self-Attention), which was originally used in the field of natural language processing. Due to its advantages in long sequence modeling and parallel computing, it quickly became the core framework for a variety of complex tasks. Its core idea is to dynamically calculate the association weights of each element in the sequence with other elements through the self-attention mechanism, thereby capturing global dependencies and breaking through the sequence dependency limitations of traditional recurrent neural networks (RNNs). The Transformer architecture is mainly composed of an encoder (Encoder) and a decoder (Decoder). The encoder is responsible for mapping the input sequence into context-aware feature vectors, and the decoder generates a target sequence based on these vectors. This architecture not only significantly improves the efficiency and accuracy of long text processing, but also solves the problem of representing sequence order information through the positional encoding mechanism, and supports parallel training to accelerate convergence.

[0037] The PSO-GA in the present invention is a hybrid optimization strategy that integrates the particle swarm optimization algorithm (PSO) and the genetic algorithm (GA), aiming to combine the advantages of the two to improve the global search capability and convergence accuracy. Its core features and mechanisms are as follows: PSO characteristics: It converges quickly through particle tracking of individual and group historical optimal solutions, but it is easy to fall into local optimality. GA characteristics: It uses selection, crossover, and mutation operations to maintain population diversity and enhance global search capabilities, but the convergence speed is slow. Hybrid strategy: Embed the crossover and mutation operations of GA into the PSO framework, or introduce GA operations after PSO iteration to break the local optimal deadlock. Serial hybrid: First execute PSO to generate a high-quality initial population, and then use GA optimization (such as crossover and mutation) to improve the quality of the solution. Parallel hybrid: Divide the population into two groups, one group is updated according to PSO, and the other group receives GA operations, and finally merges and evolves.

[0038] TensorRT in this invention is a high-performance deep learning inference optimization library and runtime engine launched by NVIDIA, designed specifically for NVIDIA GPUs. Its core goal is to improve inference efficiency through model compression, computational graph optimization, and hardware acceleration.

[0039] Example 1

[0040] The technical solution adopted by the present invention is as follows:

[0041] like Figure 1 、 Figure 2 As shown, the present invention discloses a prediction system based on multimodal time series data, including: an edge device and a cloud device. The edge device includes a multimodal data acquisition module, a data preprocessing module, and a prediction model module; the multimodal data acquisition module is used to collect multimodal data; the data preprocessing module is used to remove outliers and standardize multi-source data to obtain historical sequence data.

[0042] Industrial sites have extremely high requirements for the real-time prediction and control of pH values. Traditional cloud computing models have problems such as network latency and are difficult to meet the requirements. In this embodiment, the present invention uses the domestically produced RK3568 edge computing platform to deploy a lightweight model. On the one hand, by compressing and optimizing the model, the model parameters and the amount of calculation are reduced; on the other hand, the hardware acceleration capabilities of the RK3568 platform are utilized to achieve fast reasoning. The goal is to reduce the prediction delay to less than 500ms, achieving the sub-second response requirements of industrial sites, so that the system can respond to changes in pH values ​​in a timely manner, quickly adjust relevant process parameters, and ensure the stability and continuity of the production process.

[0043] In this embodiment, the specific operations of the data preprocessing module include:

[0044] 1) Sliding window filtering: A 5-second window is used to remove abnormal values ​​(such as jump values ​​exceeding 3σ) from the pH data. The formula is:

[0045] 2) Normalization: Normalize multi-source data using the following formula: Among them, μ is the mean of historical data and σ is the standard deviation.

[0046] The prediction model module includes an input layer, an LSTM layer, a Transformer layer and a fusion layer; wherein the input layer is used to receive historical sequence data from multiple modal sensors, the LSTM layer is used to analyze the pH historical sequence from both forward and reverse directions simultaneously, the Transformer layer is used to utilize a multi-head self-attention mechanism to capture the global correlation between multimodal data from different angles in parallel, and the fusion layer is used to splice the outputs of the LSTM and Transformer, integrate the feature information extracted by the two models, and then perform feature fusion and dimensionality reduction through a fully connected layer to output the pH prediction value.

[0047] In this embodiment, the specific model architecture is as follows:

[0048] 1) Input layer: Receives historical sequence data from five modal sensors (pH, COD, turbidity, temperature, and flow). The window length is set to 100 (i.e., collecting data for the past 100 seconds) and the time step is 1 second. This setting fully captures the changing trends of each parameter within a certain time range, providing rich information for subsequent model analysis.

[0049] 2) LSTM layer: This layer uses two layers of bidirectional LSTM units, each with 128 neurons. This bidirectional LSTM can analyze the pH sequence from both the forward and reverse directions, more comprehensively capturing the long-term dependencies of the pH sequence. For example, it can learn the changing patterns of pH values ​​over different time periods and their potential connections with other parameters. The output dimension is 64, compressing the features extracted by the LSTM layer for easier processing.

[0050] 3) Transformer layer: Utilizing a multi-head self-attention mechanism (number of heads = 4), it can simultaneously capture global correlations between multimodal data from different perspectives. For example, it can simultaneously analyze complex relationships between different parameter combinations such as COD and pH, temperature and flow, and model the interactions of multimodal data. The output dimension is also 64.

[0051] 4) Fusion Layer: In the fusion layer, the long-term dependency features of the pH sequence captured by the LSTM layer are combined with the global correlation features of the multimodal data mined by the Transformer layer. By integrating the temporal features and cross-parameter coupling features extracted by the two models, a joint feature vector containing multi-dimensional information is formed. Subsequently, a fully connected layer performs nonlinear transformation and dimensionality reduction on the joint feature vector, compressing the feature space while retaining key predictive information, ultimately achieving an accurate prediction output of the pH value for the next 30 seconds.

[0052] The LSTM layer uses multi-layer bidirectional LSTM units, and each layer has multiple neurons.

[0053] In this embodiment, the multimodal data acquisition module includes a pH sensor, a COD sensor, a turbidity sensor, a temperature sensor, and a flow sensor.

[0054] The pH electrode has an accuracy of ±0.05 pH and features ultrasonic self-cleaning. The COD sensor uses UV spectrophotometry with a range of 0-2000 mg / L. The turbidity sensor uses a scattering method with a range of 0-1000 NTU. The temperature sensor uses a PT1000 with an accuracy of ±0.5°C. The electromagnetic flowmeter has an accuracy of ±0.5% FS and achieves simultaneous data acquisition in seconds.

[0055] In this embodiment, all sensor interfaces support the Modbus RTU / ASCII protocol, a widely used communication protocol in the industrial field with advantages such as stable communication and strong anti-interference capabilities. Sensor data is connected to the edge computing gateway via the RS485 bus. The RS485 bus supports multi-point communication, enabling long-distance, high-speed data transmission, meeting the needs of multi-sensor data aggregation in industrial sites.

[0056] The prediction model module uses the TensorRT tool to quantize the model, converting model parameters from FP32 (32-bit floating point) to INT8 (8-bit integer). This process compresses model parameters by 40%, significantly reducing model storage space and computational complexity. It also reduces inference latency from 800ms to 450ms, improving model inference speed.

[0057] In this embodiment, the edge computing gateway uses the RK3568 SoC, which has a built-in NPU (neural network processing unit) with a computing power of 1.0TOPS, which can provide hardware acceleration for model reasoning. Running the Ubuntu Core system, model reasoning is implemented in C++, fully leveraging C++'s efficiency and direct control over hardware resources to ensure that the system can complete model reasoning in sub-second time, meeting the stringent real-time requirements of industrial sites.

[0058] In this embodiment, the edge receives pre-processed multimodal data in real time, rapidly executes lightweight model inference, and outputs the resulting pH prediction results to the control layer. This process ensures timely data processing and prediction, providing a basis for subsequent control decisions.

[0059] In this embodiment, the edge terminal further includes a dosing execution device, which is connected to the cloud device.

[0060] When the edge detects a prediction error exceeding ±0.2 pH for five consecutive minutes, it indicates that the current model may not be adapting to changes in operating conditions, such as water quality. At this point, an online fine-tuning mechanism is triggered, updating the model parameters using the latest hour's data. This fine-tuning mechanism, based on real-time error feedback, enables the model to quickly adapt to environmental changes and maintain high prediction accuracy.

[0061] The cloud device also includes a fuzzy PID controller, and the cloud device is further used to input the pH prediction result obtained by the prediction model as a feedforward signal into the PSO-GA optimized fuzzy PID controller.

[0062] In this embodiment, the specific operation is: the pH prediction result obtained by the prediction model is input as a feedforward signal into the PSO-GA optimized fuzzy PID controller. The combination of PSO (particle swarm optimization algorithm) and GA (genetic algorithm) can optimize the parameters of the fuzzy PID controller \(K_p\) (proportional coefficient), \(K_i\) (integral coefficient), \(K_d\) (differential coefficient) in real time, so that the controller has better adaptability and control performance. When the predicted pH value exceeds the target range of ±0.2 in the next 30 seconds, it means that abnormal pH fluctuations may occur, and the system adjusts the dosing pump frequency 10 seconds in advance. Adjustment amount calculation formula In the formula, e is the prediction deviation. The appropriate frequency adjustment of the dosing pump is calculated by this formula, and the dosing amount is adjusted in advance to stabilize the pH value and prevent the pH value from exceeding the target range, thereby achieving intelligent and precise control of the pH value.

[0063] The cloud device includes a storage module, and the storage module has an InfluxDB time series database.

[0064] The specific operation in this embodiment is as follows: At midnight each day, the cloud device trains a global model using seven days of historical data stored in the InfluxDB time series database. InfluxDB is a database specifically designed for processing time series data. It offers efficient data storage and query capabilities, making it suitable for storing large amounts of time series data collected by sensors. By training on data over a longer period of time, long-term patterns and trends in the data can be discovered, resulting in a more generalizable global model.

[0065] In this embodiment, the cloud device is used to push the optimized model parameters to the edge end through over-the-air download technology to achieve remote updating of the model.

[0066] The specific operation in this embodiment is: the cloud device pushes the optimized model parameters to the edge end through OTA (over-the-air download technology) to achieve remote update of the model. At the same time, the cloud device provides a visual interface to display the prediction accuracy indicators, such as R 2 (coefficient of determination, which measures the goodness of fit of the model to the data), RMSE (root mean square error, which reflects the degree of deviation between the predicted value and the true value), etc. In addition, it also supports manual intervention to adjust model hyperparameters, making it convenient for operators to optimize the model according to actual conditions.

[0067] The cloud device regularly retrains the model using the full amount of historical data and updates the hidden state parameters of the LSTM layer.

[0068] In another embodiment, the cloud device further has a visualization interface for displaying the prediction accuracy index.

[0069] The embodiment implements hierarchical modeling of the relationship between pH time series characteristics and multi-parameter coupling by constructing an LSTM-Transformer cascade network:

[0070] 1) Bottom LSTM layer: Captures the long-term dependency characteristics of the pH sequence (e.g., autocorrelation coefficient > 0.8), filters noise through a gating mechanism, and extracts the temporal evolution of pH values ​​(e.g., periodic pH fluctuations in the early morning hours due to production process changes).

[0071] 2) Upper Transformer layer: Utilizing a multi-head self-attention mechanism, the model analyzes nonlinear correlations between multiple parameters (e.g., the correlation coefficient between COD and pH is -0.72, indicating a strong negative correlation between increasing COD and decreasing pH) and quantifies the weight of the impact of different variables (turbidity, temperature, and flow rate) on pH. For example, when flow rate changes suddenly, the Transformer can identify its impact on the diffusion efficiency of the drug, correcting the bias of the LSTM's single time series prediction.

[0072] The present invention uses TensorRT to quantize and optimize the Transformer-LSTM fusion model. Through weight pruning (parameter reduction of 35%) and operator fusion (such as merging convolution and batch normalization layers), the model size is compressed from 128MB to 76MB.

[0073] This invention relies on the built-in NPU (neural network processing unit) of the RK3568 chip to achieve hardware-level acceleration of model reasoning, reducing the single-step reasoning time from 800ms on the cloud CPU to 450ms on the edge, fully meeting the industrial control requirements for sub-second response (delay <500ms).

[0074] This invention solves the network delay bottleneck of traditional cloud solutions. In a test at a chemical site, the edge control closed-loop cycle was shortened to 1.2 seconds (including data acquisition, prediction, and instruction execution), an improvement of more than 50% compared to the original system (cloud + PLC), and effectively suppressed pH overshoot (overshoot was reduced from 1.5pH to 0.5pH).

[0075] Example 2:

[0076] like Figure 3 As shown, the present invention also discloses a method for using a prediction system based on multimodal time series data, comprising the following steps:

[0077] S1. Collect multimodal data using a multimodal data acquisition module;

[0078] S2. Use a window of several seconds to remove outliers from pH data and standardize multi-source data to obtain historical sequence data;

[0079] S3. Use the LSTM layer to analyze the pH history sequence from both the forward and reverse directions to capture the long-term dependencies of the pH sequence and obtain training and test datasets.

[0080] S4. Using the multi-head self-attention mechanism, we can capture the global correlation between multimodal data from different perspectives in parallel and quantify the influence of different variables on pH.

[0081] S5. Perform an optimization evaluation on the prediction model. If the predicted model is optimal, use the test algorithm to test it, then evaluate the prediction accuracy, and finally output the prediction effect. If the predicted model is not optimal, retrain the model using the training data set and update the hidden state parameter judgment of the LSTM layer.

[0082] In this embodiment, when a sudden change in water quality is detected (such as a sudden change in pH of ±1.0 due to an impact load), a local parameter update is triggered based on real-time data, and the Transformer attention weight is quickly adjusted through the PSO algorithm. The prediction error can be converged from 0.3pH to within 0.15pH within 20 iteration cycles.

[0083] In this embodiment, the model is retrained regularly (e.g., weekly) using the full amount of historical data (including seasonal patterns of water quality changes) to update the hidden state parameters of the LSTM layer, thereby solving the problem of model drift caused by slow time-varying factors such as the accumulation of buffer substances during long-term operation (e.g., the impact of lower water temperature in winter on chemical reaction rates).

[0084] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0085] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0086] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0088] The above description is only a preferred embodiment of the present invention and does not limit the scope of patent protection of the present invention. Any equivalent structural transformation made by using the contents of the description and drawings of the present invention, directly or indirectly applied to other related technical fields, is also included in the scope of protection of the present invention.

Claims

1. A prediction system based on multimodal time series data, characterized in that: include: Edge device; the edge device includes a multimodal data acquisition module, a data preprocessing module and a prediction model module; The multimodal data acquisition module is used to collect multimodal data; the data preprocessing module is used to eliminate outliers and standardize multi-source data to obtain historical sequence data; The prediction model module includes an input layer, an LSTM layer, a Transformer layer and a fusion layer; wherein the input layer is used to receive historical sequence data from multiple modal sensors, the LSTM layer is used to analyze the pH historical sequence from both forward and reverse directions simultaneously, the Transformer layer is used to utilize a multi-head self-attention mechanism to capture the global correlation between multimodal data from different angles in parallel, and the fusion layer is used to splice the outputs of the LSTM and Transformer, integrate the feature information extracted by the two models, and then perform feature fusion and dimensionality reduction through a fully connected layer to output the pH prediction value.

2. The prediction system based on multimodal time series data according to claim 1, characterized in that: The multimodal data acquisition module includes a pH sensor, a COD sensor, a turbidity sensor, a temperature sensor and a flow sensor.

3. The prediction system based on multimodal time series data according to claim 1, characterized in that: The prediction model module is used to quantize the model using the TensorRT tool and convert the model parameters from FP32 to INT8.

4. The prediction system based on multimodal time series data according to claim 1, characterized in that: The LSTM layer uses multi-layer bidirectional LSTM units, and each layer has multiple neurons.

5. A prediction system based on multimodal time series data according to claim 1, 2, 3 or 4, characterized in that: It also includes a cloud device, which includes a fuzzy PID controller. The cloud device is also used to input the pH prediction result obtained by the prediction model as a feedforward signal into the PSO-GA optimized fuzzy PID controller.

6. The prediction system based on multimodal time series data according to claim 5, characterized in that: The cloud device includes a storage module, and the storage module has an InfluxDB time series database.

7. The prediction system based on multimodal time series data according to claim 5, characterized in that: The cloud device is used to push the optimized model parameters to the edge end through over-the-air download technology to achieve remote updating of the model.

8. The prediction system based on multimodal time series data according to claim 5, characterized in that: The cloud device regularly retrains the model using the full amount of historical data and updates the hidden state parameters of the LSTM layer.

9. The prediction system based on multimodal time series data according to claim 5, characterized in that: The cloud device also has a visual interface for displaying prediction accuracy indicators.

10. A method for using a prediction system based on multimodal time series data, the method being applied to the prediction system based on multimodal time series data according to any one of claims 1 to 9, characterized in that: The following steps are involved: S1. Collect multimodal data using a multimodal data acquisition module; S2. Use a window of several seconds to remove outliers from pH data and standardize multi-source data to obtain historical sequence data; S3. Use the LSTM layer to analyze the pH history sequence from both the forward and reverse directions to capture the long-term dependencies of the pH sequence and obtain training and test datasets. S4. Using the multi-head self-attention mechanism, we can capture the global correlation between multimodal data from different perspectives in parallel and quantify the influence of different variables on pH. S5. Perform an optimization evaluation on the prediction model. If the predicted model is optimal, use the test algorithm to test it, then evaluate the prediction accuracy, and finally output the prediction effect. If the predicted model is not optimal, retrain the model using the training data set and update the hidden state parameter judgment of the LSTM layer.

Images