Water quality full-spectrum absorption monitoring method and system based on Transform architecture

Abstract

The invention relates to the technical field of water quality monitoring, and provides a water quality full-spectrum absorption monitoring method and system based on a Transform architecture aiming at the problems of high response delay, insufficient prediction precision and the like of a traditional method. The method comprises the following steps: acquiring absorption spectrum data in a range of 190-820 nm through a full-band spectrum sensor, preprocessing the absorption spectrum data, inputting the preprocessed absorption spectrum data into a Transform model deployed by edge equipment for feature extraction and regression prediction, and outputting key water quality parameters such as COD, ammonia nitrogen and total phosphorus; and based on a prediction result, real-time risk early warning and sampling frequency dynamic adjustment are realized by using an AI self-diagnosis algorithm, and data are synchronized to a cloud through Internet of Things communication for storage, analysis and model optimization. The system is composed of a data acquisition module, an edge processing module, a local decision-making module and a cloud synchronization module, integrates spectrum sensing, lightweight model reasoning, multistage early warning control and multi-protocol communication functions, finally realizes high-precision and low-delay intelligent water quality monitoring, and is suitable for long-term reliable operation of water bodies such as rivers and lakes.

Description

Technical Field

[0001] This invention relates to the field of water quality monitoring technology, specifically to a water quality full-spectrum absorption monitoring method and system based on the Transformer architecture. Background Technology

[0002] Water is the source of life, and water quality safety is directly related to human health, ecosystem balance, and economic development. Traditional water quality monitoring methods mainly rely on chemical analysis, such as titration and chromatography. While these methods are highly accurate, they suffer from drawbacks such as long processing times, complex operations, and the inability to monitor in real time. With the development of sensor technology, spectral absorption monitoring methods have been gradually applied to water quality monitoring. These methods infer water quality parameters by measuring the degree of absorption of light of specific wavelengths by water samples, offering advantages such as speed, non-destructive testing, and online monitoring capabilities. However, existing spectral monitoring methods still face many challenges: First, spectral data has high dimensionality and noise, and traditional machine learning models (such as PLS and SVM) have limited feature extraction capabilities, resulting in insufficient prediction accuracy; second, monitoring equipment is usually deployed in field environments, leading to high data transmission latency and making real-time response difficult; third, the lack of intelligent decision-making mechanisms prevents dynamic adjustments to monitoring strategies to adapt to changes in water quality.

[0003] In recent years, deep learning technology has made significant progress in fields such as image processing and speech recognition. The Transformer architecture, due to its powerful sequence modeling capabilities, has demonstrated outstanding performance in natural language processing. Some studies have attempted to apply Transformers to spectral data analysis, but these are mostly limited to laboratory environments and do not consider resource constraints and real-time requirements in real-world deployments. Meanwhile, the rise of edge computing and IoT technologies has provided new ideas for distributed monitoring, but existing systems often centralize computational tasks in the cloud, resulting in high response latency at the edge and failing to meet the immediate early warning needs of high-risk water bodies.

[0004] Therefore, there is an urgent need for a water quality monitoring method that integrates high-precision analysis, low-latency response, and intelligent decision-making. This invention addresses the above problems by proposing a water quality full-spectrum absorption monitoring method and system based on the Transformer architecture. It improves the accuracy of spectral data analysis through the Transformer model, achieves localized processing using edge computing, and combines cloud-based collaborative optimization models to ultimately achieve efficient and accurate water quality monitoring. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a water quality full-spectrum absorption monitoring method and system based on the Transformer architecture. This method achieves high-precision prediction and real-time early warning of water quality parameters through full-band spectral acquisition, Transformer feature extraction, edge intelligent decision-making, and cloud data synchronization.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a water quality full-spectrum absorption monitoring method based on the Transformer architecture, comprising the following steps: S1. Data acquisition steps: The water sample is monitored in real time by a full-band spectral sensor, and the absorption spectrum data in the range of 190-820nm is collected. The full-band spectral sensor is integrated into the micro monitoring station, which supports high-frequency sampling at least once per minute and automatically deducts light source fluctuations to eliminate system errors.

[0007] S2, Edge Processing Step: Input the absorption spectrum data collected in S1 into a neural network model based on the Transformer architecture for feature extraction and regression prediction. The Transformer model is deployed on an edge computing device (such as an AI all-in-one machine) and uses a self-attention mechanism to analyze the spectral sequence data and output water quality parameter concentration values, including at least one of COD, ammonia nitrogen, and total phosphorus.

[0008] S3, Local Decision-Making Steps: Based on the prediction results of S2, real-time risk warnings are issued through the AI ​​self-diagnosis algorithm built into the edge device. When water quality parameters are abnormal, a local alarm is triggered, and control commands are generated to adjust the sampling frequency of the monitoring station.

[0009] S4. Cloud Synchronization Steps: The water quality prediction results of S2 and the alarm information of S3 are transmitted to the cloud server through the IoT communication module (supporting 4G / 5G or LoRa) to realize remote data storage, multi-terminal sharing and historical trend analysis. At the same time, the cloud server can send model update instructions to edge devices to optimize the prediction accuracy of the Transformer model.

[0010] The system of the present invention includes: The data acquisition module, used to implement step S1, includes a spectral sensor, a preprocessing unit, and a calibration unit.

[0011] The edge processing module is used to implement the S2 step, including the Transformer model unit, the model compression unit, and the inference engine.

[0012] The local decision-making module is used to implement the S3 steps and includes an early warning algorithm unit, a control command unit, and a local storage unit.

[0013] The cloud synchronization module is used to implement the S4 steps, including a communication interface unit, a data management unit, and a cloud platform unit.

[0014] Compared with existing technologies, this invention provides a water quality full-spectrum absorption monitoring method and system based on the Transformer architecture, which has the following beneficial effects: 1. This water quality full-spectrum absorption monitoring method and system based on the Transformer architecture effectively captures long-range dependencies in the spectral sequence through the self-attention mechanism of the Transformer architecture, thereby improving the accuracy of water quality parameter prediction.

[0015] 2. This water quality full-spectrum absorption monitoring method and system based on the Transformer architecture deploys the model on edge devices, reducing data transmission latency and enabling real-time monitoring and rapid response.

[0016] 3. This water quality full-spectrum absorption monitoring method and system based on the Transformer architecture, combined with an AI self-diagnosis algorithm, dynamically distinguishes abnormal types and reduces the false alarm rate.

[0017] 4. The cloud-based collaborative mechanism supports continuous model optimization to adapt to the monitoring needs of different water bodies.

[0018] 5. The system features a modular design, making it easy to expand and maintain, and suitable for large-scale deployment. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the main process of the present invention; Figure 2 This is a flowchart of the spectral data preprocessing process of the present invention; Figure 3 This is a flowchart of the Transformer model inference process of the present invention; Figure 4 This is a diagram of the edge-cloud collaborative architecture of the present invention; Figure 5 This is a flowchart of the multi-level early warning judgment process of the present invention; Figure 6 This is a flowchart of the cloud-edge collaborative learning process of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] As described in the background section, there are shortcomings in the existing technology. In order to solve the above-mentioned technical problems, this application proposes a water quality full-spectrum absorption monitoring method and system based on the Transformer architecture. Example 1

[0022] Reference Figure 1-6This embodiment details the implementation process of a water quality full-spectrum absorption monitoring method based on the Transformer architecture. This method achieves intelligent water quality monitoring through four core steps, specifically including data acquisition, edge processing, local decision-making, and cloud synchronization.

[0023] S1. Data Acquisition Steps Acquiring raw spectral data is fundamental to the monitoring process. This step utilizes a full-band spectral sensor to achieve high-frequency acquisition of the water sample's absorption spectrum. The full-band spectral sensor employs ultraviolet-visible light technology, covering a wavelength range of 190-820 nm, which includes the characteristic absorption peaks of key water quality parameters. The sensor is integrated into a miniature monitoring station, which features a waterproof and corrosion-resistant design suitable for field deployment. The sensor supports high-frequency sampling at least once per minute. During sampling, a deuterium-tungsten lamp combination is used as the light source to ensure stability across the entire wavelength range. To eliminate system errors, the sensor incorporates a built-in reference optical path, monitors real-time fluctuations in light source intensity, and automatically subtracts the effects of these fluctuations using a differential algorithm. Specific implementation details include: Obtain raw spectral data: The sensor output includes a timestamp, sensor ID, and a spectral intensity array with a dimension of 631; Data preprocessing: First, baseline correction is performed, and polynomial fitting is used to remove background drift; then, Savitzky-Golay filter is applied for noise smoothing; finally, spectral normalization is performed to scale the intensity values ​​to the [0,1] range to eliminate the influence of dimensions. Data storage: The preprocessed data is cached in local storage units in the form of a time-series sequence. Each sample is a 631-dimensional vector and the sequence length is configurable.

[0024] S2, Edge Processing Steps The core of the sequence formatting edge processing step lies in using the Transformer model to extract features and perform regression prediction on the spectral sequence. The Transformer model is deployed on edge computing devices, which are characterized by low power consumption and high performance, making them suitable for long-term operation in the field. The model input is the preprocessed spectral sequence, and the output is the concentration values ​​of water quality parameters. The specific implementation includes: Sequence formatting: Convert the spectral sequence into the model input format, with a sequence length of L, and each time step corresponding to a 631-dimensional vector. If the sequence length is less than L, pad with zeros; if it exceeds L, use a sliding window to segment the sequence. Transformer architecture: The model includes an encoder layer, a multi-head self-attention mechanism, and a feedforward network. The self-attention mechanism is calculated as: Attention(Q,K,V)=softmax(QK^T / √d_k)V, where Q, K, and V are the query, key, and value matrices, and d_k is the dimension. Through attention weights, the model focuses on key wavelengths. Regression prediction: The encoder output is mapped to water quality parameters through a fully connected layer. The loss function is mean squared error, and the optimizer is Adam. During the training phase, supervised learning is performed using historical spectral data. Lightweight Model: To adapt to edge devices, the Transformer model is pruned and quantized, reducing the model size from 500MB to 50MB, and the inference latency is less than 100ms.

[0025] S3, Local Decision-Making Steps The local decision-making process for anomaly detection enables real-time risk warnings and resource adjustments based on prediction results. Edge devices incorporate AI self-diagnostic algorithms, combined with a rule engine and machine learning models, to distinguish anomaly types. Specific implementations include: Anomaly detection: Calculate the anomaly index of water quality parameters. For example, if the COD concentration exceeds the threshold three times consecutively, an early warning is triggered. The index formula is: E=α*(C-C_thresh) / σ, where C is the current value, C_thresh is the threshold, σ is the historical standard deviation, and α is the weighting factor. Risk grading: Risk levels are determined based on anomaly indices; for example, E>2 indicates high risk and triggers a red alert. A self-diagnostic algorithm analyzes abnormal patterns; for instance, a sudden increase may indicate a contamination event, while a slow drift may indicate instrument aging. Frequency adjustment: The sampling motor of the monitoring station is controlled via the GPIO interface. The frequency is increased to 5 times per minute during high-risk periods and reduced to 1 time per minute during low-risk periods. Simultaneously, alarm logs are recorded in the local storage unit and can be exported via USB.

[0026] S4, Cloud Synchronization Steps The data transmission and cloud synchronization process enables remote data management and model optimization. The IoT communication module employs low-power wide-area network technology with configurable transmission intervals. Specific implementation includes: Data transmission: Data is encrypted with AES and then packaged and uploaded. The package structure includes device ID, timestamp, water quality parameters, and alarm flags. The cloud server receives and parses the data.

[0027] Data Management: A time-series database is built in the cloud, supporting access from multiple terminals and providing trend charts and report generation.

[0028] Model update: The Transformer model is retrained in the cloud using new data, and the model parameters are sent to edge devices through differential update technology. The devices complete the hot update of the model during idle periods, improving prediction accuracy. Example 2

[0029] This embodiment describes a water quality full-spectrum absorption monitoring system based on the Transformer architecture, which is used to implement the method described in Embodiment 1. The system adopts a modular design, including a data acquisition module, an edge processing module, a local decision-making module, and a cloud synchronization module.

[0030] Data Acquisition Module: The data acquisition module is responsible for acquiring and preprocessing spectral data. The hardware includes a full-band spectral sensor, a microcontroller, and a power management unit. The sensor's optics utilize fiber optic coupling with a 10mm optical path length to ensure strong absorbed signal intensity. The software runs on an embedded Linux system, and the preprocessing algorithm is implemented in C++, ensuring high real-time performance. The calibration unit is automatically calibrated periodically using standard samples, with a calibration cycle of 7 days.

[0031] Edge Processing Module: The core of the edge processing module is Transformer model inference. The hardware utilizes an AI all-in-one machine equipped with 4GB of RAM and GPU acceleration. The software is based on the TensorFlow Lite framework, with the model remaining resident in memory after loading and supporting multi-threaded inference. The model compression unit is implemented using the TensorFlow Model Optimization Toolkit, achieving a pruning rate of 30% and an accuracy loss of less than 2%.

[0032] Local Decision Module: The local decision module is integrated into the edge device. The software is developed in Python, and the early warning algorithm combines isolated forest anomaly detection and a rule base. The control command unit connects to the monitoring station via RS485 protocol, supporting remote configuration. The local storage unit uses eMMC flash memory with a capacity of 32GB, capable of storing 30 days of data.

[0033] Cloud synchronization module: The communication unit of the cloud synchronization module adopts a 4G module and supports a fallback mechanism. The cloud platform unit is based on a microservice architecture, provides a RESTful API, and supports integration with third-party systems. The data management unit implements data backup and disaster recovery to ensure data security. Example 3

[0034] This embodiment provides an electronic device for implementing the aforementioned water quality monitoring system. The device includes a memory, a processor, and a communication interface. The memory stores a computer program, and the processor executes the program to implement the method steps. The device is specifically configured as follows: Processor: Quad-core ARM Cortex-A72, 1.5GHz. Memory: 2.8GB LPDDR4 RAM, 64GB eMMC storage; Communication interface: 3.4G / LoRa dual-mode, Wi-Fi 802.11ac; The equipment is deployed at the monitoring station and powered by a power adapter or solar cells. Example 4

[0035] This embodiment provides a computer-readable storage medium storing a computer program. When executed by a processor, the program implements the method described in Embodiment 1. The medium is an SD card or cloud storage, and the program is written in Python, including modules for data acquisition, model inference, and decision logic.

[0036] This invention enhances spectral data analysis capabilities through a Transformer architecture and, combined with edge computing and cloud collaboration, achieves intelligent water quality monitoring. Experiments show that in multiple water body tests, the COD prediction error is less than 5%, and the response time is less than 1 second, significantly outperforming traditional methods. The system can be extended to monitor other parameters and has broad application prospects.

[0037] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A water quality full-spectrum absorption monitoring method and system based on Transformer architecture, characterized in that, Includes the following steps: S1. Data acquisition steps: The water sample is monitored in real time by a full-band spectral sensor, and the absorption spectrum data in the range of 190-820nm is collected. The full-band spectral sensor is integrated into the micro monitoring station, supports high-frequency sampling at least once per minute, and automatically deducts light source fluctuations to eliminate system errors. S2, Edge Processing Step: Input the absorption spectrum data collected in S1 into a neural network model based on the Transformer architecture for feature extraction and regression prediction. The Transformer model is deployed on an edge computing device and uses a self-attention mechanism to analyze the spectral sequence data and output water quality parameter concentration values, including at least one of COD, ammonia nitrogen, and total phosphorus. S3, Local Decision-Making Steps: Based on the prediction results of S2, real-time risk warnings are issued through the AI ​​self-diagnosis algorithm built into the edge device. When water quality parameters are abnormal, a local alarm is triggered, and control commands are generated to adjust the sampling frequency of the monitoring station. S4. Cloud Synchronization Step: The water quality prediction results of S2 and the alarm information of S3 are transmitted to the cloud server through the IoT communication module to realize remote data storage, multi-terminal sharing and historical trend analysis. At the same time, the cloud server can send model update instructions to edge devices to optimize the prediction accuracy of the Transformer model.

2. The method as described in claim 1, characterized in that, The data acquisition steps in S1 include: Acquire raw spectral data collected by a full-band spectral sensor, the raw spectral data including timestamp, sensor identifier and spectral intensity value; The raw spectral data is preprocessed, including baseline correction, noise filtering, and spectral normalization, to eliminate environmental interference and instrument errors. The preprocessed spectral data is stored as a time sequence, with each time point corresponding to a spectral vector. The dimension of the spectral vector is the number of wavelength points in the range of 190-820nm. The calibration module automatically deducts light source fluctuations and dynamically adjusts the spectral values ​​based on real-time monitoring data of the reference light source to ensure data accuracy.

3. The method as described in claim 1, characterized in that, The edge processing steps in S2 include: The preprocessed spectral data is converted into a sequence format and input into the Transformer encoder layer, which includes a multi-head self-attention mechanism and a feedforward neural network. Key features are extracted by calculating the correlation weights between different wavelength points in a spectral sequence using a self-attention mechanism. The water quality parameter concentration values ​​are output using a regression layer, which includes a fully connected layer and an activation function, to achieve a mapping from the feature space to the concentration values. Deploy lightweight Transformer models on edge devices, which are compressed using knowledge distillation techniques to adapt to edge computing resource constraints.

4. The method as described in claim 1, characterized in that, The local decision-making steps in S3 include: An anomaly index is calculated based on the concentration values ​​of water quality parameters. If the anomaly index exceeds a preset threshold, a risk warning is triggered. The AI ​​self-diagnostic algorithm distinguishes the types of anomalies, including instrument malfunctions, sudden changes in water quality, or environmental disturbances, and generates corresponding alarm levels. The sampling frequency is dynamically adjusted according to the alarm level. When the risk is high, the sampling frequency is increased to multiple times per minute, and when the risk is low, the default frequency is restored. The alert logs, including timestamps, abnormal parameter values, and processing actions, are stored locally on the edge device for offline analysis.

5. The method as described in claim 1, characterized in that, The cloud synchronization steps in S4 include: Data from edge devices is encrypted and transmitted to the cloud server via 4G / 5G or LoRa communication protocols. The cloud server aggregates and cleans the received data, builds a historical database, and supports multi-dimensional queries and visualization. Machine learning algorithms are used to analyze historical data to generate water quality trend reports and predictive model update instructions; Update commands are sent to edge devices to enable online learning and adaptive optimization of the Transformer model.

6. A water quality full-spectrum absorption monitoring system based on the Transformer architecture, characterized in that, For implementing the method as described in any one of claims 1-5, comprising: The data acquisition module is used to acquire the absorption spectrum data of water samples through a full-band spectral sensor, and to complete the preprocessing and calibration. The edge processing module, deployed on edge computing devices, is used to run Transformer models to extract features and perform regression prediction on spectral data, and output water quality parameter concentration values. The local decision-making module, integrated into the edge device, is used for real-time risk warning and sampling frequency adjustment based on the prediction results; The cloud synchronization module is used to enable remote data transmission, storage, and model updates via IoT communication.

7. The system as described in claim 6, characterized in that, The data acquisition module includes: The spectral sensor unit supports full-band monitoring from 190 to 820 nm and has built-in photoelectric conversion and signal amplification circuits. The preprocessing unit is used for data filtering, normalization, and baseline correction, and is processed in real time by a digital signal processor. The calibration unit dynamically calibrates spectral data based on a reference light source and standard samples to ensure monitoring accuracy.

8. The system as described in claim 6, characterized in that, The edge processing module includes: Transformer model units are stacked encoders and support self-attention computation and feature fusion. The model compression unit reduces model size through pruning and quantization techniques to adapt to the memory limitations of edge devices; The inference engine, based on TensorFlowLite or PyTorchMobile, enables efficient prediction.

9. The system as described in claim 6, characterized in that, The local decision-making module includes: The early warning algorithm unit enables anomaly detection and risk classification, and supports multiple rule engines; The control command unit generates a sampling frequency adjustment signal, which is connected to the monitoring station actuator via a GPIO interface. Local storage units are used to cache data and logs and support power failure protection.

10. The system as described in claim 6, characterized in that, The cloud synchronization module includes: The communication interface unit supports multiple connection methods including 4G / 5G, LoRa, and Ethernet. The data management unit enables data encryption, compression, and batch uploading; The cloud platform unit provides API interfaces for multi-terminal access and model management.

Images