Abnormality monitoring method for water quality detection equipment

Abstract

The invention relates to the technical field of equipment monitoring, in particular to an anomaly monitoring method for water quality detection equipment, which realizes multi-level anomaly detection by deploying a plurality of monitoring nodes in a target water area, constructing a distributed monitoring topological structure and combining an auto-encoder, an isolated forest and an LOF algorithm. Based on historical data, a reference model of normal equipment is established, Apache Kafka is used for collecting data streams in real time, and standardization and windowing processing is carried out through Flink. A reconstruction error is calculated based on an auto-encoder, a feature matrix is constructed, global abnormal points are screened by using an isolated forest, local outlier is verified through an LOF algorithm, and an abnormal state is comprehensively judged. The method supports incremental learning, adapts to environmental changes, has the characteristics of high real-time performance and low delay, can effectively identify abnormal conditions such as sensor aging and measurement errors, improves data reliability and system stability, and remarkably reduces maintenance cost.

Description

Technical Field

[0001] The present invention relates to the technical field of equipment monitoring, and in particular to an abnormality monitoring method for water quality detection equipment. Background Art

[0002] The rapid development of industrialization has placed higher demands on water quality testing. Current water quality testing equipment may experience anomalies due to various reasons, such as sensor aging, measurement errors, and equipment failure. If these issues are not discovered and addressed promptly, they can not only distort data and affect decision-making accuracy, but can also mask potential water quality deterioration and delay necessary response measures.

[0003] The importance of abnormal monitoring of water quality testing equipment is reflected in the following aspects:

[0004] 1) Improve data reliability: By analyzing monitoring data in real time, outliers or trends can be quickly identified to ensure that the water quality information provided is accurate.

[0005] 2) Enhance system stability: Early detection of potential equipment failures and taking corresponding maintenance measures can effectively avoid system downtime caused by sudden equipment failure and ensure the continuity and stability of water quality monitoring work.

[0006] Abnormal monitoring methods aim to improve data quality by analyzing monitoring data in real time, promptly identifying and correcting potential errors or deviations. This not only helps improve the stability and accuracy of equipment operation, but also identifies potential failures at an early stage, reducing maintenance costs and downtime. Summary of the Invention

[0007] This invention provides an anomaly monitoring method for water quality testing equipment. By performing real-time monitoring and data analysis on the equipment, it ensures water quality safety and enables timely detection and response to equipment anomalies. This method not only improves the ability to identify global and local anomalies but also reduces false alarm rates by optimizing the processing flow, thereby enhancing the continuity and reliability of water quality monitoring.

[0008] In order to achieve the purpose of the present invention, the technical solution adopted is: a method for abnormality monitoring of water quality testing equipment, comprising:

[0009] S1. Set up monitoring nodes in the target waters, install water quality testing equipment at each monitoring node, and build a distributed monitoring topology;

[0010] S2: Based on historical water quality testing equipment data, an autoencoder was used to establish a baseline model of normal water quality testing equipment. Apache Kafka was used to receive data streams from each monitoring node in real time, and Apache Flink was used to preprocess the data.

[0011] S3. Calculate the reconstruction error of the preprocessed data based on the benchmark model of normal water quality testing equipment and construct a reconstruction error feature matrix;

[0012] S4, using the isolation forest algorithm to perform global outlier screening on the reconstruction error feature matrix to obtain suspected outliers;

[0013] S5. Perform local density analysis on the suspected abnormal points using the local outlier factor (LOF) algorithm to identify local abnormal points, integrate the global and local abnormality detection results, and output a final abnormality judgment result.

[0014] As an optimization solution of the present invention, monitoring node n: n = (node_id, devices); wherein: node_id is the unique identifier of the monitoring node; devices is a list containing monitoring device information, including device_type: device type; time: collection time; value: specific value collected, status: current status of the device.

[0015] As an optimization solution of the present invention, online monitoring of the deployed water quality testing equipment is performed through data consistency checking, including:

[0016] (1) Cross-device comparison: Deploy multiple water quality testing devices of the same type in the same water area and compare the data from the water quality testing devices to determine whether a device has an abnormality;

[0017] (2) Historical data comparison: Compare the currently collected data with the historical data of the same water quality testing equipment to observe whether there are sudden changes or trend deviations.

[0018] As an optimization solution of the present invention, the benchmark model of normal water quality detection equipment is established using an autoencoder as follows:

[0019] A. Data preprocessing, including:

[0020] Data standardization: Z-score standardization is used for each water quality testing equipment to convert data of different dimensions to the same scale; Where: x i,t : the original reading of water quality testing device i at time t; μ i : the mean value of historical data of water quality testing equipment i; σ i : Standard deviation of historical data of water quality testing equipment i; z i,t : Normalized value; z i,t is the specific element of the input matrix X;

[0021] B. Baseline model architecture of normal water quality testing equipment;

[0022] Encoder: compresses the input into a low-dimensional latent space through nonlinear mapping: h = σ(W e ·X+b e ); z=σ(W z h+b z ); where: W e is the weight matrix from the input layer to the first hidden layer, W z is the weight matrix from hidden layer to latent space, b e is the bias term of the first hidden layer, b z is the bias term of the latent space, σ is the ReLU activation function, h is the middle hidden layer feature, and z is the latent space representation;

[0023] Decoder: Reconstruct the original input from the latent space: h′=σ(W d z+b d ), X′=σ(W o ·h′+b o ), where h′ is the output of the middle hidden layer of the decoder, and X′ is the reconstructed output; the output layer uses Sigmoid activation to ensure a reasonable output range;

[0024] C. Loss function design; weighted mean square error is used as the reconstruction loss L:

[0025] Where: N is the number of samples, d is the sensor dimension, α j is the weight of water quality testing equipment j, x i,j and x' i,j represent the original input and reconstructed output respectively;

[0026] As an optimization solution of the present invention, when using an autoencoder to establish a baseline model of normal water quality testing equipment, incremental learning is used to regularly fine-tune the model using newly collected data to maintain the model's ability to learn the latest data and adapt to environmental changes.

[0027] As an optimization solution of the present invention, Apache Kafka is used to create corresponding Kafka topics for water quality testing equipment. Kafka producers are deployed on each monitoring node, and consumers subscribe to corresponding topics and transmit data to Apache Flink for preprocessing.

[0028] As an optimization solution of the present invention, in S3, the reconstruction error is calculated based on the benchmark model of the normal water quality detection equipment, and the reconstruction error characteristic matrix is constructed as follows:

[0029] For each sample i and sensor dimension j, calculate its original input x i,j and reconstruct the output x′ i,j The difference between them is the reconstruction error ei,j , the reconstruction error calculation formula is: e i,j =(x i,j -x′ i,j ) 2 ; Use weighted mean square error to quantify the overall reconstruction loss L: The reconstruction error e is calculated using i,j As a new feature vector, the multidimensional reconstruction error output by the autoencoder is constructed into a new feature matrix:

[0030]

[0031] Where: d is the number of water quality testing equipment.

[0032] As an optimization solution of the present invention, 1) Isolation tree construction: for each batch of data in the feature matrix E, T isolation trees are constructed, where T is the number of isolation trees:

[0033] Randomly select features: randomly select a dimension q from d reconstruction error features;

[0034] Random partition value: randomly select a partition value p within the range of q dimension;

[0035] Recursive partitioning: Repeat the above process until the termination condition is met. The termination condition is that the tree depth reaches the upper limit log2N or the child node contains only a single sample;

[0036] 2) Path length calculation: For sample E i , calculate the path length h(E i ), represents the number of edges from the root node to the leaf node; h(E i ) = Sample E i The number of edges from the root node to the leaf nodes in the isolation tree;

[0037] 3) Anomaly score calculation: The path lengths of all isolated trees are combined to calculate the anomaly score s(E i ,N):

[0038]

[0039] Where: E[h(E i )] is sample E i The average path length in all isolated trees, C(N) is the path length correction term, which is used to standardize the expected path length under different sample sizes N.

[0040] Where: H(k) is the harmonic number, H(k)=1+

[0041] 1 / 2+…+1 / k; k is an intermediate variable.

[0042] 4) Threshold determination: Set the threshold τ, if s(E i )>τ, it is determined as a global outlier.

[0043] As an optimization solution of the present invention, the local outlier of the suspected outlier is verified by LOF. The specific process is as follows:

[0044] (1) Define parameters; k is the number of neighbors, which represents the number of nearest neighbor samples considered when calculating the local density;

[0045] (2) Calculate k-distance and k-neighborhood;

[0046] k-distance is k-distance (E i ) is sample E i The distance to its kth nearest neighbor is:

[0047] k-distance(E i )=distance(E i ,E (k) )

[0048] Where: E (k) It is E i The kth nearest neighbor sample of is usually measured by Euclidean distance:

[0049] k-neighborhood N k (E i ): Contains E i The set of k nearest neighbor samples;

[0050] (3) Reachable distance: Sample E i To neighbor E j The reachable distance is:

[0051] reach-distk(E i ,E j )=max(k-distance(E j ),distance(E i ,E j ));

[0052] (4) The local reachable density is sample E i The local density of

[0053]

[0054] Where: The denominator is E i The inverse of the average reachability distance of its k neighbors;

[0055] (5) The local outlier factor is the comparison sample E i The ratio of the local density of to the local density of its neighbors is given by:

[0056] Threshold determination: Decision rule: If LOF k (E i )>θ, θ is the threshold, then confirm E i A local outlier point.

[0057] The present invention has positive effects: 1) Through the distributed monitoring topology and the Apache Kafka+Flink stream processing framework, the present invention realizes the real-time collection and processing of water quality detection data, significantly improving the timeliness of anomaly detection, reducing processing latency, and meeting the real-time requirements of minute-level windows;

[0058] 2) This invention uses real-time monitoring data analysis and combines autoencoders, isolation forests, and LOF algorithms to perform multi-level anomaly detection on water quality testing equipment data. This can quickly identify outliers or trends, ensuring the accuracy of water quality information. This can effectively reduce data distortion caused by equipment failures.

[0059] 3) This method uses historical data analysis to establish a baseline model of normal water quality monitoring equipment. By regularly updating model parameters to adapt to environmental changes, it can detect potential equipment failures early and take preventive measures, thus avoiding system downtime caused by sudden equipment failure. This method ensures the continuity and stability of water quality monitoring operations, reducing maintenance costs and downtime.

[0060] 4) Based on a distributed monitoring topology and efficient processing mechanisms (Apache Kafka and Flink), this invention enables the real-time collection and processing of large amounts of water quality data from multiple monitoring nodes. Furthermore, through an incremental learning mechanism, the model continuously learns the latest water quality conditions, eliminating the need for frequent retraining of the entire model. This significantly saves computing resources and time. Even with large-scale data streams, it maintains high processing speeds and responds promptly to water quality changes, providing strong technical support for emergency management and decision support.

[0061] 5) This paper combines two different anomaly detection algorithms, Isolation Forest and Local Outlier Factor (LOF), using a global screening followed by local validation approach. This approach can both quickly identify the majority of obvious outliers and meticulously capture subtle anomalies hidden within a single dimension. Experimental data demonstrates that this combined approach significantly improves the overall efficiency and accuracy of anomaly detection compared to either algorithm alone. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0064] The terms "first," "second," "third," "fourth," and the like (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0065] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 In an embodiment of the present invention, a method for monitoring abnormalities of water quality detection equipment includes:

[0066] 1) Set up monitoring nodes in the target waters, install water quality testing equipment at each monitoring node, and build a distributed monitoring topology;

[0067] Water quality testing equipment includes pH meters, dissolved oxygen meters, turbidity meters, ammonia nitrogen analyzers, and heavy metal analyzers. Monitoring node n: n = (node_id, devices); where node_id is the unique identifier of the monitoring node; devices is a list containing monitoring device information, including device_type (device type); time (collection time); value (collected value); and status (current device status).

[0068] The data structure can be converted into JSON format for easy storage and transmission.

[0069]

[0070] The distributed monitoring topology includes the rational setting of multiple monitoring nodes in the target water area. Each monitoring node is equipped with a set of specific water quality testing equipment, including but not limited to pH meters, dissolved oxygen meters, turbidity meters, ammonia nitrogen analyzers, and heavy metal analyzers. This ensures that the water quality can be evaluated from multiple aspects. Each device collects data at a set time interval and records the collected data (including timestamps, measurement values, and device status). The data is formatted in JSON format for subsequent storage and transmission. The communication mechanism between monitoring nodes and with the central processing system is an important part of the distributed monitoring topology. Wireless communication technologies (such as Wi-Fi, LoRaWAN, or 5G, etc.) are used to ensure that data can be transmitted from each monitoring node to the data center in real time and stably.

[0071] 2) Based on historical water quality testing equipment data, an autoencoder was used to establish a baseline model of normal water quality testing equipment. Apache Kafka was used to receive the data stream from each monitoring node in real time, and Flink was used for preprocessing. Using an autoencoder to establish a baseline model of normal water quality testing equipment is an effective approach, especially when the data has high dimensionality and nonlinear characteristics.

[0072] Online monitoring of deployed water quality testing equipment is performed through data consistency checks, including:

[0073] (3) Cross-device comparison: Deploy multiple water quality testing devices of the same type in the same water area and compare the data from different water quality testing devices to determine whether a device has an abnormality. If the readings of a device are significantly different from those of other devices, there may be a problem with the device.

[0074] (4) Historical data comparison: Compare the currently collected data with the historical data of the same water quality testing equipment to observe whether there is a sudden change or trend deviation. Based on the historical water quality testing equipment data, the autoencoder is used to establish a baseline model of normal water quality testing equipment. Specifically:

[0075] B. Data preprocessing, including:

[0076] Data standardization: Z-score standardization is used for each water quality testing equipment to convert data of different dimensions to the same scale; Where: x i,t : the original reading of water quality testing device i at time t; μ i : the mean value of historical data of water quality testing equipment i; σ i : Standard deviation of historical data of water quality testing equipment i; z i,t : Normalized value. i,t is the specific element of the input matrix X.

[0077] B. The architecture of the baseline model (autoencoder baseline model) of normal water quality testing equipment;

[0078] Encoder: compresses the input into a low-dimensional latent space through nonlinear mapping: h = σ(W e ·X+b e ); z=σ(W z h+b z ); where: W e is the weight matrix from the input layer to the first hidden layer, W z is the weight matrix from hidden layer to latent space, b e is the bias term of the first hidden layer, b z is the bias term of the latent space, σ is the ReLU activation function, h is the middle hidden layer feature, and z is the latent space representation.

[0079] Decoder: Reconstruct the original input from the latent space: h′=σ(W d z+b d ), X′=σ(W o ·h′+b o ), where h′ is the output of the decoder’s intermediate hidden layer and X′ is the reconstructed output. The output layer uses Sigmoid activation to ensure a reasonable output range.

[0080] C. Loss function design: weighted mean square error (WMSE) is used as the reconstruction loss L: Where: N is the number of samples, d is the sensor dimension, α j is the weight of water quality testing equipment j, x i,j and x' i,j Representing the original input and reconstructed output, respectively. During training, the autoencoder model architecture uses a weighted mean squared error (L) as the loss function. The model attempts to minimize the gap between the input data and the reconstructed data after encoding and decoding.

[0081] D. Through incremental learning, the model is regularly fine-tuned using newly collected data to maintain the model's ability to learn the latest data and adapt to environmental changes.

[0082] The specific steps for using Apache Kafka to receive data streams from each monitoring node in real time and preprocessing them through Flink are as follows:

[0083] (1) Install water quality testing equipment on each monitoring node n and convert the data from these devices into JSON format for storage and transmission. This data needs to be sent to a central server or cloud platform in real time for further processing. Apache Kafka is used as a message queue system to receive data streams from each monitoring node.

[0084] (2) Use Apache Kafka for data stream management; the steps include:

[0085] Create Kafka topics: Create corresponding Kafka topics for water quality testing equipment (pH meter, dissolved oxygen meter, turbidity meter, etc.).

[0086] Producer configuration: Configure a Kafka producer on each monitoring node to send the data from the water quality testing equipment in JSON format to the corresponding Kafka topic; ensure that each monitoring node can efficiently and reliably send the data from the water quality testing equipment to the Kafka cluster.

[0087] Consumer Configuration: Set up a Kafka consumer to subscribe to the above topic to obtain real-time data streams from water quality testing equipment and pass them to Apache Flink for preprocessing. This enables real-time monitoring and consumption of water quality testing equipment data in the Kafka topic, ensuring that the data can be promptly processed in the next stage (i.e., Flink preprocessing).

[0088] Flink preprocessing specifically includes:

[0089] Initialize the Flink environment: Configure a Flink cluster to process large-scale data streams.

[0090] Read Kafka data streams: Use the Kafka connector provided by Flink to read data from Kafka topics.

[0091] Parsing JSON data: Convert the received JSON string into an object structure that is easy to process.

[0092] Data standardization: The Z-score standardization method is applied so that data of different dimensions can be compared on the same scale.

[0093] Data windowing: Divide a continuous data stream into fixed-size time windows to facilitate subsequent analysis.

[0094] Apache Kafka is used to efficiently collect data from water quality testing equipment at various monitoring nodes and transmit it to a central system or cloud platform in real time. This approach ensures rapid data analysis and processing. Integrating an autoencoder model with online monitoring of water quality testing equipment data can quickly identify potential anomalies. Autoencoders, a deep learning model, can extract key features from high-dimensional and nonlinear water quality data and establish a baseline model. This is particularly effective in identifying subtle changes.

[0095] 1) For the pre-processed data, based on the benchmark model of normal water quality testing equipment, the isolation forest is used to quickly screen global outliers; for suspected outliers, the local outliers are verified by LOF. The specific process is as follows:

[0096] The reconstruction error of real-time data is calculated using the baseline model of normal water quality testing equipment, which is used as input features for the subsequent isolation forest + LOF. The specific steps for combining the autoencoder baseline model with the two-step anomaly detection (isolation forest + LOF) on the preprocessed data are as follows:

[0097] The execution process based on the autoencoder baseline model is as follows;

[0098] Input: Normalized data stream preprocessed by Flink (batch data divided by time windows).

[0099] 1) Reconstruction error calculation: The preprocessed data is fed into a trained autoencoder baseline model. The encoder part of the model compresses the input data into a latent space representation z. The decoder part then attempts to reconstruct the original input X′ from this low-dimensional representation z. The autoencoder baseline model is used to calculate the reconstruction error on the real-time data and uses it as input features for subsequent anomaly detection algorithms (such as Isolation Forest and LOF).

[0100] For each sample i and sensor dimension j, calculate its original input x i,j and reconstruct the output x′ i,j The difference between them is the reconstruction error e i,j , the reconstruction error calculation formula is: e i,j =(x i,j -x′ i,j ) 2 ; Use weighted mean square error (WMSE) to quantify the overall reconstruction loss L: The reconstruction error e is calculated using i,j As a new feature vector, it is used in subsequent anomaly detection algorithms (such as Isolation Forest and LOF).

[0101] The specific steps of reconstructing the error feature vector are: constructing the multi-dimensional reconstruction error output by the autoencoder into a new feature matrix:

[0102]

[0103] Among them: e i,j =(x i,j -x′ i,j ) 2 represents the reconstruction error of the j-th water quality testing device for the i-th sample, and d is the number of water quality testing devices.

[0104] For the preprocessed data, based on the benchmark model of normal water quality testing equipment, isolation forest is used to quickly screen global outliers. Isolation forest quickly isolates outliers by randomly dividing the feature space. Because outliers deviate from the normal distribution, fewer divisions (path lengths) are required; because normal points are densely distributed, the path length is longer.

[0105] The specific implementation process is:

[0106] 1) Isolation tree construction: For each batch of data in the feature matrix E (from the Flink time window), construct T isolation trees (iTree), where T is the number of isolation trees:

[0107] Randomly select features: randomly select a dimension q (randomly selected feature dimension) from the d reconstruction error features;

[0108] Random partition value: randomly select a partition value p within the range of q dimension;

[0109] Recursive partitioning: Repeat the above process until the termination condition is met (the tree depth reaches the upper limit log2N or the child node contains only a single sample);

[0110] 2) Path length calculation: For sample E i , calculate the path length h(E i ), represents the number of edges from the root node to the leaf node. h(E i ) = Sample E i The number of edges from the root node to the leaves in the isolation tree.

[0111] 3) Anomaly score calculation: The path lengths of all isolated trees are combined to calculate the anomaly score s(E i ,N):

[0112]

[0113] Where: E[h(E i )] is sample E i The average path length in all isolated trees, C(N) is the path length correction term, which is used to standardize the expected path length under different sample sizes N.

[0114] Where: H(k) is the harmonic number, H(k)=1+

[0115] 1 / 2+…+1 / k;

[0116] 4) Threshold determination: Set the threshold τ (such as 0.7), if s(E i )>τ, it is determined as a global outlier.

[0117] For suspected outliers, LOF is used to verify local outliers. The specific process is as follows:

[0118] Input data: A subset of the reconstruction error feature matrix E corresponding to the global outliers output by the isolation forest (i.e., samples with s(Ei)>τ).

[0119] Feature space: Each sample E i It is composed of d-dimensional reconstruction error, namely E i =[e i,1 ,e i,2 ,…,e i,d ], where e i,j =(x i,j -x′ i,j ) 2 .

[0120] (1) Define parameters; k is the number of neighbors, which represents the number of nearest neighbor samples considered when calculating the local density.

[0121] (2) Calculate k-distance and k-neighborhood;

[0122] k-distance is k-distance (E i ) is sample E i The distance to its kth nearest neighbor is:

[0123] k-distance(E i )=distance(E i ,E (k) )

[0124] Where: E (k) It is E i The kth nearest neighbor sample of is usually measured by Euclidean distance:

[0125] k-neighborhood N k (E i ): Contains E i The set of k nearest neighbor samples.

[0126] (3) Reachable distance: Sample E i To neighbor E j The reachable distance is used to avoid density estimation bias caused by too dense neighboring points. The formula is:

[0127] reach-distk(E i ,E j )=max(k-distance(E j ),distance(E i ,E j ));

[0128] (4) Local Reachability Density (LRD) is the sample E i The local density of a node reflects the compactness of its neighbors.

[0129]

[0130] Where: The denominator is E i The reciprocal of the average reachable distance of its k neighbors. The larger the LRD is, the closer the E i The denser the distribution of neighbors (normal point), the smaller the LRD is, indicating that E i The more sparsely distributed the surrounding neighbors are (potential outliers).

[0131] (5) Local outlier factor (LOF Score) is the comparison sample E i The ratio of the local density of to the local density of its neighbors. The formula is: LOF≈1:E i The local density of is close to the neighbor density and is judged as a normal point. LOF>1: E i The local density of the point is lower than the neighbor density, and it is judged as a local outlier (the larger the value, the higher the degree of anomaly). Threshold judgment: Judgment rule: If LOF k (E i )>θ (θ is the threshold, such as θ=1.5), then confirm E i A local outlier point.

[0132] Isolation forests are used to perform a preliminary global screening of outliers on the reconstruction error feature matrix E calculated by the autoencoder. Because the isolation forest algorithm has near-linear time complexity, it is particularly well-suited for anomaly detection in high-dimensional datasets, identifying most obvious outliers in a short period of time. The LOF algorithm is further applied to suspected outliers to analyze local density differences. This helps identify local sparse regions that occur within a single sensor dimension, known as hidden local anomalies. For example, when a heavy metal analyzer malfunctions, the corresponding sensor dimension may exhibit local density characteristics that differ significantly from those of healthy devices. LOF can accurately capture these subtle changes. By first using the isolation forest to filter out most healthy samples and then applying the LOF algorithm to only the small number of suspected outliers, the resource consumption required for direct full LOF computation can be significantly reduced. Given the demands of real-time streaming processing, this approach is particularly well-suited for data volumes within Flink's time window, which are O(NlogN), maintaining high processing speed even with large data volumes. It not only effectively identifies anomalies in a single sensor (such as sudden changes or small-scale drift), but also addresses complex anomalies arising from multiple sensors. By combining the nonlinear feature representation capabilities of autoencoders with the strengths of two anomaly detection algorithms, the system can more comprehensively cover a wide range of possible anomaly patterns, thereby improving the overall detection rate. Regular updates to model parameters ensure that they always reflect the latest water quality conditions and can quickly adapt to environmental changes. Furthermore, flexible adjustments to parameters such as thresholds based on actual needs ensure high efficiency while maintaining high accuracy.

[0133] Status: The current status of the device is determined by combining the following methods:

[0134] 1) Based on the result feedback of the anomaly detection algorithm; when the measurement value of a device deviates significantly from historical data or devices of the same type (manifested as high reconstruction error, high isolation forest score, high LOF value), it may be marked as an anomaly.

[0135] Once the device is confirmed to have abnormal behavior, its status is set to "abnormal" or a similar identifier; otherwise, it remains "normal".

[0136] 2) Data consistency check mechanism;

[0137] The system performs data consistency comparison in the following two ways to assist in determining the device status:

[0138] (1) Cross-device comparison:

[0139] If the reading of a device differs significantly from that of other similar devices (exceeding a set threshold), the device is considered to be faulty; in such cases, the status of the device will be marked as abnormal.

[0140] (2) Historical data comparison:

[0141] Compare the data in the current time window with the historical data of the device for trend comparison;

[0142] If there are inconsistencies such as mutations, drifts, and periodic deviations, it may also be a sign of equipment abnormality.

[0143] 3) Equipment hardware status monitoring;

[0144] Device operation log: records whether the device has restarted, communication interrupted, sensor timed out, and other problems; some high-end devices have self-test functions and can output status codes such as "sensor aging" and "probe contamination".

[0145] 4) Incremental learning feedback of the autoencoder model;

[0146] If a device consistently causes high model reconstruction errors, and adjustments to the model do not improve the results, this may indicate a problem with the device itself. The presence of any one or more of the above conditions constitutes an anomaly.

[0147] The following experimental data demonstrates that the combined approach (Isolation Forest + LOF) significantly improves performance compared to the individual algorithms. As shown in Table 1, the combined approach (Isolation Forest + LOF) achieves significant improvements (F1 improvement of 11.7%) compared to the individual algorithms, due to the complementary nature of global screening and local validation. The processing latency meets real-time requirements (200ms per 1,000 records) and is well-suited to Flink's time window (1-minute window).

[0148] Table 1 Comparison of combined methods and single algorithms

[0149]

[0150] An embodiment of the present invention further provides a computer device, which may be a server. The computer device includes a processor, memory, display screen, input device, network interface, and database connected via a system bus. The processor of the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above-mentioned method is implemented.

[0151] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., various media that can store program code.

[0153] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring abnormalities in water quality testing equipment, characterized in that: include: S1. Set up monitoring nodes in the target waters, install water quality testing equipment at each monitoring node, and build a distributed monitoring topology; S2: Based on historical water quality testing equipment data, an autoencoder was used to establish a baseline model of normal water quality testing equipment. Apache Kafka was used to receive data streams from each monitoring node in real time, and Apache Flink was used to preprocess the data. S3. Calculate the reconstruction error of the preprocessed data based on the benchmark model of normal water quality testing equipment and construct a reconstruction error feature matrix; S4, using the isolation forest algorithm to perform global outlier screening on the reconstruction error feature matrix to obtain suspected outliers; S5. Perform local density analysis on the suspected abnormal points using the local outlier factor (LOF) algorithm to identify local abnormal points, integrate the global and local abnormality detection results, and output a final abnormality judgment result.

2. The abnormality monitoring method for water quality testing equipment according to claim 1, characterized in that: Monitoring node n: n = (node_id, devices); where: node_id is the unique identifier of the monitoring node; devices is a list containing monitoring device information, including device_type: device type; time: collection time; value: the specific value collected, status: the current status of the device.

3. The abnormality monitoring method for water quality testing equipment according to claim 2, characterized in that: Online monitoring of deployed water quality testing equipment is performed through data consistency checks, including: (1) Cross-device comparison: Deploy multiple water quality testing devices of the same type in the same water area and compare the data from the water quality testing devices to determine whether a device has an abnormality; (2) Historical data comparison: Compare the currently collected data with the historical data of the same water quality testing equipment to observe whether there are sudden changes or trend deviations.

4. The abnormality monitoring method for water quality testing equipment according to claim 3, characterized in that: The specific benchmark model for normal water quality testing equipment using the autoencoder is: A. Data preprocessing, including: Data standardization: Z-score standardization is used for each water quality testing equipment to convert data of different dimensions to the same scale; Where: x i,t : the original reading of water quality testing device i at time t; μ i : the mean value of historical data of water quality testing equipment i; σ i : Standard deviation of historical data of water quality testing equipment i; z i,t : Normalized value; z i,t is the specific element of the input matrix X; B. Baseline model architecture of normal water quality testing equipment; Encoder: compresses the input into a low-dimensional latent space through nonlinear mapping: h = σ(W e ·X+b e ); z=σ(W z h+b z ); where: W e is the weight matrix from the input layer to the first hidden layer, W z is the weight matrix from hidden layer to latent space, b e is the bias term of the first hidden layer, b z is the bias term of the latent space, σ is the ReLU activation function, h is the middle hidden layer feature, and z is the latent space representation; Decoder: Reconstruct the original input from the latent space: h′=σ(W d z+b d ), X′=σ(W o ·h′+b o ), where h′ is the output of the middle hidden layer of the decoder, and X′ is the reconstructed output; the output layer uses Sigmoid activation to ensure a reasonable output range; C. Loss function design; weighted mean square error is used as the reconstruction loss L: Where: N is the number of samples, d is the sensor dimension, α j is the weight of water quality testing equipment j, x i,j and x' i,j represent the original input and reconstructed output respectively.

5. The abnormality monitoring method for water quality testing equipment according to claim 4, characterized in that: When using the autoencoder to build a baseline model for normal water quality testing equipment, incremental learning is used to regularly fine-tune the model using newly collected data to maintain the model's ability to learn the latest data and adapt to environmental changes.

6. The abnormality monitoring method for water quality testing equipment according to claim 5, characterized in that: The Apache Kafka is used to create corresponding Kafka topics for the water quality testing equipment. The Kafka producer is deployed on each monitoring node. The consumer subscribes to the corresponding topic and transmits the data to Apache Flink for preprocessing.

7. The abnormality monitoring method for water quality testing equipment according to claim 6, characterized in that: In S3, the reconstruction error is calculated based on the benchmark model of normal water quality testing equipment, and the reconstruction error feature matrix is constructed as follows: For each sample i and sensor dimension j, calculate its original input x i,j and reconstruct the output x′ i,j The difference between them is the reconstruction error e i,j , the reconstruction error calculation formula is: e i,j =(x i,j -x′ i,j ) 2 ; Use weighted mean square error to quantify the overall reconstruction loss L: The reconstruction error e is calculated using i,j As a new feature vector, the multidimensional reconstruction error output by the autoencoder is constructed into a new feature matrix: Where: d is the number of water quality testing equipment.

8. The abnormality monitoring method for water quality testing equipment according to claim 7, characterized in that: In S4, the isolation forest algorithm is used to perform global outlier screening on the reconstruction error feature matrix, and the suspected outliers are obtained as follows: 1) Isolation tree construction: For each batch of data in the feature matrix E, construct T isolation trees, where T is the number of isolation trees; Randomly select features: randomly select a dimension q from d reconstruction error features; Random partition value: randomly select a partition value p within the range of q dimension; Recursive partitioning: Repeat the above process until the termination condition is met. The termination condition is that the tree depth reaches the upper limit log2N or the child node contains only a single sample; 2) Path length calculation: For sample E i , calculate the path length h(E i ), represents the number of edges from the root node to the leaf node; h(E i ) = Sample E i The number of edges from the root node to the leaf nodes in the isolation tree; 3) Anomaly score calculation: The path lengths of all isolated trees are combined to calculate the anomaly score s(E i ,N): Where: E[h(E i )] is sample E i The average path length in all isolated trees, C(N) is the path length correction term, which is used to standardize the expected path length under different sample sizes N. Where: H(k) is the harmonic number, H(k) = 1 + 1 / 2 + ... + 1 / k; 4) Threshold determination: Set the threshold τ, if s(E i )>τ, it is determined as a global outlier.

9. The abnormality monitoring method for water quality testing equipment according to claim 8, characterized in that: For suspected outliers, LOF is used to verify local outliers. The specific process is as follows: (1) Define parameters; k is the number of neighbors, which represents the number of nearest neighbor samples considered when calculating the local density; (2) Calculate k-distance and k-neighborhood; k-distance is k-distance (E i ) is sample E i The distance to its kth nearest neighbor is: k-distance(E i )=distance(E i ,AND (k) ) Where: E (k) It is E i The kth nearest neighbor sample of is usually measured by Euclidean distance: k-neighborhood N k (E i ): Contains E i The set of k nearest neighbor samples; (3) Reachable distance: Sample E i To neighbor E j The reachable distance is: reach-distk(E i ,E j )=max(k-distance(E j ),distance(E i ,E j )); (4) The local reachable density is sample E i The local density of Where: The denominator is E i The inverse of the average reachability distance of its k neighbors; (5) The local outlier factor is the comparison sample E i The ratio of the local density of to the local density of its neighbors is given by: Threshold determination: Decision rule: If LOF k (E i )>θ, θ is the threshold, then confirm E i A local outlier point.

Images