Water supply network data anomaly detection method, system, medium and device

Abstract

A data anomaly detection method and system for a water supply network, a medium and equipment, relate to the technical field of data detection. The method comprises: acquiring target monitoring data of a water supply pipe; performing feature extraction on the target monitoring data to obtain feature data; performing anomaly detection on the water supply pipe based on the feature data to obtain an anomaly score; acquiring pipe parameters of the water supply pipe and environmental parameters in which the water supply pipe is located, adjusting the anomaly score based on the pipe parameters and the environmental parameters to obtain a target score; and generating a detection result of the water supply pipe based on the target score. The technical solution provided in the application improves the accuracy of anomaly data detection of the pipe.

Description

Technical Field

[0001] This application relates to the field of data detection technology, specifically to a method, system, medium, and equipment for detecting data anomalies in a water supply network. Background Technology

[0002] As a crucial component of urban infrastructure, water supply networks bear the vital responsibility of safely and efficiently delivering water resources to users. Their operational status directly impacts the safety and efficiency of the water supply. With the support of Internet of Things (IoT) technology, the deployment of numerous sensors enables real-time monitoring of key indicators such as flow rate and pressure. This provides technical support for the timely detection of anomalies such as leaks and pipe bursts, which is essential for ensuring water supply safety and improving the operational efficiency of the network.

[0003] Anomaly detection in water supply systems primarily relies on various data collected by sensors, such as changes in water flow, pressure, and water quality. This involves acquiring these data in real time and comparing them with thresholds to determine if any anomalies exist. However, this anomaly detection method depends on relatively fixed and simple threshold comparisons, resulting in low accuracy in identifying anomalies in pipeline data. Summary of the Invention

[0004] This application provides a method, system, medium, and equipment for detecting data anomalies in a water supply network, which can improve the accuracy of detecting abnormal data in pipelines.

[0005] Firstly, this application provides a method for detecting data anomalies in a water supply network, the method comprising: Obtain target monitoring data for the water supply pipes; Feature data is obtained by extracting features from the target monitoring data; Anomaly detection is performed on the water supply pipe based on the feature data to obtain an anomaly score; Obtain the pipe parameters of the water supply pipe and the environmental parameters of the water supply pipe, and adjust the anomaly score based on the pipe parameters and the environmental parameters to obtain the target score; The detection results of the water supply pipe are generated based on the target score.

[0006] By adopting the above technical solution, unlike the existing technology that relies solely on sensor data for simple threshold judgment in anomaly detection, this application extracts features from monitoring data to obtain feature data, and then uses a machine learning model based on the feature data to achieve a more accurate initial anomaly score. This can effectively detect potential anomalies in the water supply network. In particular, by acquiring pipeline parameters and environmental parameters, and adjusting the anomaly score based on these parameters, the pipeline's own state and environmental influences can be comprehensively considered to correct the detection results, avoid false alarms caused by environmental changes, and further improve the accuracy of anomaly data detection in pipelines.

[0007] Optionally, acquiring the target monitoring data of the water supply pipe includes: Real-time acquisition of raw monitoring data from water supply pipes; The original monitoring data is subjected to noise reduction and missing value imputation processes, and the processed data is then normalized to obtain the target monitoring data.

[0008] By adopting the above technical solution, unlike the existing technology that directly relies on the original monitoring data, this application obtains the target monitoring data by preprocessing the original monitoring data. The preprocessing includes noise removal, missing value filling, normalization and other processing of the original monitoring data, which can filter out noise in the original data and make subsequent feature extraction and anomaly detection more accurate and reliable.

[0009] Optionally, the step of extracting features from the target monitoring data to obtain feature data includes: The target monitoring data is input into a preset autoencoder to obtain low-dimensional features; The low-dimensional features are decoded to obtain the decoded features; Feature data is obtained by extracting features from the decoded features using cross-validation.

[0010] By adopting the above technical solution, the target monitoring data is reduced in dimensionality by constructing an autoencoder, then decoded to obtain decoded features, and cross-validation is used to extract features from the decoded features to obtain higher quality feature data. This process makes full use of the dimensionality reduction and reconstruction function of the autoencoder, which can extract potential features from the monitoring data, which is beneficial for subsequent anomaly detection and improves the accuracy of detection.

[0011] Optionally, before performing anomaly detection on the water supply pipe based on the feature data and obtaining an anomaly score, the method further includes: Obtain historical sample data of the water supply pipe, wherein the historical sample data is feature data after data preprocessing and feature extraction; Multiple isolation trees are constructed based on the historical sample data, and an isolation forest model is formed based on each of the isolation trees; Based on the aforementioned feature data, anomaly detection is performed on the water supply pipe to obtain an anomaly score, including: The feature data is input into the isolated forest model, the path length of the feature data under each isolated tree in the isolated forest model is calculated, and the average of the path lengths is calculated to obtain the anomaly score.

[0012] By adopting the above technical solution, a more reliable anomaly detection is achieved by constructing an isolated forest model. This involves acquiring historical sample data, processing the data, and extracting features to construct multiple isolated trees. These isolated trees are then combined to form an isolated forest. The feature data is then input into the forest model, and the path length of each tree is calculated and averaged to obtain an anomaly score. This fully utilizes the isolated forest model trained with big data, enabling a more comprehensive and intelligent judgment of data anomalies. It avoids false alarms caused by simple rules and improves the accuracy of anomaly detection.

[0013] Optionally, adjusting the anomaly score based on the pipeline parameters and the environmental parameters to obtain the target score includes: The pipeline parameters and environmental parameters are respectively input into a preset encoder to obtain the pipeline parameter code and the environmental parameter code; The anomaly score, the pipeline parameter code, and the environmental parameter code are concatenated into a fusion vector; The fusion vector is input into the attention mechanism model to obtain a weighted sum of the fusion vectors as the target vector. The weight coefficients of the attention mechanism model are dynamically allocated based on the correlation between the anomaly score, the pipeline parameter encoding, and the environment parameter encoding. The target vector is input into the decoder to obtain the target score.

[0014] By adopting the above technical solution, an attention mechanism model is constructed, and pipeline parameters and environmental parameters are introduced to adjust the anomaly score. Specifically, the pipeline parameters and environmental parameters are encoded separately, concatenated with the anomaly score to form a fusion vector, and then input into the attention model for weighting. The weight coefficients are dynamically allocated based on relevance. This allows the anomaly score to be corrected according to the pipeline's own state and environmental conditions, avoiding false alarms caused by environmental changes and improving the accuracy of anomaly detection.

[0015] Optionally, generating the detection result of the water supply pipe based on the target score includes: If the target score is less than or equal to the low anomaly threshold, the detection result of the water supply pipe is determined to be in a normal state. If the target score is greater than the low anomaly threshold and less than the high anomaly threshold, then the detection result of the water supply pipe is determined to be a slightly abnormal state. If the target score is greater than or equal to the high anomaly threshold, then the detection result of the water supply pipe is determined to be in a severely abnormal state.

[0016] By adopting the above technical solution and setting a dual-threshold rule, three states can be distinguished: normal, mildly abnormal, and severely abnormal. The judgment is based on the relationship between the target score and the set low and high thresholds. A target score below the low threshold is considered normal, above the low threshold but below the high threshold is considered mildly abnormal, and above or equal to the high threshold is considered severely abnormal. This quantitative method of judging the degree of abnormality allows for differentiated responses to different levels of abnormality, which is highly targeted and conducive to optimizing maintenance and processing procedures.

[0017] Optionally, the method further includes: The detection results of abnormal water supply pipes within a preset time period are collected regularly, and the corresponding target monitoring data are used as feedback data. The network parameters in the isolated forest model are updated based on the feedback data to obtain the updated isolated forest model; Based on the updated isolation, anomaly detection is performed on the new water supply pipe to obtain the anomaly score corresponding to the new water supply pipe.

[0018] By adopting the above technical solution, the isolated forest model is self-optimized through the collection of feedback data. Specifically, the detection results of abnormal states and corresponding monitoring data are collected periodically as feedback data. The model parameters are updated based on the feedback data to obtain an optimized isolated forest model. This model is used for new anomaly detection, which can continuously accumulate detection experience and improve detection accuracy. This incremental learning and continuous optimization mechanism allows the model's detection capability to continuously improve with data, significantly enhancing detection performance.

[0019] A second aspect of this application provides a pipeline leakage noise detection system, the system comprising: The data acquisition module is used to acquire target monitoring data of the water supply pipe; The feature extraction module is used to extract features from the target monitoring data to obtain feature data; An anomaly detection module is used to perform anomaly detection on the water supply pipe based on the feature data and obtain an anomaly score; The scoring adjustment module is used to obtain the pipe parameters of the water supply pipe and the environmental parameters of the water supply pipe, and adjust the abnormal score based on the pipe parameters and the environmental parameters to obtain the target score; The result determination module is used to generate the detection result of the water supply pipe based on the target score.

[0020] A third aspect of this application provides a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps described above.

[0021] A fourth aspect of this application provides an electronic device comprising: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the method steps described above.

[0022] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: Unlike existing technologies that rely solely on simple threshold judgments based on sensor data for anomaly detection, this application extracts features from monitoring data to obtain feature data. Then, based on the feature data, a machine learning model is used to achieve a more accurate initial anomaly score, which can effectively detect potential anomalies in water supply networks. In particular, by acquiring pipeline parameters and environmental parameters and adjusting the anomaly score based on these parameters, the pipeline's own condition and environmental influences can be comprehensively considered to correct the detection results, avoid false alarms caused by environmental changes, and further improve the accuracy of anomaly data detection in pipelines. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a data anomaly detection method for a water supply network provided in an embodiment of this application; Figure 2 This is a schematic diagram of a data anomaly detection system for a water supply network provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0024] Explanation of reference numerals in the attached drawings: 300, electronic device; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0026] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0027] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0028] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0029] Please refer to Figure 1 A flowchart illustrating a data anomaly detection method for water supply networks is presented. This method can be implemented using a computer program, a microcontroller, or run on a data anomaly detection system for water supply networks. The computer program can be integrated into a detection and control platform or run as a standalone application. Specifically, the method includes steps 10 to 50, as follows: Step 10: Obtain target monitoring data for the water supply pipe.

[0030] Water supply pipes refer to the pipes used to transport tap water in an urban water supply network. The water supply network is an important part of urban infrastructure, consisting of a large number of pipelines and ancillary facilities such as valves and water pumps, which transport water resources from the water source to various users.

[0031] Target monitoring data refers to monitoring data that, after preprocessing, can be directly used for anomaly detection and analysis. Various sensors can be deployed at key nodes of the water supply network to collect raw monitoring data in real time, such as flow rate, pressure, and water quality data. Target monitoring data can be obtained by preprocessing these raw monitoring data, such as denoising, missing value imputation, and normalization.

[0032] Specifically, various sensors are deployed at key nodes of the water supply network to collect raw monitoring data in real time, such as flow meters for flow data, pressure sensors for pressure data, and water quality monitoring equipment for water quality parameters such as turbidity and residual chlorine. This raw monitoring data cannot be directly used for modeling due to noise and missing information; therefore, data preprocessing is required.

[0033] In the preprocessing stage, the raw monitoring data is first de-noising to remove noise and outliers. Then, interpolation and averaging methods are used to estimate and impute missing data, ensuring data continuity. Finally, the processed data undergoes normalization and other standardization processes to eliminate the influence of different units of measurement, obtaining a unified numerical range and thus acquiring the target monitoring data. Preprocessing the raw detection data minimizes the adverse effects of data quality factors, thereby improving the effectiveness and reliability of the model. Simultaneously, the rich information contained in the target monitoring data better reflects the actual operating status of the water supply network, making the anomaly detection results based on it more accurate and reliable.

[0034] Based on the above embodiments, as an optional embodiment, the step of acquiring target monitoring data of the water supply pipe may further include the following steps: Step 101: Obtain the raw monitoring data of the water supply pipe in real time.

[0035] Specifically, the operating status of water supply networks is dynamic, and various possible anomalies such as leaks and blockages can occur at any time. Only by collecting and monitoring data in a timely and continuous manner can these anomalies be detected immediately, and based on this, detection and alarms can be triggered to prevent the anomalies from worsening and causing greater losses.

[0036] Various monitoring sensors, such as flow meters, pressure sensors, and turbidity monitors, can be deployed at key nodes and potential leakage areas of the water supply network. These sensors transmit multi-dimensional raw monitoring data to a data center in real time via wired or wireless means. The data center needs to store this multi-dimensional raw monitoring data in real time, including flow rate, pressure, water quality, signal strength, and electricity consumption. Traditional relational databases, due to their row-column storage structure, struggle to efficiently handle such large-scale time-series data continuously generated by timestamps. Therefore, this embodiment employs InfluxDB, a database specifically optimized for time-series data. Leveraging its powerful parallel writing and data compression capabilities, it can quickly store large amounts of timestamp-indexed monitoring data.

[0037] Step 102: Perform noise removal and missing value imputation on the original monitoring data, and then normalize the processed data to obtain the target monitoring data.

[0038] Specifically, to provide a fast data access mechanism, real-time data from InfluxDB is synchronously cached in the in-memory database Redis. Due to Redis's high-performance in-memory access capabilities, it can respond to data query requests in milliseconds, thus meeting the stringent timeliness requirements of anomaly detection. After obtaining the real-time cached data, the data stream is read from Redis using the Source function of the Flink streaming computing framework, and data preprocessing operations are performed in parallel using Flink's FlatMap operator. Specifically, denoising, missing value imputation, and numerical normalization steps can be performed on different types of data, such as traffic and pressure data, to obtain high-quality target monitoring data. Denoising can employ algorithms based on statistical principles, such as median filtering and Kalman filtering, to remove outliers exceeding the normal fluctuation range. Simultaneously, for continuous anomalous data segments, more complex time-series analysis methods, such as the ARIMA model, can be applied to reconstruct and repair the affected data. Missing value imputation can utilize algorithms such as moving averages and LOCF (Last Observed Continuation) to reasonably estimate and impute missing values. Finally, to eliminate the impact of differences in the range of monitoring data with different dimensions on modeling, the processed data needs to be normalized or standardized to convert it into a uniform range of 0-1 or a standard normal distribution. Normalization methods that can be used include Min-Max standardization and Z-Score standardization.

[0039] Step 20: Extract features from the target monitoring data to obtain feature data.

[0040] In this embodiment of the application, feature data refers to the deep-level data feature representation automatically mined from target monitoring data. Feature data can be understood as a high-level abstraction and dimensionality reduction representation of target monitoring data, containing deep-level pattern information inherent in the data, and is an important data foundation for realizing intelligent anomaly detection.

[0041] Specifically, although target monitoring data undergoes preprocessing such as noise reduction, missing value imputation, and normalization, it still retains high dimensionality and redundant information. Directly inputting this high-dimensional data into the model for calculation would not only significantly increase computational complexity but also make it difficult to effectively extract the inherent essential pattern information of the data, thus affecting the accuracy and generalization ability of anomaly detection. In this embodiment, an autoencoder model from deep learning is used for unsupervised feature extraction from target monitoring data. The autoencoder consists of an encoder and a decoder. Using a pre-trained autoencoder model, target monitoring data is input into the encoder, automatically obtaining low-dimensional feature vectors, achieving efficient compression and noise reduction of the original data. However, directly using the encoder output as feature data may still result in redundancy and noise. Therefore, the low-dimensional features output by the encoder are input into the decoder to reconstruct the original input data. By comparing the difference between the decoded output and the original data, the effectiveness of the low-dimensional features in reconstructing the data can be evaluated, thereby filtering the feature quality. After decoding, features with large decoding reconstruction errors are considered to fail to capture the key information of the monitoring data and should be discarded. Features with smaller decoding errors are considered more representative and informative. Based on this principle, feature selection mechanisms such as cross-validation are further introduced to score and rank the decoded features, and only the highest-scoring and most representative feature subset is retained as the final output feature data.

[0042] Based on the above embodiments, as an optional embodiment, the step of receiving the audio signal corresponding to the noise acquisition point may further include the following steps: Step 201: Input the target monitoring data into the preset autoencoder to obtain low-dimensional features.

[0043] Specifically, the target monitoring data is first effectively compressed and its features abstracted using an autoencoder model. This maps high-dimensional data to a low-dimensional space, resulting in a dimensionality-reduced feature representation. This not only retains the main information of the original data but also removes redundancy and reduces noise, thereby simplifying the feature space, mitigating the risk of overfitting, and laying a solid data foundation for subsequent anomaly detection. Firstly, an autoencoder neural network model needs to be trained using a large number of monitoring data samples. The encoder maps the input data x to a low-dimensional hidden layer representation z through a nonlinear coding layer, which can be expressed as z = f(Wx + b), where z is the low-dimensional feature, W is the weight matrix, b is the bias vector, and f is the nonlinear activation function. The autoencoder automatically learns the effective representation z of the data by minimizing the difference between the input data and the reconstructed data. This ensures that z not only contains the main information of the original data but also removes redundancy and reduces noise.

[0044] Step 202: Decode the low-dimensional features to obtain the decoded features.

[0045] Specifically, although the low-dimensional features z obtained through encoder network compression have advantages over the original high-dimensional data, such as redundancy removal, noise resistance, and dimensionality reduction, they may still contain some invalid components and cannot fully reflect all the effective information of the original data x. Therefore, it is necessary to use an autoencoder decoder network to decode and reconstruct the low-dimensional features z, mapping them back to the original high-dimensional space. The decoding process can be represented as: x' = g(Vz + c), where x' is the reconstructed and decoded feature, V is the weight matrix of the decoder, c is the bias vector, and g is the nonlinear activation function of the decoder layer. By inputting the low-dimensional features z into the decoder layer, we can obtain decoded features x' with a similar dimension to the original input x.

[0046] Step 203: Extract features from the decoded features using cross-validation to obtain feature data.

[0047] Specifically, although features processed by an autoencoder have advantages such as dimensionality reduction and noise reduction compared to the original data, their feature space may still have redundancy and noise issues. This application's embodiment employs a cross-validation strategy for feature extraction. Cross-validation is a machine learning model evaluation and selection method. By dividing the original data into training and validation sets and repeatedly using them interchangeably, the importance and quality of features can be evaluated more objectively and reliably. A large amount of existing historical monitoring data is divided into K parts, for example, 10 parts. Then, each time, one part is used as the validation set, and the remaining K-1 parts are used as the training set. An anomaly detection model is trained on the training set, and the anomaly detection performance of the model is evaluated on the validation set. The contribution weight of each feature dimension to the performance is recorded. This process is repeated K times, with one part being swapped as the validation set each time. Through the above cross-validation process, feature dimensions that generally contribute low weights across all validation sets can be judged to have little effect on anomaly detection and should be eliminated. Feature dimensions that have high weights in most cases can be considered more representative and discriminative and should be retained. Therefore, the decoded features were sorted and filtered across all dimensions, retaining only the highest-scoring subset as the optimized output feature data. This feature extraction not only further compressed and optimized the feature space but also preserved the key feature dimensions with the strongest anomaly detection capabilities, thereby significantly improving the accuracy and robustness of subsequent anomaly detection.

[0048] Based on the above embodiments, as an optional embodiment, a method for detecting data anomalies in a water supply network further includes a process for constructing an isolated forest model, the specific process of which is as follows: Specifically, in the anomaly detection of water supply network monitoring data in this application embodiment, the isolated forest model is adopted. Historical sample data of the water supply pipeline operation is obtained. This historical sample data has undergone the aforementioned data preprocessing and feature extraction steps and exists in the form of optimized feature data. Moreover, this historical sample data is positive sample data, that is, pipeline monitoring data when the water supply pipeline is normal.

[0049] Based on these high-quality historical sample data, an isolation forest model was constructed. As an unsupervised anomaly detection algorithm, the isolation forest model boasts advantages such as high detection accuracy, good training efficiency, and low memory requirements, making it highly suitable for complex scenarios with large data volumes and high dimensionality, such as water supply networks. The isolation forest consists of multiple independent isolation trees. Each isolation tree repeatedly and randomly partitions the input samples, "isolating" normal data points as much as possible to the leaf nodes until a certain condition is met. Anomaly data points, due to their more skewed feature value distributions, require more random partitioning to be isolated, resulting in longer paths. Based on this principle, the isolation forest calculates the average path length required for each input sample to be isolated, serving as a measure of anomaly score. These scores fit the implicit distribution of the input data and can be used to identify anomalies. A smaller score indicates that the sample point is more easily isolated and more likely to be normal data; conversely, a large score indicates an anomaly. Multiple isolation trees were trained based on the obtained historical sample data. The random partitioning features and partitioning points of each isolation tree were independently sampled, which reduces errors and improves overall detection performance by constructing multiple trees. Once each isolation tree is trained, it forms an isolation forest model corresponding to the water supply pipeline. This isolation forest model can then be used to perform real-time anomaly scoring on newly collected online monitoring data.

[0050] For example, suppose a dataset X corresponds to an isolation tree T, containing multiple levels of nodes N. At each level of node N... ij Above, a feature q and its value range p are randomly selected, and data less than or equal to p are assigned to the left child node N. (i+1)jl Nodes greater than p are partitioned to their right child node N. (i+1)(j+1)r When the set maximum number of layers is reached or a node contains a single data point, the segmentation stops and a complete isolation tree is obtained. Finally, the multiple isolation trees built based on each random representation are integrated into an isolation forest model using stochastic gradient descent. This ensemble strategy not only reduces the variance of individual models but also improves the generalization ability of detection.

[0051] Step 30: Perform anomaly detection on the water supply pipe based on feature data and obtain anomaly score.

[0052] Specifically, the trained isolation forest model learns the inherent distribution of the normal operating mode of the water supply pipes, enabling it to efficiently identify abnormal situations that deviate from this distribution. The model continuously acquires the latest online monitoring raw data from each water supply pipe and performs the same preprocessing steps as the training data on this new data, including data cleaning and autoencoder feature extraction. Through this process, the new data is transformed into low-dimensional optimized feature data with the same feature form as the training samples. Then, the obtained feature data at the current time point is input into the trained isolation forest model, which can calculate the anomaly score of the water supply pipe based on the learned normal operating mode. Specifically, normal data points, because they conform to the learned distribution pattern, are quickly isolated at shallower nodes in the tree; while abnormal data, due to their significantly deviated feature value distribution, requires multiple random partitions to be completely isolated to leaf nodes. Therefore, the "isolation path" for abnormal data in the isolation tree is significantly longer. For each isolation tree, the path length for completely isolating the input feature data is calculated, and then these path length values ​​are averaged as the final anomaly score for that feature data. The higher the anomaly score, the more the water supply pipe deviates from normal operating conditions, and vice versa.

[0053] Step 40: Obtain the pipe parameters of the water supply pipe and the environmental parameters of the water supply pipe. Adjust the anomaly score based on the pipe parameters and environmental parameters to obtain the target score.

[0054] Specifically, real-time data on water supply pipeline parameters such as flow rate, pressure, and velocity, as well as environmental parameters such as temperature, humidity, and construction conditions, are acquired from the field data acquisition station. These pipeline and environmental parameters are then input into a pre-trained encoder model to obtain corresponding encoding vectors: pipeline parameter codes and environmental parameter codes. Next, the initial anomaly score output from the isolated forest model in the previous step is concatenated with the pipeline and environmental parameter codes to construct a multi-dimensional fusion vector. This fusion vector integrates comprehensive information on the pipeline network's operational status and the external environment. This fusion vector is then input into an attention mechanism model. The attention mechanism autonomously learns the correlation between the components in the fusion vector and dynamically assigns weight coefficients, resulting in a weighted target vector. Specifically, if a parameter code has a higher correlation with the initial score, the attention model will assign it a larger weight, making its role in the final target vector more prominent. For example, if the pipeline pressure parameter code has a high correlation with the anomaly score, the attention model will assign a larger weight to the pressure code to emphasize the impact of pressure anomalies on the score. Similarly, the weights of environmental factors such as temperature and construction conditions are automatically adjusted, while components less relevant to the score are adaptively assigned smaller weights to avoid noise interference. Through this intelligent weighted summation calculation, a highly focused and optimized target vector is obtained, which automatically incorporates the complex nonlinear relationships between pipeline parameters, environmental parameters, and the preliminary score. Finally, decoding this vector outputs the final target anomaly score.

[0055] Based on the above embodiments, as an optional embodiment, the step of adjusting the anomaly score based on pipeline parameters and environmental parameters to obtain the target score may further include the following steps: Step 401: Input the pipeline parameters and environmental parameters into the preset encoder to obtain the pipeline parameter code and environmental parameter code respectively.

[0056] Specifically, it's important to note that pipeline parameters (such as flow rate, pressure, and velocity) and environmental parameters (such as temperature, humidity, and construction conditions) are highly heterogeneous. They not only differ in dimension but also in their units of measurement and numerical distribution. Simply concatenating them into a model can easily "overwhelm" effective features, affecting subsequent feature representation and model learning. Therefore, these heterogeneous parameters are input separately into a pre-trained encoder. This encoder can be a deep neural network structure capable of automatically learning the inherent distribution characteristics of the parameter data and effectively encoding them into continuous encoding vectors. These encoding vectors not only have uniform dimension but also retain the original feature information of the parameters to the maximum extent. Traditional simple concatenation methods struggle to discover deep relationships between parameters and are prone to information loss. Therefore, these parameters are input separately into the pre-trained encoder network to obtain corresponding encoding vectors, namely, pipeline parameter encoding and environmental parameter encoding. These encodings have strong feature representation capabilities and can fully preserve the original parameter information. Step 402: Concatenate the anomaly score, pipeline parameter code, and environmental parameter code into a fusion vector.

[0057] Specifically, relying solely on preliminary anomaly scores or single-parameter codes is insufficient to fully reflect the complex operational status of pipeline networks. However, by concatenating these into a fusion vector, multi-source heterogeneous information can be fully integrated, capturing the intrinsic relationships between parameters and providing a more comprehensive and richer information carrier for score adjustments. The concatenation method involves sequentially piecing together preliminary anomaly scores, pipeline parameter codes, and environmental parameter codes to form the fusion vector. This fusion vector integrates comprehensive information such as the current anomaly score status of the pipeline network, pipeline operational status, and external environmental influences, providing rich and complex feature representations for subsequent models. More importantly, the construction of the fusion vector enables the subsequent attention adjustment model to autonomously discover and capture the intrinsic relationships between various components, rather than simply performing a linear weighted summation of the components.

[0058] Step 403: Input the fusion vector into the attention mechanism model to obtain the weighted sum of the fusion vectors as the target vector.

[0059] Specifically, after constructing a multi-dimensional fusion vector containing preliminary anomaly scores, pipeline parameter codes, and environmental parameter codes, the influence of these heterogeneous information components on the final score varies. Therefore, simply summing them with equal weights is insufficient to accurately represent the complex intrinsic mapping relationship between parameters and scores. To address this issue, this application's embodiments design an adaptive weighted fusion model based on an attention mechanism. This model takes the fusion vector as input, automatically analyzes the influence of each component on the anomaly score, and assigns an adaptive attention weight coefficient to each component. Specifically, if the model finds a strong correlation between a component (such as pipeline parameter codes) and the score, that component will be assigned a higher attention weight; conversely, if the influence of a component (such as environmental parameter codes) is small, that component will be assigned a lower attention weight. After obtaining the adaptive attention weights for each component, the attention model uses these weights to perform a weighted summation of the components of the fusion vector to obtain a target vector. This target vector integrates all component information and fully considers the different influences of each component on the score, thus more accurately representing the complex nonlinear mapping relationship between heterogeneous information and the score.

[0060] Step 404: Input the target vector into the decoder to obtain the target score.

[0061] Specifically, the target vector, being a continuous vector, is difficult to directly output as an anomaly score. Therefore, the target vector needs to be input into the decoder to map it back into the scalar space of the score, obtaining the actual target score value. The decoder typically employs a deep neural network structure, capable of autonomously learning the high-dimensional mapping relationship between the target vector and the score scalar. During network training, known standard score targets are used as supervised signals to guide the decoder network in gradually optimizing its parameters, ultimately achieving high-precision vector decoding capabilities. After the target vector output from the attention model is input into the fully trained decoder, the decoder decodes and normalizes the rich information contained in the target vector based on the learned mapping pattern, thereby accurately outputting a final target score value in scalar form. This target score value integrates comprehensive information such as the initial score status of the pipeline network, pipeline operating parameters, and external environmental influences.

[0062] Step 50: Generate the test results of the water supply pipe based on the target score.

[0063] Specifically, this application embodiment pre-sets two thresholds: a low anomaly threshold and a high anomaly threshold. These two thresholds divide the range of scoring values ​​into three intervals, corresponding to the normal state, the mildly abnormal state, and the severe abnormal state, respectively.

[0064] When the target score is less than or equal to the low anomaly threshold, it indicates that the water supply pipeline is operating normally with no abnormal risks, and is therefore classified as normal. When the target score is greater than the low anomaly threshold but less than the high anomaly threshold, it indicates that the water supply pipeline has some abnormal risks, but has not yet reached a severe level, and is therefore classified as a mild anomaly. In the mild anomaly state, pipeline maintenance personnel can take appropriate preventive measures to prevent the abnormal situation from worsening. When the target score is greater than or equal to the high anomaly threshold, it indicates that the water supply pipeline is facing a high anomaly risk and is highly likely to experience a serious failure, and is therefore classified as a severe anomaly. Once this occurs, maintenance personnel need to immediately take emergency measures, investigate the cause of the failure, and prevent more serious consequences. Different anomaly levels correspond to different follow-up handling strategies. Maintenance personnel can rationally allocate resources and formulate response measures according to specific circumstances to achieve more scientific and efficient intelligent management of the water supply network.

[0065] Based on the above embodiments, as an optional embodiment, a data anomaly detection method for a water supply network further includes a model update process, specifically comprising the following steps: Specifically, during routine operation and maintenance, the abnormal status of each water supply pipe in the pipeline network is continuously monitored. For those pipes identified as being in abnormal states—that is, those with minor or severe anomalies—all monitoring data during the period in which the anomaly occurred is collected and retained as a feedback data source. This feedback data includes not only the scoring results at the time of the anomaly but also information such as all operating parameters and environmental parameters of the pipeline network. Based on these known abnormal state detection results, the network parameters of each sub-module in the isolated forest model, such as the encoder, attention model, and decoder, are re-optimized and adjusted. Specifically, through supervised learning, the model iteratively strengthens parameters that can improve the accuracy of anomaly detection while weakening parameters that are not beneficial to the results, ultimately resulting in an updated model with better performance and a more realistic understanding of the situation. After the parameter update is completed, this optimized isolated forest model can be applied to new water supply pipelines for anomaly detection and scoring. At this point, because the model parameters have been optimized and improved, the output anomaly scoring results will be more accurate and reliable than before, accurately reflecting the actual operational risk status of the new pipelines.

[0066] Please see Figure 2 This is a schematic diagram of a data anomaly detection system for a water supply network provided in an embodiment of this application. The system may include: a data acquisition module, a feature extraction module, an anomaly detection module, a scoring adjustment module, and a result determination module, wherein: The data acquisition module is used to acquire target monitoring data of the water supply pipe; The feature extraction module is used to extract features from the target monitoring data to obtain feature data; An anomaly detection module is used to perform anomaly detection on the water supply pipe based on the feature data and obtain an anomaly score; The scoring adjustment module is used to obtain the pipe parameters of the water supply pipe and the environmental parameters of the water supply pipe, and adjust the abnormal score based on the pipe parameters and the environmental parameters to obtain the target score; The result determination module is used to generate the detection result of the water supply pipe based on the target score.

[0067] Optionally, the data acquisition module is also used to acquire the raw monitoring data of the water supply pipe in real time; to perform noise removal and missing value filling processing on the raw monitoring data respectively, and to perform data normalization processing on the processed data to obtain the target monitoring data.

[0068] Optionally, the feature extraction module is further configured to input the target monitoring data into a preset autoencoder to obtain low-dimensional features; decode the low-dimensional features to obtain decoded features; and extract features from the decoded features based on cross-validation to obtain feature data.

[0069] Optionally, the anomaly detection module is further configured to acquire historical sample data of the water supply pipe, wherein the historical sample data is feature data after data preprocessing and feature extraction; construct multiple isolation trees based on the historical sample data, and form an isolation forest model based on each isolation tree; perform anomaly detection on the water supply pipe based on the feature data to obtain an anomaly score, including: inputting the feature data into the isolation forest model, calculating the path length of the feature data under each isolation tree in the isolation forest model, and averaging the path lengths to obtain the anomaly score.

[0070] Optionally, the scoring adjustment module is further configured to input the pipeline parameters and the environment parameters into a preset encoder to obtain pipeline parameter codes and environment parameter codes respectively; concatenate the abnormal score, the pipeline parameter codes, and the environment parameter codes into a fusion vector; input the fusion vector into an attention mechanism model to obtain a weighted sum of the fusion vectors as a target vector, wherein the weight coefficients of the attention mechanism model are dynamically allocated based on the correlation between the abnormal score, the pipeline parameter codes, and the environment parameter codes; and input the target vector into a decoder to obtain a target score.

[0071] Optionally, the result determination module is further configured to determine the detection result of the water supply pipe as normal if the target score is less than or equal to the low anomaly threshold; determine the detection result of the water supply pipe as slightly abnormal if the target score is greater than the low anomaly threshold and less than the high anomaly threshold; and determine the detection result of the water supply pipe as severely abnormal if the target score is greater than or equal to the high anomaly threshold.

[0072] Optionally, a data anomaly detection system for a water supply network further includes a model update module. The model update module is used to periodically collect detection results of abnormal states of the water supply pipes within a preset time period, and to use the corresponding target monitoring data as feedback data; to update the network parameters in the isolated forest model based on the feedback data, thereby obtaining an updated isolated forest model; and to perform anomaly detection on new water supply pipes based on the updated isolated forest model, thereby obtaining an anomaly score corresponding to the new water supply pipes.

[0073] It should be noted that the system provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0074] This application also provides a computer storage medium that can store multiple instructions. The instructions are adapted to be loaded and executed by a processor as described in the above embodiment of a data anomaly detection method for a water supply network. For the specific execution process, please refer to the detailed description of the above embodiment, which will not be repeated here.

[0075] Please refer to Figure 3 This application also discloses an electronic device. Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. The electronic device 300 may include: at least one processor 301, at least one network interface 304, a user interface 303, a memory 305, and at least one communication bus 302.

[0076] The communication bus 302 is used to enable communication between these components.

[0077] The user interface 303 may include a display screen and a camera. Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0078] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0079] The processor 301 may include one or more processing cores. The processor 301 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 305, and by calling data stored in memory 305. Optionally, the processor 301 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 301 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 301 and may be implemented as a separate chip.

[0080] The memory 305 may include random access memory (RAM) or read-only memory. Optionally, the memory 305 may include a non-transitory computer-readable storage medium. The memory 305 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 305 may also be at least one storage device located remotely from the aforementioned processor 301. (Refer to...) Figure 3 The memory 305, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for a data anomaly detection method for a water supply network.

[0081] exist Figure 3 In the illustrated electronic device 300, the user interface 303 is mainly used to provide an input interface for the user and acquire user input data; while the processor 301 can be used to call an application program stored in the memory 305 for a data anomaly detection method for a water supply network. When executed by one or more processors 301, the electronic device 300 performs one or more of the methods described in the above embodiments. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0082] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0083] In the various embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.

[0084] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0086] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0087] The above description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will be readily apparent to those skilled in the art upon consideration of the specification and the disclosure of practical truths.

[0088] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A method for detecting data anomalies in a water supply network, characterized in that, The method includes: Obtain target monitoring data for the water supply pipes; The process of extracting features from the target monitoring data to obtain feature data includes: inputting the target monitoring data into a preset autoencoder to obtain low-dimensional features; decoding the low-dimensional features to obtain decoded features; and extracting features from the decoded features based on cross-validation to obtain feature data. Anomaly detection is performed on the water supply pipe based on the feature data to obtain an anomaly score; The process involves obtaining the pipe parameters of the water supply pipe and the environmental parameters of the water supply pipe's location, and adjusting the anomaly score based on the pipe parameters and environmental parameters to obtain a target score. This includes: inputting the pipe parameters and environmental parameters into a preset encoder to obtain pipe parameter codes and environmental parameter codes; concatenating the anomaly score, pipe parameter codes, and environmental parameter codes into a fusion vector; inputting the fusion vector into an attention mechanism model to obtain a weighted sum of the fusion vectors as a target vector, where the weight coefficients of the attention mechanism model are dynamically allocated based on the correlation between the anomaly score, pipe parameter codes, and environmental parameter codes; and inputting the target vector into a decoder to obtain the target score. The detection results of the water supply pipe are generated based on the target score; Before performing anomaly detection on the water supply pipe based on the aforementioned feature data and obtaining an anomaly score, the process further includes: Obtain historical sample data of the water supply pipe, wherein the historical sample data is feature data after data preprocessing and feature extraction; Multiple isolation trees are constructed based on the historical sample data, and an isolation forest model is formed based on each of the isolation trees; Based on the aforementioned feature data, anomaly detection is performed on the water supply pipe to obtain an anomaly score, including: The feature data is input into the isolated forest model, the path length of the feature data under each isolated tree in the isolated forest model is calculated, and the average of the path lengths is calculated to obtain the anomaly score; The method further includes: periodically collecting detection results of abnormal water supply pipes within a preset time period, and using the corresponding target monitoring data as feedback data; updating the network parameters in the isolation forest model based on the feedback data to obtain an updated isolation forest model; and performing anomaly detection on new water supply pipes based on the updated isolation forest model to obtain anomaly scores corresponding to the new water supply pipes.

2. The data anomaly detection method for a water supply network according to claim 1, characterized in that, The acquisition of target monitoring data for the water supply pipe includes: Real-time acquisition of raw monitoring data from water supply pipes; The original monitoring data is subjected to denoising and missing value imputation, and the processed data is then normalized to obtain the target monitoring data.

3. The data anomaly detection method for a water supply network according to claim 1, characterized in that, The generation of the detection result for the water supply pipe based on the target score includes: If the target score is less than or equal to the low anomaly threshold, the detection result of the water supply pipe is determined to be in a normal state. If the target score is greater than the low anomaly threshold and less than the high anomaly threshold, then the detection result of the water supply pipe is determined to be a slightly abnormal state. If the target score is greater than or equal to the high anomaly threshold, then the detection result of the water supply pipe is determined to be in a severely abnormal state.

4. A data anomaly detection system for a water supply network, characterized in that, The system for performing the data anomaly detection method for a water supply network as described in claim 1 includes: The data acquisition module is used to acquire target monitoring data of the water supply pipe; The feature extraction module is used to extract features from the target monitoring data to obtain feature data; An anomaly detection module is used to perform anomaly detection on the water supply pipe based on the feature data and obtain an anomaly score; The scoring adjustment module is used to obtain the pipe parameters of the water supply pipe and the environmental parameters of the water supply pipe, and adjust the abnormal score based on the pipe parameters and the environmental parameters to obtain the target score; The result determination module is used to generate the detection result of the water supply pipe based on the target score.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted to be loaded by a processor and executed as described in any one of claims 1-3.

6. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-3.

Images