Abnormal data detection and recovery method and system for report form of water plant management and control platform

Abstract

The invention provides an abnormal data detection and recovery method and system for reports of a water plant management and control platform, and relates to the technical field of water plant data detection.The abnormal data detection and recovery method comprises the steps that report data of the water plant management and control platform are collected, then the threshold range of the report data is obtained through threshold detection, and the report data exceeding the threshold range are marked as potential abnormal data; performing exception scoring on each potential exception data in the report data by adopting an exception detection model, and then marking the potential exception data with an exceptional exception scoring result as burr data points; and deleting each burr data point in the report data, and then estimating missing data points by utilizing a linear relationship between adjacent data points of the deleted burr data points to carry out linear interpolation so as to carry out data recovery on the report data. The method has the beneficial effects that the accuracy and stability of data analysis of the water plant management and control platform are improved through multi-stage anomaly detection, and efficient detection and recovery of abnormal data are realized through combination of statistical analysis and machine learning technologies.

Description

Technical Field

[0001] The present invention relates to the technical field of water plant data detection, and in particular to a method and system for detecting and recovering abnormal data in a water plant management and control platform report. Background Art

[0002] In the operation of modern water plant control platforms, data quality has always been a key issue. However, due to various complex factors such as environmental factors, sensor failures, and data transmission, abnormal data may appear in reports from time to time. The existence of these abnormal data will greatly affect the accuracy and reliability of the data, bringing difficulties to the monitoring and decision-making of the water plant. Therefore, in order to address this problem, an efficient and reliable method is needed to detect and repair these abnormal data. Summary of the invention

[0003] In view of the problems existing in the prior art, the present invention provides a method for detecting and recovering abnormal data in a water plant control platform report, comprising:

[0004] Step S1, collecting report data from the water plant management and control platform, then obtaining a threshold range of the report data through threshold detection, and marking each report data exceeding the threshold range as potential abnormal data;

[0005] Step S2, using a pre-trained anomaly detection model to perform anomaly scoring on each of the potential abnormal data in the report data to obtain a corresponding abnormal scoring result, and then marking the potential abnormal data whose abnormal scoring result exceeds a set scoring range as a burr data point;

[0006] Step S3, deleting each of the burr data points in the report data, and then using the linear relationship of adjacent data points of the deleted burr data points to estimate the missing data points for linear interpolation, so as to recover the report data.

[0007] Preferably, the report data includes multiple data dimensions, and the step S1 includes:

[0008] Step S11, calculating the mean and standard deviation of each data dimension, and then obtaining the threshold range according to the mean and the standard deviation;

[0009] Step S12: marking the data points in the report data that exceed the threshold range as the potential abnormal data.

[0010] Preferably, the training process of the isolation forest algorithm model includes:

[0011] Step A1, obtaining historical time series data, and converting the historical time series data into an isolation forest algorithm format to obtain a feature data set;

[0012] Step A2, performing feature statistics on the feature data set to obtain statistical features, and then setting the contamination rate parameter of the isolation forest algorithm as the initial contamination rate parameter to perform anomaly detection on the feature data set to obtain a corresponding preliminary anomaly detection result;

[0013] Step A3, adjusting the contamination rate parameter of the isolation forest algorithm according to the preliminary anomaly detection result and the statistical characteristics to obtain the anomaly detection model.

[0014] Preferably, before threshold detection is performed on the report data, a data preprocessing process is also included, and the data preprocessing process includes:

[0015] The report data is cleaned, then missing values ​​are filled in the cleaned report data, and finally data smoothing is performed on the filled report data to obtain the report data for threshold detection.

[0016] Preferably, after executing step S3, the method further includes:

[0017] Step S4, performing data smoothing and moving average processing on the report data after interpolation restoration, then using a polynomial regression correction method to perform data correction on the report data after moving average, and finally using a polynomial regression model to perform data fitting on the corrected report data to obtain the revised report data.

[0018] Preferably, after executing step S3, a model optimization process is also included, including:

[0019] The audit results of the auditor on each of the burr data points are input into the anomaly detection model as feedback data, and the anomaly detection model is retrained to adjust the model parameters of the anomaly detection model.

[0020] The present invention also provides an abnormal data detection and recovery system for a water plant control platform report, which uses the above-mentioned abnormal data detection and recovery method, including:

[0021] A potential marking module is used to collect report data from the water plant management and control platform, and then obtain a threshold range of the report data through threshold detection, and mark each report data exceeding the threshold range as potential abnormal data;

[0022] an anomaly detection module, connected to the potential marking module, for using a pre-trained anomaly detection model to perform anomaly scoring on each of the potential anomaly data in the report data to obtain a corresponding anomaly scoring result, and then marking the potential anomaly data whose anomaly scoring result exceeds a set scoring range as a burr data point;

[0023] Each of the potential abnormal data is scored for abnormality, and then the potential abnormal data is marked as a burr data point according to each of the abnormality scoring results;

[0024] A data recovery module is connected to the anomaly detection module and is used to delete each of the burr data points in the report data, and then use the linear relationship of adjacent data points of the deleted burr data points to estimate the missing data points for linear interpolation to recover the report data.

[0025] Preferably, the report data includes multiple data dimensions, and the potential marking module includes:

[0026] A range calculation unit, used to calculate the mean and standard deviation of each data dimension, and then obtain the threshold range according to the mean and the standard deviation;

[0027] A potential marking unit is connected to the range calculation module and is used to mark the data points in the report data that exceed the threshold range as the potential abnormal data.

[0028] Preferably, a model training module is further included, connected to the anomaly detection module, including:

[0029] A data set acquisition unit, used to acquire historical time series data, and convert the historical time series data into an isolation forest algorithm format to obtain a feature data set;

[0030] an initial detection unit, connected to the data set acquisition unit, for performing feature statistics on the feature data set to obtain statistical features, and then setting the contamination rate parameter of the isolation forest algorithm as the initial contamination rate parameter to perform anomaly detection on the feature data set to obtain a corresponding preliminary anomaly detection result;

[0031] A parameter adjustment unit is connected to the initial detection unit and is used to adjust the contamination rate parameter of the isolation forest algorithm according to the preliminary anomaly detection result and the statistical feature to obtain the anomaly detection model.

[0032] Preferably, it also includes a data preprocessing module, which is connected to the potential marking module and is used to clean the report data, then fill the missing values ​​of the cleaned report data, and finally smooth the filled report data to obtain the report data for threshold detection.

[0033] The above technical solution has the following advantages or beneficial effects: with the actual report data of the water plant management and control platform as the core, multi-level anomaly detection can effectively identify and process abnormal data in the report, thereby improving the accuracy and stability of data analysis of the water plant management and control platform. By combining statistical analysis and machine learning technology, efficient detection and recovery of abnormal data in the water plant management and control platform are achieved, providing a reliable data foundation for water plant managers and promoting intelligent and refined management of water plant operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The present invention is a flow chart of a method for detecting and recovering abnormal data in a water plant control platform report in a preferred embodiment of the present invention;

[0035] Figure 2 A schematic diagram of a sub-process of step S1 in a preferred embodiment of the present invention;

[0036] Figure 3 A flowchart of the training process of an anomaly detection model in a preferred embodiment of the present invention;

[0037] Figure 4 A schematic diagram of visualization of abnormal data detection in a preferred embodiment of the present invention;

[0038] Figure 5 The present invention is a schematic diagram of the result structure of an abnormal data detection and recovery system for a water plant management and control platform report in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The present invention is not limited to this embodiment, and other embodiments may also fall within the scope of the present invention as long as they conform to the gist of the present invention.

[0040] In a preferred embodiment of the present invention, based on the above problems existing in the prior art, a method for detecting and recovering abnormal data in a water plant control platform report is provided. Figure 1 As shown, including:

[0041] Step S1, collecting report data from the water plant management and control platform, then obtaining a threshold range of the report data through threshold detection, and marking each report data exceeding the threshold range as potential abnormal data;

[0042] Step S2, using a pre-trained anomaly detection model to perform anomaly scoring on each potential abnormal data in the report data to obtain a corresponding abnormal scoring result, and then marking the potential abnormal data whose abnormal scoring result exceeds the set scoring range as a burr data point;

[0043] Step S3, deleting each burr data point in the report data, and then using the linear relationship of adjacent data points of the deleted burr data point to estimate the missing data point for linear interpolation, so as to recover the report data.

[0044] Specifically, this example takes the actual report data of the water plant management and control platform as the core, and realizes efficient detection of abnormal points (i.e., abnormal data) in the report data by applying the isolation forest algorithm, and uses a specific recovery strategy to correct the detected abnormal data. The method of the present invention realizes efficient detection and recovery of abnormal data in the water plant management and control platform by combining statistical analysis with machine learning technology. Multi-level anomaly detection can effectively identify and process abnormal data in the report, thereby improving the accuracy and stability of data analysis of the water plant management and control platform, providing a reliable data foundation for water plant managers, and promoting intelligent and refined management of water plant operations.

[0045] Furthermore, the report data includes multiple data dimensions, and step S1 includes:

[0046] Step S11, calculating the mean and standard deviation of each data dimension, and then obtaining a threshold range based on the mean and standard deviation;

[0047] Step S12: Mark the data points in the report data that exceed the threshold range as potential abnormal data.

[0048] Specifically, in this embodiment, the data report adopts a report of water volume related data recorded by the water plant management and control platform of a smart water plant in Ningbo, Zhejiang. The report has a daily scheduled task that reads yesterday's 24-hour data from the device platform at a granularity of 5 minutes and stores it in a data table in the database. The collected report data is time series data. The data dimensions in the report include clear water tank level (CLEAR_POOL_WATER_LEVEL), water plant inlet flow (WATER_INFLOW_INSTANT), water plant outlet flow (WATER_SUPPLY_INSTANT), clear water tank level, clear water tank inlet and outlet flow, water plant inlet and outlet pressure, etc.

[0049] In this example, threshold detection is first performed to calculate the mean (μ) and standard deviation (σ) of each data dimension, and the threshold range based on the standard deviation is determined to be [μ-3σ,μ+3σ]. Data points outside this range are marked as potential abnormal data.

[0050] Specifically, in one of the embodiments, taking the water plant outflow as an example, the following Table 1 is part of the water plant outflow data in the collected water plant data report, the collection time is 9:00-10:00, and the collection interval is 5 minutes.

[0051] Table 1 Water plant outflow data table (unit: cubic meters / second)

[0052]

[0053]

[0054] After calculation, the mean of the 12 data is about 3.04, and the standard deviation is about 0.60. Further, the threshold range based on the standard deviation is:

[0055] [3.04-3x0.60,3.04+3x0.60]=[1.24,4.84]

[0056] In Table 1, the data 5.21 at 9:25 exceeds the threshold range, so 5.21 is marked as potential abnormal data.

[0057] Specifically, in another embodiment, taking the clear water tank level as an example, the following Table 2 is part of the clear water tank level data in the collected water plant data report, the collection time is 9:00-11:00, and the collection interval is 5 minutes.

[0058] Table 2 Clear water tank level data table (unit: meter)

[0059]

[0060] After calculation, the mean of these 24 data is about 2.73, and the standard deviation is about 1.17. Furthermore, the threshold range based on the standard deviation is:

[0061] [2.73-3x1.17,2.73+3x1.17]=[-0.78,6.24]

[0062] In Table 2, the data 7.22 at 9:50 exceeds the threshold range, so 7.22 is marked as potential abnormal data.

[0063] Secondly, each potential abnormal data point is input into the anomaly detection model to calculate the anomaly score of each potential data point. According to the scoring result, the potential abnormal data point that exceeds the set scoring range is marked as a burr data point.

[0064] Furthermore, in a preferred embodiment of the present invention, the training process of the anomaly detection model includes:

[0065] Step A1, obtaining historical time series data, converting the historical time series data into the isolation forest algorithm format to obtain a feature data set;

[0066] Step A2, performing feature statistics on the feature data set to obtain statistical features, and then setting the contamination rate parameter of the isolation forest algorithm as the initial contamination rate parameter to perform anomaly detection on the feature data set to obtain a corresponding preliminary anomaly detection result;

[0067] Step A3, adjusting the contamination rate parameter of the isolation forest algorithm according to the preliminary anomaly detection results and statistical characteristics to obtain an anomaly detection model.

[0068] Specifically, in this embodiment, an isolation forest algorithm model is constructed to count abnormal data in the report of the water plant management and control platform. First, the historical time series data is converted into an input format suitable for the isolation forest algorithm to obtain a feature data set, which is used to train the isolation forest model to obtain an anomaly detection model;

[0069] In order to further optimize the detection effect, in the training process of the isolation forest algorithm model, an adaptive contamination rate method is introduced. The contamination rate parameter determines the proportion of abnormal data points. According to the actual distribution of the data, the contamination rate parameter in the isolation forest algorithm is dynamically adjusted to accurately identify abnormal data points to improve the detection effect of the model. In addition, the cross-validation method is introduced to train and validate the isolation forest model. Through cross-validation, the performance of the model can be better evaluated to avoid overfitting problems.

[0070] The calculation of the adaptive pollution rate introduced in this example includes the following steps:

[0071] First, statistical analysis is performed to calculate statistical characteristics such as mean, standard deviation, skewness, and kurtosis of the feature data set;

[0072] Then, preliminary anomaly detection of the isolation forest algorithm is performed using the initially set contamination rate parameter;

[0073] Finally, based on the preliminary detection results and statistical characteristics, the contamination rate parameters are dynamically adjusted using empirical formulas; the adjusted contamination rate parameters enable the isolation forest model to more accurately identify abnormal data points.

[0074] The trained anomaly detection model is used for scoring, and according to the scoring results, the abnormal data points are marked as burr data points. In this way, the adaptive contamination rate method effectively improves the accuracy and robustness of anomaly data detection.

[0075] After detecting the glitch data points, first remove all the glitch data points marked as abnormal from the report data. In order to ensure the continuity and integrity of the data, the linear interpolation method is then used to restore these missing data points. Linear interpolation estimates the value of the missing data point by using the values ​​of the adjacent data points before and after the glitch data point, thereby generating a smooth and continuous time series data to complete data recovery.

[0076] This method effectively fills the gaps caused by the deletion of abnormal data, ensures the integrity and consistency of the data, and facilitates subsequent data analysis and application. Specifically, through linear interpolation, the trend and fluctuation of the data can be effectively restored, so that the entire time series data still maintains a reasonable change pattern and logical consistency after processing.

[0077] Specifically, in one of the embodiments, the abnormal data recovery process for the water plant outflow data table in Table 1 in the aforementioned embodiment is as follows:

[0078] The data sets in this data table are:

[0079] {3.01,2.41,2.87,2.66,3.23,5.21,3.07,2.73,2.89,2.50,3.12,2.77}

[0080] After the anomaly detection model has been used to score, 5.21 is determined to be a burr data and deleted from the data table. The data set after deletion is:

[0081] {3.01,2.41,2.87,2.66,3.23,3.07,2.73,2.89,2.50,3.12,2.77}

[0082] The data point before 5.21 is 3.23, and the data point after is 3.07.

[0083] The linear interpolation formula is:

[0084] y=y1+(x2-x1 / y2-y1)×(x-x1)

[0085] But in this embodiment, we don't actually need to use the x-coordinate (because the data points are arranged in order and there is no clear x-coordinate corresponding to them), we just need to know between which two data points the interpolation is performed. Therefore, the formula can be simplified to:

[0086] Interpolation = y1 + (data point interval y2-y1) × interpolation position offset

[0087] In this particular embodiment, since the data points are arranged consecutively and what we want to recover is the value of the sixth data point, the "interpolation position offset" is actually 1 (because what we want to find is an intermediate value between two known points, but in this embodiment, since there is only one missing value and interpolation is performed directly between these two points, the offset of 1 may not be accurate, but it can be regarded as a simplified processing).

[0088] But since we are going to operate according to the linear interpolation method, we can simply take the average of the two data points before and after as the interpolation result (this is actually a special case of linear interpolation in a specific case, that is, when the interpolation position is exactly in the middle of the two data points):

[0089] Interpolation = (y1 + y2) / 2 = (3.23 + 3.07) / 2 = 3.15

[0090] Therefore, 3.15 can be used as the missing value to fill in the position of 5.21.

[0091] In another embodiment, the process of deleting burr data and restoring data in the clean water tank level data table in Table 2 in the above embodiment is as follows:

[0092] The data sets in this data table are:

[0093] {1.75, 2.20, 2.85, 1.90, 3.30, 2.50, 2.05, 3.00, 2.65, 1.60, 7.22, 2.90, 3.03, 2.66, 2.85, 2.55, 2.97, 2.69, 3.06, 2.49, 2.79, 2.63, 2.59, 2.74}

[0094] After the anomaly detection model has been used to score, 5.21 is determined to be a burr data and deleted from the data table. The data set after deletion is:

[0095] {1.75,2.20,2.85,1.90,3.30,2.50,2.05,3.00,2.65,1.60,[missing],2.90,3.03,2.66,2.85,,2.55,2.97,2.69,3.06,2.49,2.79,2.63,2.59,2.74}

[0096] Among them, [missing] indicates the vacancy left after 7.22 is deleted. The data point before 7.22 is 1.60, and the data point after 7.22 is 2.90.

[0097] Next, we need to use linear interpolation to recover the missing value. Linear interpolation usually estimates the unknown value between two known data points.

[0098] The linear interpolation formula is:

[0099] y=y1+(y2-y1 / x2-x1)×(x-x1)

[0100] However, in this embodiment, since the data points are arranged in time order and the collection interval of each data point is fixed (5 minutes), x1 and x2 can be regarded as time indexes (for example, if 1.60 is the first data point, then x1=1, but in this embodiment, the actual time value is not required, only the relative position between them needs to be known). Therefore, the interpolation formula can be simplified to:

[0101] Interpolation = y1 + (y2-y1 / number of data point intervals) × interpolation position offset

[0102] However, in this embodiment, since the data points are arranged continuously and what we want to recover is a missing value between two known data points, we can directly calculate the average of the two data points as the interpolation result (this is actually a special case of linear interpolation when the data point interval is 1):

[0103] Interpolation = (y1 + y2) / = (1.60 + 2.90) / 2 = 2.25

[0104] Therefore, 2.25 can be used as the missing value to fill in the position of 7.22.

[0105] Furthermore, in a preferred embodiment of the present invention, data correction is performed after executing step S3, including:

[0106] Step S4, performing data smoothing and moving average processing on the report data after interpolation restoration, then using a polynomial regression correction method to perform data correction on the report data after moving average, and finally using a polynomial regression model to perform data fitting on the corrected report data to obtain the revised report data.

[0107] Specifically, data recovery and correction in this embodiment are important data processing steps. To ensure the continuity and consistency of the data, this example first smoothes the recovered data. Then, a polynomial regression correction method is used to fit the data using a polynomial regression model. The polynomial regression model corrects the interpolated data by fitting a polynomial curve, which can effectively adjust the data, ensure the smoothness and accuracy of the data, and is suitable for data correction of nonlinear relationships.

[0108] In a preferred embodiment of the present invention, a data preprocessing process is also included before threshold detection is performed on the report data. The data preprocessing process includes:

[0109] The report data is cleaned, and then the missing values ​​are filled in the cleaned report data. Finally, the filled report data is smoothed to obtain the report data for threshold detection.

[0110] Specifically, in this embodiment, data preprocessing of the report data helps to improve the robustness of the isolation forest model. In the data preprocessing step, missing values ​​are filled and data is smoothed and obvious erroneous data points are deleted. The specific steps of this example are as follows:

[0111] Fill missing values: Use forward fill and backward fill methods to fill missing values. Forward fill uses the previous valid data point to fill missing values, and backward fill uses the next valid data point to fill missing values. These methods can effectively handle missing values ​​in time series data and ensure data continuity.

[0112] Data smoothing: Use the moving average method to smooth the data. The moving average method reduces short-term fluctuations in the data by calculating the average value of the data in a fixed window, thereby reducing noise and making the data smoother and more stable. This example uses a 3-point moving average:

[0113] MA t =(X t-1 +X t +X t+1 ) / 3

[0114] Among them, X t is the original data at time t, X t-1 is the original data at time t-1, X t+1 is the original data at time t+1, MA t is the smoothed data.

[0115] In a preferred embodiment of the present invention, after executing step S3, a model optimization process is also included, including:

[0116] The auditor's audit results on each burr data point are input into the anomaly detection model as feedback data, and the anomaly detection model is retrained to adjust the model parameters of the anomaly detection model.

[0117] Specifically, this example will smooth the restored data in the water plant control platform report again to ensure the continuity and consistency of the data. In addition, the feedback mechanism collects abnormal data points and conducts manual review, and then feeds the review results back to the system to update the model; the feedback data is input into the isolation forest algorithm to retrain and adjust the model parameters. Subsequently, the model is evaluated and tested using the new data environment to ensure that its detection performance is continuously optimized. By continuously introducing new feedback data, the performance of the model is further optimized, enhancing the accuracy and robustness of anomaly detection.

[0118] Furthermore, data visualization can be achieved. The processed and corrected data will be integrated into the report of the water plant management and control platform to improve the accuracy and reliability of data analysis, thereby providing water plant managers with a more reliable basis for decision-making. At the same time, this example will visualize the test results, such as Figure 4 As shown, it is presented in the form of charts, graphs, etc., so that managers can intuitively monitor the operation status of the water plant and make corresponding decisions.

[0119] The present invention also provides a water plant control platform report abnormal data detection and recovery system, using the above abnormal data detection and recovery method, such as Figure 5 As shown, including:

[0120] Potential marking module 1 is used to collect report data from the water plant management and control platform, and then obtain the threshold range of the report data through threshold detection, and mark each report data exceeding the threshold range as potential abnormal data;

[0121] The anomaly detection module 2 is connected to the potential marking module 1 and is used to use the pre-trained anomaly detection model to perform anomaly scoring on each potential anomaly data in the report data to obtain a corresponding anomaly scoring result, and then mark the potential anomaly data whose anomaly scoring result exceeds the set scoring range as a burr data point;

[0122] The data recovery module 3 is connected to the anomaly detection module 2 and is used to delete each burr data point in the report data, and then use the linear relationship of the adjacent data points of the deleted burr data point to estimate the missing data point for linear interpolation to recover the report data.

[0123] In a preferred embodiment of the present invention, the report data includes multiple data dimensions, such as Figure 5 As shown, the potential marking module 1 includes:

[0124] A range calculation unit 11 is used to calculate the mean and standard deviation of each data dimension, and then obtain a threshold range according to the mean and standard deviation;

[0125] The potential marking unit 12 is connected to the range calculation module 11 and is used to mark data points in the report data that exceed the threshold range as potential abnormal data.

[0126] In a preferred embodiment of the present invention, a model training module 4 is further included, and a connection anomaly detection module 2 includes:

[0127] The data set acquisition unit 41 is used to obtain historical time series data and convert the historical time series data into an isolation forest algorithm format to obtain a feature data set;

[0128] The initial detection unit 42 is connected to the data set acquisition unit 41 and is used to perform feature statistics on the feature data set to obtain statistical features, and then set the contamination rate parameter of the isolation forest algorithm as the initial contamination rate parameter to perform anomaly detection on the feature data set to obtain a corresponding preliminary anomaly detection result;

[0129] The parameter adjustment unit 43 is connected to the initial detection unit 42 and is used to adjust the contamination rate parameter of the isolation forest algorithm according to the preliminary anomaly detection result and the statistical characteristics to obtain the anomaly detection model.

[0130] In a preferred embodiment of the present invention, it also includes a data preprocessing module 5, which is connected to the potential marking module 1 and is used to clean the report data, then fill the missing values ​​of the cleaned report data, and finally smooth the filled report data to obtain the report data for threshold detection.

[0131] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the contents of this specification and illustrations should be included in the protection scope of the present invention.

Claims

1. A method for detecting and recovering abnormal data in a water plant control platform report, characterized in that: include: Step S1, collecting report data from the water plant management and control platform, then obtaining a threshold range of the report data through threshold detection, and marking each report data exceeding the threshold range as potential abnormal data; Step S2, using a pre-trained anomaly detection model to perform anomaly scoring on each of the potential abnormal data in the report data to obtain a corresponding abnormal scoring result, and then marking the potential abnormal data whose abnormal scoring result exceeds a set scoring range as a burr data point; Step S3, deleting each of the burr data points in the report data, and then using the linear relationship of adjacent data points of the deleted burr data points to estimate the missing data points for linear interpolation, so as to recover the report data.

2. The abnormal data detection and recovery method according to claim 1, characterized in that: The report data includes multiple data dimensions, and the step S1 includes: Step S11, calculating the mean and standard deviation of each data dimension, and then obtaining the threshold range according to the mean and the standard deviation; Step S12: marking the data points in the report data that exceed the threshold range as the potential abnormal data.

3. The abnormal data detection and recovery method according to claim 1, characterized in that: The training process of the isolation forest algorithm model includes: Step A1, obtaining historical time series data, and converting the historical time series data into an isolation forest algorithm format to obtain a feature data set; Step A2, performing feature statistics on the feature data set to obtain statistical features, and then setting the contamination rate parameter of the isolation forest algorithm as the initial contamination rate parameter to perform anomaly detection on the feature data set to obtain a corresponding preliminary anomaly detection result; Step A3, adjusting the contamination rate parameter of the isolation forest algorithm according to the preliminary anomaly detection result and the statistical characteristics to obtain the anomaly detection model.

4. The abnormal data detection and recovery method according to claim 1, characterized in that: Before threshold detection is performed on the report data, a data preprocessing process is also included, and the data preprocessing process includes: The report data is cleaned, then missing values ​​are filled in the cleaned report data, and finally data smoothing is performed on the filled report data to obtain the report data for threshold detection.

5. The abnormal data detection and recovery method according to claim 1, characterized in that: After executing step S3, the method further includes: Step S4, performing data smoothing and moving average processing on the report data after interpolation restoration, then using a polynomial regression correction method to perform data correction on the report data after moving average, and finally using a polynomial regression model to perform data fitting on the corrected report data to obtain the revised report data.

6. The abnormal data detection and recovery method according to claim 1, characterized in that: After executing step S3, a model optimization process is also included, including: The audit results of the auditor on each of the burr data points are input into the anomaly detection model as feedback data, and the anomaly detection model is retrained to adjust the model parameters of the anomaly detection model.

7. An abnormal data detection and recovery system for water plant control platform reports, characterized in that: The method for detecting and recovering abnormal data according to any one of claims 1 to 6 is applied, comprising: A potential marking module is used to collect report data from the water plant management and control platform, and then obtain a threshold range of the report data through threshold detection, and mark each report data exceeding the threshold range as potential abnormal data; an anomaly detection module, connected to the potential marking module, for using a pre-trained anomaly detection model to perform anomaly scoring on each of the potential anomaly data in the report data to obtain a corresponding anomaly scoring result, and then marking the potential anomaly data whose anomaly scoring result exceeds a set scoring range as a burr data point; A data recovery module is connected to the anomaly detection module and is used to delete each of the burr data points in the report data, and then use the linear relationship of adjacent data points of the deleted burr data points to estimate the missing data points for linear interpolation to recover the report data.

8. The abnormal data detection and recovery system according to claim 7, characterized in that: The report data includes multiple data dimensions, and the potential marking module includes: A range calculation unit, used to calculate the mean and standard deviation of each data dimension, and then obtain the threshold range according to the mean and the standard deviation; A potential marking unit is connected to the range calculation module and is used to mark the data points in the report data that exceed the threshold range as the potential abnormal data.

9. The abnormal data detection and recovery system according to claim 7, further comprising a model training module, connected to the abnormal data detection module, comprising: A data set acquisition unit, used to acquire historical time series data, and convert the historical time series data into an isolation forest algorithm format to obtain a feature data set; an initial detection unit, connected to the data set acquisition unit, for performing feature statistics on the feature data set to obtain statistical features, and then setting the contamination rate parameter of the isolation forest algorithm as the initial contamination rate parameter to perform anomaly detection on the feature data set to obtain a corresponding preliminary anomaly detection result; A parameter adjustment unit is connected to the initial detection unit and is used to adjust the contamination rate parameter of the isolation forest algorithm according to the preliminary anomaly detection result and the statistical feature to obtain the anomaly detection model.

10. The abnormal data detection and recovery system according to claim 7 further includes a data preprocessing module, connected to the potential marking module, for performing data cleaning on the report data, then filling missing values ​​on the cleaned report data, and finally smoothing the filled report data to obtain the report data for threshold detection.

Images