Test data outlier adaptive elimination and credibility evaluation method

Abstract

The application provides a test data outlier adaptive elimination and credibility evaluation method, and belongs to the technical field of test data processing. The application solves the problems of poor adaptability and lack of credibility quantitative evaluation of traditional outlier detection methods. The technical scheme of the method comprises the following points: adopting a sliding window dynamic threshold to preliminarily screen outliers, using a density clustering algorithm to accurately judge the preliminary screening result and eliminate outliers, combining K nearest neighbor interpolation to repair missing data, extracting multi-dimensional statistical features and equipment reliability indexes, and evaluating data credibility through a machine learning model, and finally adjusting detection parameters according to the credibility score feedback. The application is suitable for data quality optimization in the fields of scientific experiments, industrial measurement and control, and environmental monitoring, and can effectively improve data reliability and decision accuracy.

Description

Technical Field

[0001] This invention relates to the field of experimental data processing technology, and more specifically, to an adaptive outlier removal and credibility assessment method for experimental data that integrates dynamic threshold detection, density clustering analysis, and machine learning evaluation. Background Technology

[0002] In scientific experiments and industrial measurement and control activities, the collected experimental data often contains outliers due to sensor malfunctions, environmental noise, transmission errors, or human error. These outliers can severely interfere with subsequent data analysis and decision-making. Existing data processing technologies mainly suffer from the following shortcomings: First, outlier detection methods have significant limitations. Traditional statistical methods, such as the 3σ principle and box plots, heavily rely on the assumption that the data follows a normal distribution. They perform poorly on non-normally distributed or high-dimensional data, easily leading to false positives or false negatives. Furthermore, detection methods based on fixed thresholds cannot adapt to dynamic fluctuations in data during experiments, making it difficult to guarantee detection accuracy.

[0003] Second, there is a lack of quantitative assessment of data reliability. Most existing technologies focus on identifying and removing outliers, but lack a scientific and quantifiable reliability assessment metric for evaluating the overall quality of the processed data. Users find it difficult to determine whether the cleaned data meets the requirements for subsequent high-precision analysis.

[0004] Third, the ability to process high-dimensional data is generally insufficient. With the popularization of multi-sensor fusion technology, the dimensionality of experimental data has increased dramatically. Traditional methods face the "curse of dimensionality" when processing high-dimensional feature spaces, making it difficult to effectively distinguish between outliers and normal points.

[0005] Therefore, there is an urgent need for a method that can adaptively identify various outliers and quantitatively evaluate the reliability of processed data in order to effectively improve the overall quality of experimental data and the reliability of decision-making. Summary of the Invention

[0006] (a) Technical problems to be solved This invention aims to address the technical problems in existing experimental data processing methods, such as outlier detection relying on data distribution assumptions, poor adaptability of fixed thresholds, lack of data reliability quantification and evaluation methods, and insufficient high-dimensional data processing capabilities.

[0007] (II) Technical Solution To address the aforementioned issues, this invention proposes an adaptive outlier removal and credibility assessment method for experimental data. The core concept of this invention lies in combining the rapid initial screening capability of a sliding window dynamic threshold with the precise discrimination capability of density clustering. After removing outliers, neighborhood interpolation is used to repair the data. Then, a multi-dimensional feature machine learning model is used to quantify and score the overall credibility of the data, and this score is used to guide the closed-loop adaptive adjustment of the front-end detection parameters.

[0008] This method is as follows Figure 1 As shown, the specific steps include: Step 1: Data Preprocessing. First, the raw experimental data is cleaned, and completely duplicated records are removed. For missing values, preliminary imputation is performed using linear interpolation or K-nearest neighbor interpolation. Then, the data is standardized, for example, using Z-score standardization (formula: ...). Alternatively, normalization methods can be used to eliminate the influence of different units of measurement and form a standardized dataset.

[0009] Step 2: Adaptive Outlier Detection. This step consists of two levels: First, a dynamic threshold screening is performed. A sliding window technique is used, where the window size w can be preset or adaptively adjusted according to data characteristics. The mean of the data within the current window is then calculated. and standard deviation According to the formula Determine data points Whether a point is an outlier candidate is determined by the adaptive adjustment coefficient k, which is dynamically calculated based on the real-time noise level of the data. The coefficient k ranges from 1.5 to 3, increasing as data fluctuations increase. All labeled candidate points constitute the outlier candidate set. .

[0010] Then, density clustering is used for fine-grained evaluation. The DBSCAN algorithm is applied to the candidate set, with a neighborhood radius (e.g., 0.8) and a minimum number of samples (MinPts, e.g., 3). The algorithm calculates the density of each point within its neighborhood, and ultimately identifies isolated points with densities below a preset threshold as outliers, removing these outliers from the original dataset. The neighborhood radius and minimum number of samples in the DBSCAN algorithm are preset or adaptively adjusted based on the distribution density and noise level of the dataset.

[0011] Step 3: Outlier Removal and Data Repair. After removing the final outliers identified in Step 2 from the dataset, new missing data points will be generated. For each missing point, the K-nearest neighbor algorithm is used to select its K nearest normal data points in the complete data space. The repair value is calculated using a weighted average or regression method, thus obtaining a complete and cleaned dataset.

[0012] Step 4: Credibility Assessment Model Construction and Application. Extract multidimensional features from the repaired complete dataset, specifically including: statistical features such as data variance and coefficient of variation. Outlier percentage Equipment reliability characteristics, such as historical sensor failure rates and calibration records, and time characteristics, such as sampling interval consistency and long-term trend stability, are input into a pre-trained machine learning model, such as Gradient Boosting Tree (GBDT), neural network, or support vector machine. The model outputs a confidence score S between 0 and 1, with S closer to 1 indicating higher data confidence.

[0013] Step 5: Feedback Optimization Mechanism. The credibility score S obtained in the previous step is fed back to the detection stage in Step 2, forming a closed loop. Specifically: If the score S is below 0.6, it indicates that the overall credibility of the data is low, and there may be a large number of missed detections. In this case, the system automatically increases the sliding window w and widens the neighborhood radius of DBSCAN clustering or reduces the minimum number of samples to reduce detection sensitivity and reduce false positives. If the score S is above 0.9, it indicates that the data quality is high. The system can narrow the window w and tighten the clustering parameters to improve detection sensitivity and capture more subtle abnormal fluctuations.

[0014] (III) Beneficial Effects Compared with the prior art, the present invention has the following beneficial effects: 1. This invention achieves accurate identification of outliers through a two-level detection architecture of sliding window dynamic threshold and density clustering DBSCAN. This method does not rely on data distribution assumptions and can maintain good adaptability to non-normal and high-dimensional data, significantly improving detection accuracy and recall.

[0015] 2. This invention introduces a reliable quantitative evaluation based on a multi-dimensional feature machine learning model into the outlier removal process. The output 0-1 continuous score can provide users with clear and scientific data quality criteria, which can strongly support subsequent high-risk decision-making.

[0016] 3. This invention establishes a closed-loop feedback mechanism of "detection-evaluation-optimization", which enables the detection parameters to be automatically adjusted according to the data quality, realizes continuous adaptive optimization of the processing process, and avoids the tedious manual repeated parameter adjustment.

[0017] 4. This invention has strong versatility. Its technical solution can be seamlessly adapted to various data formats such as single variable, multivariate, and time series, and can be widely applied to experimental data quality optimization scenarios in different fields such as scientific experiments, smart factories, and environmental monitoring stations. Attached Figure Description

[0018] Figure 1 This is the overall flowchart of the adaptive outlier removal and credibility assessment method for experimental data of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are for illustrative purposes only and do not constitute any limitation on the scope of protection of this invention.

[0020] It should be noted that the algorithm parameters involved in each step of this invention (such as the number of neighbors K in the K-Nearest Neighbors algorithm, the neighborhood radius and minimum number of samples in DBSCAN clustering, the sliding window size, etc.) are all configurable. Specifically, in the preferred embodiment of this invention, the number of neighbors K in the K-Nearest Neighbors algorithm can be a small odd number such as 3, 5, or 7; the neighborhood radius and minimum number of samples in DBSCAN clustering are preset or adaptively adjusted according to the distribution density of the dataset; and the initial size of the sliding window is set according to the data sampling frequency. Those skilled in the art will understand that the specific values ​​of the above parameters can be flexibly selected according to the characteristics of the actual experimental data without affecting the implementation of the core method of this invention.

[0021] Example 1: Industrial Sensor Data Processing This embodiment uses the detection of abnormal data from a temperature sensor array in a smart factory as an example. The process is as follows: Figure 1 As shown.

[0022] First, data preprocessing was performed. The system acquired 1000 consecutive temperature data points, which were then cleaned to identify and remove duplicate records. For the five missing values, linear interpolation was used for initial imputation. Subsequently, Z-score standardization was applied to all data, transforming it into a standardized sequence with a mean of 0 and a standard deviation of 1.

[0023] Entering the adaptive anomaly detection phase. The system uses a sliding window with an initial size of 50, calculating the mean and standard deviation of the data within the window in real time. Based on the formula... An initial screening was conducted, with the k-value initially set to 2.5. This initial screening identified 20 candidate anomalous points. Next, the DBSCAN algorithm was applied to these 20 candidate points, with a neighborhood radius of 0.8 and a minimum sample size of 3. Clustering results showed that 12 points were clustered together and were classified as normal fluctuations; the remaining 8 isolated points were ultimately identified as outliers, which were verified to be caused by temporary sensor malfunctions.

[0024] Next came data repair and evaluation. After removing the eight outliers from the original dataset, data gaps were created. The system used the K-nearest neighbor interpolation method, setting K=5, and selected the five nearest normal data points before and after each missing point for weighted averaging, successfully repairing the data. After repair, the system extracted features such as variance, coefficient of variation, outlier percentage (0.8%), and historical failure rate of the batch of sensors (1%) from the repaired dataset. These features were input into a pre-trained gradient boosting tree (GBDT) model, and the model output a confidence score of S=0.92.

[0025] Finally, the feedback optimization mechanism took effect. Since S=0.92>0.9, the system determined that the current data quality was high and the detection parameters could be further optimized. Therefore, it automatically reduced the sliding window size for subsequent data processing from 50 to 30 and appropriately tightened the neighborhood radius of DBSCAN, improving the real-time performance of subsequent data stream processing and its sensitivity to subtle anomalies.

[0026] Example 2: Multi-parameter data processing for environmental monitoring This embodiment takes the processing of monitoring data on atmospheric pollutant concentrations (PM2.5, SO2, NO2) as an example.

[0027] During the data preprocessing stage, the system acquired 300 sets of multidimensional data containing three parameters. After cleaning and removing data with malformed formats, all parameter values ​​were linearly scaled to the [0, 1] range using a normalization method to unify the magnitude differences between different pollutant concentrations.

[0028] In the adaptive detection phase, the dynamic threshold calculation is no longer based on a single parameter but extends to a multivariate covariance space to identify anomalous patterns across parameters, such as an unusually negative correlation between PM2.5 and SO2 concentrations. The initially screened suspicious samples are then fed into the DBSCAN algorithm for cluster analysis in a high-dimensional feature space. Ultimately, the algorithm identified three completely isolated sample points, which were verified to be errors in manual recording.

[0029] In the data repair and evaluation phase, the system utilizes data from adjacent time points and the correlation characteristics between parameters to repair outliers. Subsequently, features including parameter covariance and historical data volatility are extracted and input into a trained neural network model. The model outputs a confidence score S=0.85, indicating that the processed data is of good quality and sufficient to support atmospheric trend analysis and early warning decisions.

[0030] This specific embodiment is for illustrative purposes only and is not intended to limit the scope of the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention; therefore, all equivalent technical solutions also fall within the protection scope of this invention.

Claims

1. A method for adaptive outlier removal and reliability assessment of experimental data, characterized in that, Includes the following steps: Data preprocessing: The raw experimental data are cleaned to remove duplicate data, missing values ​​are filled by interpolation, and standardization is performed to eliminate dimensional differences. Dynamic threshold initial screening: The mean and standard deviation of the data within the window are calculated using the sliding window technique, based on the formula... Identify a candidate set of outliers, where For the data points to be detected, and , respectively, are the mean and standard deviation of the data within the sliding window, and k is an adaptive adjustment coefficient between 1.5 and 3; Density clustering precision judgment: The DBSCAN algorithm is applied to the candidate set of outliers. By setting the neighborhood radius and the minimum number of samples, isolated points with a density lower than a preset threshold are determined as the final outliers and removed from the dataset. Data repair: For missing data points resulting from outlier removal, the K nearest neighbor algorithm is used to select the K nearest normal data points for weighted averaging or regression interpolation to form a complete dataset; Credibility assessment: Extract statistical features, equipment reliability indicators and time features of the repaired data, construct and use a trained machine learning model to reason about the complete dataset, and output a credibility score with a value range of [0,1]. Feedback optimization: The credibility score is fed back to the dynamic threshold initial screening and density clustering fine judgment stages, and the sliding window size, cluster neighborhood radius and detection sensitivity parameters are dynamically adjusted according to the score.

2. The method according to claim 1, characterized in that, The standardization process in the data preprocessing adopts the Z-score standardization or normalization method.

3. The method according to claim 1, characterized in that, The adaptive adjustment coefficient k in the dynamic threshold initial screening is dynamically calculated based on the real-time noise level of the data, and the value of k increases accordingly when the data fluctuation amplitude increases.

4. The method according to claim 1, characterized in that, In the density clustering precision judgment step, the neighborhood radius and minimum number of samples of the DBSCAN algorithm are preset or adaptively adjusted according to the distribution density and noise level of the dataset.

5. The method according to claim 1, characterized in that, In the data repair step, the number of neighbors K in the K-nearest neighbor algorithm is 3, 5, or 7.

6. The method according to claim 1, characterized in that, The statistical features extracted in the credibility assessment step include data variance, coefficient of variation, and outlier percentage; the equipment reliability indicators include sensor calibration records and historical failure rates; and the time features include consistency of data acquisition time intervals and long-term trend stability.

7. The method according to claim 1, characterized in that, The machine learning model used in the credibility assessment step is a gradient boosting tree, a neural network, or a support vector machine.

8. The method according to claim 1, characterized in that, The feedback optimization steps are as follows: when the confidence score is below 0.6, increase the sliding window width and widen the neighborhood radius of density clustering or reduce the minimum number of samples to reduce false positives; when the confidence score is above 0.9, narrow the sliding window width and shrink the neighborhood radius of density clustering or increase the minimum number of samples to identify subtle anomalies.

9. The method according to claim 1, characterized in that, The initial size of the sliding window is preset based on the data type and sampling frequency, and is dynamically adjusted during the feedback optimization process.

10. The method according to claim 1, characterized in that, The method is applicable to outlier handling and reliability assessment of univariate data, multivariate data, and time series data.

Images