A data governance method for automatically detecting data outliers based on deep learning

Abstract

The application relates to the technical field of data processing, in particular to a data management method for automatically detecting data abnormal values based on deep learning, which comprises the following steps: based on an input data set, a plurality of data sources are synchronized, missing values are filled, and data is standardized and normalized to obtain a pretreated data set.In the application, by executing data preprocessing, the consistency and quality of data input are optimized, the recognition ability of a model to abnormal data is enhanced by extracting features and layer-by-layer training of the model, the precision and efficiency of data abnormality detection are improved, the accuracy of abnormal data scoring is improved by combining residual error and reconstruction error calculation, the false alarm rate is reduced by dynamically adjusting the detection threshold according to data characteristics, the abnormal development trend is evaluated by combining time series analysis, and the reliability and stability of the data processing procedure are ensured by comparing and analyzing abnormal data and normal data modes and correcting data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data governance method based on deep learning for automatically detecting outliers in data. Background Technology

[0002] The field of data processing technology focuses on the systematic transformation, analysis, and interpretation of collected raw data through various computer science and information technology methods. It processes batches of structured and unstructured data through multiple steps, including data collection, verification, storage, cleaning, transformation, analysis, and visualization. By combining machine learning and artificial intelligence, it improves data processing efficiency and accuracy, enabling support for decision-making, strategic planning, and operational optimization. It solves problems related to batch data storage, high-speed data transmission, and real-time data analysis, and has applications in multiple industries such as financial services, healthcare, e-commerce, and government management.

[0003] Among them, the data governance method of automatically detecting outliers aims to automatically detect outliers in datasets and perform data governance. Through automated processing, it improves data quality, optimizes data accuracy and reliability, and supports data analysis and decision-making, thereby improving the efficiency and accuracy of data quality management. By using deep learning models, it learns the behavior and patterns of normal data and automatically detects data points that deviate from standard patterns. Through automated anomaly detection, it reduces the need for manual intervention, enhances the speed and reliability of data processing, and is applied in multiple areas such as financial risk control, cybersecurity, and market analysis, helping enterprises and organizations ensure the integrity and availability of data.

[0004] Traditional data governance technologies that automatically detect outliers cannot effectively synchronize and unify data formats and standards when processing batches of data from multiple sources. This leads to inconsistencies in data processing and the accumulation of errors. In terms of anomaly detection, they rely on static rules or simple statistical models, which limits their effectiveness when processing high-dimensional and complex relational data, resulting in high false positive or false negative rates. In dynamic data environments, this fixed threshold method is difficult to adapt to the natural fluctuations of the data itself over time, leading to decision-making delays or errors. In terms of predicting the development trend of data anomalies, there is a lack of effective predictive tools, making it impossible to identify potential risks in advance and affecting the timeliness and accuracy of enterprise strategy adjustments. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a data governance method based on deep learning for automatically detecting outliers in data.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a data governance method for automatically detecting outliers based on deep learning, comprising the following steps:

[0007] S1: Based on the input dataset, synchronize multiple data sources, fill in missing values, and standardize and normalize the data to obtain a preprocessed dataset;

[0008] S2: Based on the preprocessed dataset, feature extraction is performed on the data. By adjusting the parameter settings of the input layer, hidden layer, and output layer, the model is trained to construct an abnormal data learning model.

[0009] S3: Based on the aforementioned abnormal data learning model, perform anomaly detection on the dataset, analyze the residuals and reconstruction errors of data points in the model output space, calculate the anomaly score of the data points, and generate anomaly score results by combining probability density estimation and neighborhood similarity analysis.

[0010] S4: Based on the anomaly scoring results, adjust the data detection threshold according to the data type, analyze the input data, identify data points that exceed the threshold and record them as anomalies, and obtain data detection records;

[0011] S5: Based on the data detection records, the abnormal data are clustered to identify the abnormal type, and the development trend of the data abnormality is evaluated by combining time series analysis to obtain trend evaluation information;

[0012] S6: Based on the trend assessment information, the cause of the error is identified by comparing the patterns of abnormal and normal data, and the data is corrected in combination with the anomaly type to generate data processing results.

[0013] As a further aspect of the present invention, the preprocessed dataset includes missing value imputation records, scale parameter standardization results, and normalized range parameters; the anomaly learning model includes weight parameter optimization records, bias parameter adjustment results, and accuracy indicators; the anomaly scoring results include calculated residual values, estimated reconstruction error data, and data anomaly scores; the data detection records include labeled anomaly data points, adjusted detection thresholds, and identified data types; the trend assessment information includes classified anomaly types, calculated cluster centers, and development trend assessment results; and the data processing results include data entity relationship correction records, logical relationship matching results, and data integrity optimization results.

[0014] As a further aspect of the present invention, based on the input dataset, multiple data sources are synchronized, missing values ​​are filled, and the data is standardized and normalized to obtain a preprocessed dataset. The specific steps are as follows:

[0015] S101: By inputting a dataset, and through connecting to API interfaces, databases, and file reads, the data is synchronized to the data warehouse in real time, the integrity of the data is analyzed, and the data synchronization results are obtained;

[0016] S102: Based on the data synchronization results, identify missing items and fill in the data, identify outlier data points and clean the data, and generate missing value processing data.

[0017] S103: Based on the missing value processing data, standardize the numerical features and normalize the features to optimize the consistency of data input and generate a preprocessed dataset.

[0018] As a further aspect of the present invention, based on the preprocessed dataset, feature extraction is performed on the data, and the model is trained by adjusting the parameter settings of the input layer, hidden layer, and output layer. The specific steps for constructing an anomaly data learning model are as follows:

[0019] S201: Based on the preprocessed dataset, feature extraction and transformation are performed on the input data. By calculating the rate of change and frequency distribution of multiple features, the impact on the target variable is evaluated, and feature extraction results are obtained.

[0020] S202: Using the feature extraction results, construct a data analysis framework, transform and abstract the input data by stacking multiple processing layers, complete the model construction, and obtain the network structure design results;

[0021] S203: Based on the network structure design results, through multiple iterations, adjust the connection weights and bias values ​​of multilayer neurons, evaluate the stability and accuracy of the model, optimize the data fit, and output the abnormal data learning model.

[0022] As a further aspect of the present invention, based on the aforementioned abnormal data learning model, the dataset is subjected to anomaly detection, the residuals and reconstruction errors of data points in the model output space are analyzed, anomaly scores of data points are calculated, and anomaly score results are generated by combining probability density estimation and neighborhood similarity analysis. The specific steps are as follows:

[0023] S301: Based on the abnormal data learning model, the model input data is processed by calculating the difference between the actual output and the expected output of multiple data points in the model output layer, which is recorded as residuals to obtain the data residual calculation results.

[0024] S302: Based on the data residual calculation results, evaluate the statistical position of multiple data points in the dataset, use the residual values ​​to calculate the probability density, identify abnormal data points, and obtain the probability density estimation results;

[0025] S303: Based on the probability density estimation results and combined with the neighborhood similarity between data points, the anomaly of the data points is quantitatively evaluated, and an anomaly score result is generated.

[0026] As a further aspect of the present invention, the specific formula for calculating the probability density is as follows:

[0027]

[0028] Where p(d(x)) represents the probability density estimate at the residual value d(x) obtained after kernel density estimation, and d(x) represents the residual value of the currently evaluated data point, used to determine the degree of anomaly of the data point. i The remainder of point d(x) represents the residual value of other data points in the dataset, used to compare with d(x) to estimate the density of d(x). n represents the total number of data points in the dataset, i is the summation index, indicating the i-th data point currently being processed, and K is the kernel function used to calculate the relative density of point d(x) with respect to point d(x). i The relative density contribution of ) The standardized distance, which is the difference between the residuals of the current data point and the residuals of other points divided by the bandwidth h, is the input to the kernel function K. h is the bandwidth parameter of the kernel function, controlling the smoothness of the density estimation. i It is for each data point d(x) i The weight of a data point is adjusted based on its characteristics, such as frequency, importance, or age.

[0029] As a further aspect of the present invention, based on the anomaly scoring result, the data detection threshold is adjusted according to the data type, the input data is analyzed, and data points exceeding the threshold are identified and recorded as anomalies. The specific steps for obtaining the data detection record are as follows:

[0030] S401: Based on the anomaly scoring results, adjust the threshold range to match the characteristics of the data type, including continuous and discrete types, according to the anomaly scores of multiple data points, and perform threshold sensitivity analysis to obtain a threshold setting adjustment record;

[0031] S402: Adjust the record according to the threshold setting, compare the abnormal scores of multiple data points, identify data points whose scores exceed the threshold and record them as abnormal, and generate abnormal data point marking records;

[0032] S403: Based on the abnormal data point marker record, record the values ​​of multiple abnormal data points, the central location of the dataset, the timestamp, and the abnormal type information to obtain the data detection record.

[0033] As a further aspect of the present invention, based on the data detection records, the specific steps for identifying anomaly types by calculating the clustering of abnormal data, and combining time series analysis to assess the development trend of data anomalies and obtain trend assessment information are as follows:

[0034] S501: Based on the data detection records, perform similarity analysis on abnormal data points, and perform clustering operations according to feature vectors to analyze the group affiliation of data points and obtain abnormal data clustering results;

[0035] S502: Based on the clustering results of the abnormal data, perform time series analysis on multiple clusters, identify and record the time dependence and development trend of multiple clusters, and obtain the abnormal trend analysis results;

[0036] S503: Based on the abnormal trend analysis results, analyze and identify multiple factors affecting the abnormal trend of data, calculate the correlation between multiple factors and abnormal trends in the data set, and generate trend assessment information.

[0037] As a further aspect of the present invention, the specific formula for calculating the correlation between multiple factors and abnormal trends in the dataset is as follows:

[0038]

[0039] Where, x i This refers to the observed value of a single factor at a specific point in time, used to calculate the correlation with outlier data. y represents the average of all factor observations, used to standardize the observations of each factor and reduce the influence of different magnitudes. i Refers to x i The observed values ​​of the outlier data are the target variables for correlation analysis. The average value of all outlier data is used to standardize the observation value of each outlier data point. r is the correlation coefficient, used to quantify the strength and direction of the linear relationship between x and y.

[0040] As a further aspect of the present invention, based on the trend assessment information, the specific steps for identifying the cause of the error by comparing the patterns of abnormal and normal data, and correcting the data according to the anomaly type to generate the data processing result are as follows:

[0041] S601: Based on the trend assessment information, analyze various anomaly types, identify key differences between abnormal data and normal data patterns, analyze the causes of data anomalies, and obtain anomaly cause analysis results;

[0042] S602: Based on the results of the anomaly cause analysis, logical verification is performed on the identified error causes. By checking whether the data items violate known business rules and logical relationships, the data points that need to be adjusted are identified, and the data tags to be corrected are obtained.

[0043] S603: Based on the data markers to be corrected, adjust and correct the data values ​​and relationships of abnormal data points according to business rules and logical requirements, verify the consistency and accuracy of the data, and generate data processing results.

[0044] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0045] In this invention, by performing data preprocessing, the consistency and quality of data input are optimized. By extracting features and training the model layer by layer, the model's ability to identify abnormal data is enhanced, improving the accuracy and efficiency of data anomaly detection. By combining the calculation of residuals and reconstruction errors, the accuracy of scoring abnormal data is improved. By dynamically adjusting the detection threshold using data characteristics, the false alarm rate is reduced. By combining time series analysis, the development trend of anomalies is evaluated. By comparing and analyzing abnormal data and normal data patterns and correcting the data, the reliability and stability of the enhanced data processing flow are ensured. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the main steps of the present invention;

[0047] Figure 2 This is a detailed schematic diagram of S1 of the present invention;

[0048] Figure 3 This is a detailed schematic diagram of S2 of the present invention;

[0049] Figure 4 This is a detailed schematic diagram of S3 of the present invention;

[0050] Figure 5 This is a detailed schematic diagram of S4 of the present invention;

[0051] Figure 6 This is a detailed schematic diagram of S5 of the present invention;

[0052] Figure 7 This is a detailed schematic diagram of S6 of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0054] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0055] Please see Figure 1 This invention provides a technical solution: a data governance method based on deep learning for automatically detecting outliers in data, comprising the following steps:

[0056] S1: Based on the input dataset, synchronize multiple data sources, fill in missing values, and standardize and normalize the data to obtain a preprocessed dataset;

[0057] S2: Based on the preprocessed dataset, feature extraction is performed on the data. By adjusting the parameter settings of the input layer, hidden layer, and output layer, the model is trained to build an abnormal data learning model.

[0058] S3: Based on the abnormal data learning model, perform anomaly detection on the dataset, analyze the residuals and reconstruction errors of data points in the model output space, calculate the anomaly score of the data points, and generate anomaly score results by combining probability density estimation and neighborhood similarity analysis.

[0059] S4: Based on the anomaly scoring results and the data type, adjust the data detection threshold, analyze the input data, identify data points that exceed the threshold and record them as anomalies, and obtain the data detection record;

[0060] S5: Based on data detection records, identify anomaly types by calculating clusters of abnormal data, and combine time series analysis to assess the development trend of data anomalies and obtain trend assessment information;

[0061] S6: Based on trend assessment information, identify the causes of errors by comparing the patterns of abnormal and normal data, and correct the data according to the anomaly type to generate data processing results.

[0062] The preprocessed dataset includes records of missing value imputation, scale parameter standardization results, and normalized range parameters. The anomaly learning model includes records of weight parameter optimization, bias parameter adjustment results, and accuracy metrics. The anomaly scoring results include calculated residual values, estimated reconstruction error data, and data anomaly scores. The data detection records include labeled anomaly data points, adjusted detection thresholds, and identified data types. The trend assessment information includes classified anomaly types, calculated cluster centers, and development trend assessment results. The data processing results include records of data entity relationship correction, logical relationship matching results, and data integrity optimization results.

[0063] Please see Figure 2 Based on the input dataset, multiple data sources are synchronized, missing values ​​are filled, and the data is standardized and normalized to obtain the preprocessed dataset. The specific steps are as follows:

[0064] S101: By inputting a dataset, and through connecting to API interfaces, databases, and file reads, the data is synchronized to the data warehouse in real time, the integrity of the data is analyzed, and the data synchronization results are obtained;

[0065] In the above content, by connecting to the API interface, database, and file reading, according to the formula D =

[0066] N / T, calculates the data synchronization speed;

[0067] In the formula, D represents the data synchronization speed, N represents the number of data records to be synchronized, and T represents the synchronization time.

[0068] Detailed explanation of the formula and its calculation derivation:

[0069] Assuming 500 data entries were synchronized in 50 seconds, calculate D:

[0070] D = 500 / 50 = 10 messages / second

[0071] The result of 10 data points per second indicates high data synchronization efficiency. This result is used to evaluate data synchronization performance by synchronizing the data.

[0072] S102: Based on the data synchronization results, identify missing items and fill in the data, identify outlier data points and clean the data, and generate missing value processing data.

[0073] In sub-step S102, based on the data synchronization results, data integrity analysis is performed. The Pandas library in Python is used to quantitatively analyze the missing data ratio in the dataset, identifying fields with high missing rates, such as the common missing current and voltage readings in power company monitoring data. Next, an appropriate imputation method is selected based on the data type and distribution characteristics. Mean imputation is used for numerical fields, while K-nearest neighbor imputation is used for more sensitive or volatile data, such as voltage peak values. The average of the nearest neighbors to the missing data point is calculated for imputation. For outlier data points, the Z-score is calculated for each data point. Any data point exceeding ±3 standard deviations is marked as an outlier. Quartile ranges are used for validation and extreme value removal. Outlier data points are assessed for their impact on the model and analysis before a decision is made whether to correct or delete them. After data processing, the procedure is modified to verify accuracy and consistency through cross-validation and regression testing, ensuring that data repair operations do not introduce new biases. The updated dataset is uploaded back to the data warehouse to support further analysis and decision-making.

[0074] S103: Based on missing value processing, standardize numerical features and normalize features to optimize the consistency of data input and generate a preprocessed dataset.

[0075] In sub-step S103, data is processed based on missing values. Numerical features are standardized and normalized to optimize data input consistency. Numerical feature standardization uses the Z-score standardization method, converting the data into a distribution with a mean of 0 and a standard deviation of 1. In the power industry, standardization and normalization effectively monitor and analyze grid load anomalies, optimize energy allocation, ensure the stability and efficiency of power supply, and help power companies identify potential equipment failures and consumption anomalies in advance, reducing power outages and improving customer satisfaction. Normalization uses MinMaxScaler, which scales each feature to between a given minimum and maximum value, enhancing the model's predictive accuracy and improving the algorithm's convergence speed. The processed data undergoes data quality checks, including consistency checks and outlier re-detection, ensuring high-quality data processing. The resulting preprocessed dataset integrates the advantages of various data processing steps, providing high-quality input data for deep learning or machine learning models and enhancing the efficiency and effectiveness of model training.

[0076] Please see Figure 3 Based on the preprocessed dataset, features are extracted from the data. By adjusting the parameter settings of the input layer, hidden layer, and output layer, the model is trained. The specific steps for building an anomaly data learning model are as follows:

[0077] S201: Based on the preprocessed dataset, feature extraction and transformation are performed on the input data. By calculating the rate of change and frequency distribution of multiple features, the impact on the target variable is evaluated, and the feature extraction results are obtained.

[0078] In sub-step S201, features in the preprocessed dataset are extracted and transformed using various algorithms. Time series analysis is employed to calculate the rate of change of features. In the power industry, the rate of change and frequency distribution analysis of features can be used to monitor the cyclical changes in power demand and supply, and to promptly identify abnormal patterns that may lead to power supply instability. This is of great significance for ensuring the reliability of the power grid and optimizing load allocation. Frequency distribution analysis is achieved through Fourier transform, identifying and quantifying the regularity and anomalies of feature changes over time. The rate of change of features provides sensitivity information on the changes of data points relative to time, while frequency distribution reveals the periodicity and abnormal frequency components of the data. For example, in financial data analysis, the rate of change is used to reveal the volatility of asset prices, and frequency distribution can indicate the periodic peaks of trading activities. By evaluating multiple features in the dataset, the most significant positive features that have the greatest impact on the target variable are determined. The resulting feature extraction result is a comprehensive feature score, summarizing the influence of each feature on the target.

[0079] S202: Utilize the feature extraction results to construct a data analysis framework. By stacking multiple processing layers, the input data is transformed and abstracted to complete the model construction and obtain the network structure design results.

[0080] In the above content, multiple processing layers are stacked, according to formula W new =W old +ηΔW, calculate the weight adjustment;

[0081] In the formula, W new W represents the new weight value. old η represents the old weight values, η represents the learning rate, and ΔW represents the gradient change of the weights.

[0082] Detailed explanation of the formula and its calculation derivation:

[0083] Assuming the original weight W old Given a learning rate η of 0.01 and a gradient change ΔW of 0.5, calculate W. new :

[0084] W new =0.5 + 0.01 × 0.1 = 0.501

[0085] The result of 0.501 indicates that the weights were slightly adjusted to optimize the network weights. By adjusting the learning rate and gradient, the learning effect and accuracy of the model were improved, and the network structure design result was obtained.

[0086] S203: Based on the network structure design results, through multiple iterations, the connection weights and bias values ​​of multilayer neurons are adjusted to evaluate the stability and accuracy of the model, optimize the data fit, and output the abnormal data learning model.

[0087] In sub-step S203, based on the network structure design results, the gradient descent algorithm is used to optimize the learning efficiency of the model by adjusting the weights and biases of the multi-layer neural network, resulting in an optimized abnormal data learning model. The specific formula for calculating the weight adjustment is as follows:

[0088]

[0089] Among them, W t Represents the weight parameters at training step t, and η represents the learning rate. Represents the weight parameter W t The gradient of the corresponding error function E, W t+1 This represents the weight parameters updated in training step t+1. This formula directly reduces the value of the error function by calculating the weight adjustment amount at each step, thereby improving the model's ability to capture data features and enhancing the model's accuracy in identifying abnormal data.

[0090] formula:

[0091]

[0092] Detailed explanation of the formula and its calculation derivation:

[0093] The formula is used to update the weights in a neural network using the gradient descent method, thereby reducing the error function E and improving the model's ability to identify outlier data.

[0094] Parameter meanings and settings:

[0095] W t The weights for the current training step t are obtained during model initialization and calculated through forward and backward propagation using mini-batch data. η is the learning rate, assumed to be 0.01, adjusted based on model performance in pre-experiments to ensure that the learning progress is neither too fast (leading to oscillations) nor too slow (leading to excessively long convergence times). The loss function E is related to the weights W. t The gradient, obtained through backpropagation of the current batch of data, indicates the rate of change of the error function with respect to the current weights. It guides the direction and magnitude of weight updates. Assuming that in a certain step, the network weights W... t The gradient of the loss function is 1.5. The calculation result is 0.05;

[0096] Substitute the parameters into the formula to calculate:

[0097] W t+1 =1.5 - 0.01 × 0.05 = 1.5 - 0.0005 = 1.4995

[0098] The result of 1.4995 indicates that the weights were slightly adjusted, demonstrating that the current learning rate and gradient adjustment effectively optimized the weights in the direction of reducing error, thus reducing the model's loss and enhancing its learning efficiency and accuracy. This result is crucial for confirming the correct adjustment of the weights during the training process, ensuring that the model can continuously learn and improve its ability to identify outliers.

[0099] Please see Figure 4 Based on an anomaly data learning model, anomaly detection is performed on the dataset. The residuals and reconstruction errors of data points in the model output space are analyzed, and anomaly scores of data points are calculated. Combining probability density estimation and neighborhood similarity analysis, the specific steps to generate anomaly score results are as follows:

[0100] S301: Based on the learning model of abnormal data, the model input data is processed by calculating the difference between the actual output and the expected output of multiple data points in the model output layer, which is recorded as residuals to obtain the data residual calculation results.

[0101] In sub-step S301, the input data is processed using an anomaly data learning model. The difference between the actual output and the expected output at each data point in the model's output layer is calculated, known as the residual. This residual measures the accuracy of the model's prediction for the target data point. In the power industry, by monitoring the residual between the output power of a power plant and the predicted value, it is possible to promptly identify equipment performance degradation or maintenance needs, optimize power generation efficiency and maintenance scheduling, help prevent overload or power loss, and ensure the stable operation of the power grid. The residual calculation involves statistical methods of the sum of squared errors or absolute errors, reflecting the model's performance at each data point. A large residual value indicates that the model's prediction at the data point differs significantly from the actual situation, suggesting that the model has failed to capture the key characteristics of the data or that the data point itself has anomalies. The obtained residual calculation results provide basic data for subsequent steps to analyze and optimize the model's performance.

[0102] S302: Based on the data residual calculation results, evaluate the statistical position of multiple data points in the dataset, use the residual values ​​to calculate the probability density, identify outlier data points, and obtain the probability density estimation results;

[0103] The specific formula for calculating the probability density is as follows:

[0104]

[0105] Where p(d(x)) represents the probability density estimate at the residual value d(x) obtained after kernel density estimation, and d(x) represents the residual value of the currently evaluated data point, used to determine the degree of anomaly of the data point. i The remainder of point d(x) represents the residual value of other data points in the dataset, used to compare with d(x) to estimate the density of d(x). n represents the total number of data points in the dataset, i is the summation index, indicating the i-th data point currently being processed, and K is the kernel function used to calculate the relative density of point d(x) with respect to point d(x). i The relative density contribution of ) The standardized distance, which is the difference between the residuals of the current data point and the residuals of other points divided by the bandwidth h, is the input to the kernel function K. h is the bandwidth parameter of the kernel function, controlling the smoothness of the density estimation. i It is for each data point d(x) i The weight of a data point is adjusted based on its characteristics, such as frequency, importance, or age.

[0106] formula:

[0107]

[0108] Parameter meanings and settings:

[0109] d(x) is the residual value of the current data point, which is the difference between the observed value and the model prediction value. The residual value reflects the degree of deviation of the data point from the model prediction and is used for subsequent anomaly scoring. Assume d(x) = 5.

[0110] d(x i ) represents the residual value of other data points in the dataset. The residual value is obtained by performing the same residual calculation as d(x) on each data point in the dataset. It is used to provide a reference residual value, which is compared with d(x) and kernel density is estimated. Assume that the residual set of data points is [3, 7, 5, 9, 6].

[0111] n is the total number of data points, obtained by counting the number of data points in the statistical data set. It is used to calculate the bandwidth and the sum in the standardized kernel density estimation. It is assumed to be 5.

[0112] h represents the kernel bandwidth, a smoothing parameter in kernel density estimation, calculated using Silverman's rule. It controls the smoothness of the kernel function, affecting the precision and generalization ability of the density estimation. Based on the standard deviation of the residuals, h is calculated as h = 1.06 × std(d) × n. -1 / 5 Assuming the standard deviation std(d) is 2, h = 1.06 × 2 × 5 -1 / 5 ≈1.87;

[0113] K is the Gaussian kernel function, used to calculate the weighted contribution between different residual values ​​and estimate the probability density.

[0114] w i The weights are used to adjust the importance of each data point's contribution when calculating the kernel density estimate. It is assumed that the weights are uniformly distributed, with each w... i =1;

[0115] Given a set of residuals for data points [3, 7, 5, 9, 6], calculate the contribution of each point;

[0116] For d(x) i ) = 3:

[0117]

[0118] For d(x) i ) = 7:

[0119]

[0120] For d(x) i ) = 5:

[0121]

[0122] For d(x) i ) = 9:

[0123]

[0124] For d(x) i ) = 6:

[0125]

[0126] Substitute the parameters into the formula to calculate:

[0127]

[0128]

[0129]

[0130] The calculated result p(d(x))≈0.133 represents the probability density estimate of the current data point d(x)=5. The value reflects the degree of anomalousness of this point relative to other points in the dataset. A lower density value indicates that the point is a potential outlier. Density values ​​are used to quickly identify data points that need attention in a large amount of data.

[0131] S303: Based on the probability density estimation results and combined with the neighborhood similarity between data points, the anomaly of the data points is quantitatively evaluated, and anomaly score results are generated;

[0132] In substep S303, based on the probability density estimation results, a quantitative evaluation is performed using the neighborhood similarity between data points. The anomaly score for each data point is then calculated. The Local Anomaly Factor (LOF) algorithm is used to perform anomaly detection, identifying local anomalies that are significantly different from other data points in their neighborhood. This is particularly effective for complex datasets containing clusters. Initially, the synthetic dataset containing both normal and anomalous data is standardized. Then, key parameters of the LOF algorithm are set, including the neighborhood size. This parameter determines the number of neighboring points considered when calculating the local density and directly affects the algorithm's sensitivity. For each data point, its local reachability density and local density are calculated based on its distance from its neighbors. Anomaly factors are used, where points with low local reachability density are considered potential anomalies. The algorithm generates a LOF score for each data point, indicating its relative degree to being an outlier. Points with scores higher than a set threshold are marked as anomalies. Finally, normal and anomalous data points are visualized through a scatter plot, intuitively showing the detection effect of the LOF algorithm. Based on the results, response measures are recommended, including system configuration adjustments or enhanced security protection, to support data analysis and decision-making. By identifying and isolating potential anomalous data points, the quality of data processing and the stability of the model are improved. The generated anomaly score results are of great significance for subsequent data monitoring and the formulation of anomaly handling strategies.

[0133] Please see Figure 5 Based on the anomaly scoring results and the data type, the data detection threshold is adjusted. The input data is analyzed to identify data points exceeding the threshold and record them as anomalies. The specific steps for obtaining the data detection records are as follows:

[0134] S401: Based on the anomaly scoring results, adjust the threshold range to match the characteristics of the data type, including continuous and discrete types, according to the anomaly scores of multiple data points, and perform threshold sensitivity analysis to obtain the threshold setting adjustment record;

[0135] In sub-step S401, based on the anomaly scoring results, the threshold range is adjusted by analyzing the characteristics of data types, including the differentiated processing of continuous and discrete data. Threshold settings for continuous data consider the data distribution characteristics, including the mean and standard deviation, while discrete data depends on the frequency or probability of occurrence of a category. In the power industry, different thresholds are set to monitor abnormal fluctuations in grid load and electricity consumption, promptly identify abnormal consumption caused by demand peaks or equipment failures, optimize energy allocation, and prevent grid failures. Threshold sensitivity analysis uses sensitivity analysis and ROC curves to evaluate the impact of different threshold settings on model performance, determining the optimal threshold to maximize the accuracy of anomaly detection and reduce false alarm rates. The adjustment record covers the results of each threshold test and the corresponding model performance indicators, providing data support and decision-making basis for subsequent threshold optimization, ensuring that threshold settings match data characteristics and detection requirements.

[0136] S402: Adjust the records according to the threshold setting, compare the abnormal scores of multiple data points, identify data points whose scores exceed the threshold and record them as abnormal, and generate abnormal data point marking records;

[0137] The above content involves adjusting records based on threshold settings, comparing anomaly scores for multiple data points, and applying the formula. Calculate the total difference in scores exceeding the threshold;

[0138] In the formula, S represents the total difference, x i represents the anomaly score of the i-th data point, T represents the threshold, and n represents the total number of data points;

[0139] Detailed explanation of the formula and its calculation derivation:

[0140] Assuming there are 5 data points with anomaly scores of 72, 78, 65, 80, and 67, and a threshold T of 75, calculate S:

[0141] S = (72-75) 2 +(78-75) 2 +(65-75) 2 +(80-75) 2 +(67-75) 2

[0142] S = 9 + 9 + 100 + 25 + 64 = 207

[0143] Result 207 shows that the total difference in scores exceeding the threshold is used by a formula to evaluate the total deviation between abnormal scores and the set threshold, identify and record data points exceeding the threshold as abnormal, and generate a marked record of abnormal data points.

[0144] S403: Based on the abnormal data point marking record, record the values ​​of multiple abnormal data points, the central location of the dataset, the timestamp, and the abnormality type information to obtain the data detection record;

[0145] In sub-step S403, based on the anomaly data point marking records, the value, central location, timestamp, and anomaly type information of each anomaly data point are recorded. Data tracking and recording technologies, including database logs and time series analysis tools, are used to ensure that the information of each anomaly point is accurately recorded and stored. Anomaly type information includes data noise, deviation anomalies, or potential data tampering. Target information is used for subsequent data repair, model adjustment, and risk assessment. In the power industry, the marking and management of anomaly data points is crucial for preventing system failures and optimizing energy distribution. By monitoring grid operation data in real time, anomalies such as overloads or voltage instability can be identified and handled promptly, ensuring the stability and efficiency of the power supply system. Data points marked as anomalies are then addressed. Appropriate governance measures are taken. For anomalous data with low importance, data removal algorithms are applied, such as using the Z-Score method to automatically identify and remove data points that are statistically significantly outside the normal range. After removing anomalous data points from the dataset, the statistical summary of the dataset is recalculated to verify data consistency. For data of general importance, data validation and correction tools are used to automatically correct data format errors and range errors. After each correction, logs are recorded and the data model is reloaded to check the correction effect. For critical data, cause analysis and impact scope investigation are conducted. Decision tree analysis is used to identify the root cause of the anomaly. During the investigation, the input path and historical records of relevant data are collected to confirm the starting point of the data anomaly and the business processes affected.

[0146] Please see Figure 6 Based on data detection records, the specific steps for identifying anomaly types by calculating clusters of abnormal data and combining time series analysis to assess the development trend of data anomalies and obtain trend assessment information are as follows:

[0147] S501: Based on data detection records, perform similarity analysis on abnormal data points, and perform clustering operations based on feature vectors to analyze the group affiliation of data points and obtain abnormal data clustering results;

[0148] In sub-step S501, similarity analysis is performed on abnormal data points, and clustering is carried out based on feature vectors. The K-means clustering algorithm is used to divide the data into several groups according to the feature vectors of the data points. The distance between the data points and the center point in each group is minimized. During the clustering process, K data points are randomly selected as the initial group centers, and each data point is assigned to the nearest group center. Then, the center position of each group is updated, and the process is repeated until the group centers no longer change. Abnormal data points with similar characteristics are grouped together, and the common features and behaviors of the data points in each group are analyzed. This is particularly important in the power industry. In power grid monitoring, clustering analysis can effectively distinguish different types of abnormal power consumption, such as peak abnormalities or continuous low-efficiency operation, which helps to optimize energy allocation and prevent power waste. The obtained abnormal data clustering results are used to identify potential patterns and clusters of abnormal behaviors in the data.

[0149] S502: Based on the clustering results of abnormal data, perform time series analysis on multiple clusters, identify and record the time dependence and development trend of multiple clusters, and obtain the abnormal trend analysis results;

[0150] In sub-step S502, based on the clustering results of abnormal data, an autoregressive moving average model is used to analyze the time dependence and development trend of each cluster. By combining autoregressive and moving average models, the time series characteristics of the data are matched. By calculating the linear combination of data at each time point with the data at the previous few time points and the weighted sum of random errors, the dependence and inherent trend of the time series are revealed. The process involves selecting the number of matching lag terms, estimating model parameters, and performing model fitting and validation. In the power industry, time series analysis is of great value for monitoring and predicting fluctuations in power demand, changes in power generation efficiency, or abnormal grid load. By analyzing the time series patterns of historical consumption data, future peak power demand can be predicted, and grid operation and energy allocation strategies can be optimized. The results of the analysis show the changing patterns and trends of different clusters in the time dimension. The obtained abnormal trend analysis results provide a scientific basis for understanding the behavior of each cluster over time and support the prediction and decision-making of future data.

[0151] S503: Based on the results of abnormal trend analysis, analyze and identify multiple factors that affect the abnormal trend of data, calculate the correlation between multiple factors and abnormal trends in the dataset, and generate trend assessment information;

[0152] The specific formula for calculating the correlation between multiple factors and outlier trends in a dataset is as follows:

[0153]

[0154] Where, x iThis refers to the observed value of a single factor at a specific point in time, used to calculate the correlation with outlier data. y represents the average of all factor observations, used to standardize the observations of each factor and reduce the influence of different magnitudes. i Refers to x i The observed values ​​of the outlier data are the target variables for correlation analysis. The average value of all outlier data is used to standardize the observation value of each outlier data point. r is the correlation coefficient, used to quantify the strength and direction of the linear relationship between x and y.

[0155] formula:

[0156]

[0157] Detailed explanation of the formula and its calculation derivation:

[0158] The formula is used to calculate the correlation between multiple clusters and time in the clustering results of outlier data, and the correlation coefficient r is used to measure the degree of linear correlation between the clustering index and time in the time series.

[0159] Parameter meanings and settings:

[0160] x i The clustering index represents the i-th time point, which is the dependent variable considered in time series analysis, such as the anomalous strength of a specific cluster at different time points;

[0161] y i For the corresponding x i The point in time, i.e. the independent variable, is usually a continuous time series (such as the number of days, the number of hours, etc.);

[0162] For all x i The average value represents the average level of the clustering index during the analysis period;

[0163] For all y i The average value represents the average time point during the analysis period;

[0164] n represents the sample size, the total number of days or time points included in the analysis;

[0165] Substitute the parameters into the formula to calculate:

[0166] Assume the clustering index x = [50, 55, 53, 60, 58] at 5 time points, and the time series y = [1, 2, 3, 4, 5];

[0167] Calculate the average value and

[0168]

[0169]

[0170] Calculate r:

[0171]

[0172]

[0173]

[0174] The result r≈0.008 indicates that the clustering index has a low correlation with time in this target dataset, suggesting that the clustering index hardly changes over time within the selected time period.

[0175] Please see Figure 7 Based on trend assessment information, the specific steps for identifying errors by comparing patterns in abnormal and normal data, and then correcting the data according to the anomaly type to generate data processing results are as follows:

[0176] S601: Based on trend assessment information, analyze various anomaly types, identify key differences between abnormal and normal data patterns, analyze the causes of data anomalies, and obtain anomaly cause analysis results;

[0177] The above content analyzes various anomaly types and follows the formula. Calculate the percentage of different exception types;

[0178] In the formula, R represents the percentage of exception types, and n e n represents the number of data points representing the target anomaly type. t Represents the total number of data points;

[0179] Detailed explanation of the formula and its calculation derivation:

[0180] Assume a total of n data points t The number of data points n for a specific anomaly type is 1000. e Given 150, calculate R:

[0181]

[0182] The result of 15% indicates the proportion of the target anomaly type in the total data. The formula is used to identify and analyze abnormal patterns in the data and key differences from normal data patterns, helping to understand the probability and frequency of abnormal data occurrence and obtain the results of anomaly cause analysis.

[0183] S602: Based on the results of the anomaly cause analysis, perform logical verification on the identified error causes. By checking whether the data items violate known business rules and logical relationships, identify the data points that need to be adjusted and obtain the data tags to be corrected.

[0184] In sub-step S602, based on the anomaly cause analysis results, assertion detection and a rule engine are used to verify the identified error causes. Assertion detection checks whether data items violate business rules by setting assertions during data processing, including numerical range, data type, and logical consistency. The rule engine automatically checks the logical relationships of data using preset business rules to ensure that data points meet business logic and data integrity requirements. In the power industry, this is crucial for ensuring the accuracy and consistency of power consumption data. By automatically detecting data points that are significantly inconsistent with historical consumption patterns through the rule engine, metering errors and data tampering are identified, and timely measures are taken to avoid financial losses and power supply problems. Data points that violate logic are marked, and the resulting data tags to be corrected include a problem description for each data point and recommended corrective measures. This information is crucial for data quality management and maintenance, helping the data management team to quickly locate and handle data anomalies.

[0185] S603: Based on the data to be corrected, adjust and correct the data values ​​and relationships of abnormal data points according to business rules and logical requirements, verify the consistency and accuracy of the data, and generate data processing results;

[0186] In sub-step S603, based on business rules and logical requirements, necessary adjustments and corrections are made to the abnormal data points in the data to be corrected. Data cleaning techniques and constraint satisfaction programming are used to ensure the correctness of data values ​​and relationships. Data cleaning includes data value replacement, format adjustment, and relationship reconstruction. Constraint satisfaction programming restores data consistency by setting and resolving data constraint issues. After the adjustments are completed, data consistency and accuracy are verified by running data quality checks and using data integrity tests again to ensure that the corrected data points conform to business logic and rules. In the power industry, ensuring the accuracy and consistency of data is crucial for monitoring and managing the efficiency of the power grid. By correcting power usage data, power companies can accurately predict power demand and adjust power supply plans, reduce power waste, and optimize resource allocation. The generated data processing results mark all corrected and verified data points, providing high-quality and reliable data support for business decisions.

[0187] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A data governance method for automatically detecting data outliers based on deep learning, characterized in that, The method comprises the following steps: Based on the input data set, the data is synchronized to the data warehouse in real time by connecting the API interface, database and file reading, the integrity of the data is analyzed, such as the missing current and voltage readings in the power company monitoring data, the missing values are filled, and the data is standardized and normalized to obtain a preprocessed data set; Based on the preprocessed data set, the data is feature extracted, the model is trained by adjusting the parameter settings of the input layer, hidden layer and output layer, and an abnormal data learning model is constructed; Based on the abnormal data learning model, the data set is detected for abnormality, the residual error and reconstruction error of the data points in the model output space are analyzed, the abnormal score of the data points is calculated, and the abnormal score result is generated by combining the probability density estimation and neighborhood similarity analysis; The specific steps of the abnormal data learning model for detecting the abnormality of the data set, analyzing the residual error and reconstruction error of the data points in the model output space, calculating the abnormal score of the data points, combining the probability density estimation and neighborhood similarity analysis, and generating the abnormal score result are as follows: Based on the abnormal data learning model, the model input data is calculated by calculating the difference between the actual output and the expected output of multiple data points in the model output layer, and the residual error is recorded to obtain the data residual error calculation result; Based on the data residual error calculation result, the statistical position of multiple data points in the data set is evaluated, the residual error value is used to calculate the probability density, and the abnormal data points are identified to obtain the probability density estimation result; Based on the probability density estimation result, the abnormality of the data points is quantitatively evaluated by combining the neighborhood similarity between the data points, and the abnormal score result is generated; According to the abnormal score result, according to the data type, the data detection threshold is adjusted, the input data is analyzed, the data points exceeding the threshold are identified and recorded as abnormal, and the data detection record is obtained; Based on the data detection record, the abnormal type is identified by calculating the clustering of abnormal data, the development trend of data anomaly is evaluated by combining time series analysis, and the trend evaluation information is obtained; According to the trend evaluation information, the error reason is identified by comparing the patterns of abnormal data and normal data, the data is corrected according to the abnormal type, and the data processing result is generated.

2. The data governance method of automatically detecting data outliers based on deep learning according to claim 1, wherein, The preprocessed data set includes missing value filling records, scale parameter standardization processing results, and normalized range parameters, the abnormal data learning model includes weight parameter optimization records, bias parameter adjustment results, and accuracy indicators, the abnormal score result includes calculated residual values, estimated reconstruction error data, and data anomaly scores, the data detection record includes marked abnormal data points, adjusted detection thresholds, and identified data types, the trend evaluation information includes classified abnormal types, calculated clustering centers, and development trend evaluation results, and the data processing result includes data entity relationship correction records, logical relationship matching results, and data integrity optimization results. 3.The deep learning based data governance method for automatically detecting data outliers according to claim 1, wherein, Based on the input data set, multiple data sources are synchronized, missing values are filled, and data is standardized and normalized to obtain a preprocessed data set. By inputting the data set, synchronizing the data to the data warehouse in real time through the connection API interface, database and file reading, analyzing the integrity of the data, and obtaining the data synchronization result; Based on the data synchronization result, missing items are identified and data filling is performed, outlier data points are identified and data cleaning is performed, and missing value processing data is generated; Based on the missing value processing data, numerical value type features are standardized and features are normalized to optimize the consistency of data input and generate a preprocessed data set.

4. The data governance method of automatically detecting data outliers based on deep learning according to claim 1, wherein, Based on the preprocessed data set, feature extraction is performed on the data, the parameter settings of the input layer, hidden layer and output layer are adjusted, the model is trained, and the specific steps of constructing the abnormal data learning model are as follows: Based on the preprocessed data set, feature extraction and conversion are performed on the input data, the influence on the target variable is evaluated by calculating the change rate and frequency distribution of multiple features, and the feature extraction result is obtained; Using the feature extraction result, a data analysis framework is constructed, multiple processing layers are stacked to transform and abstract the input data, the model is constructed, and the network structure design result is obtained; Based on the network structure design result, the connection weights and bias values of multiple layers of neurons are adjusted through multiple iterations, the stability and accuracy of the model are evaluated, the fitting degree of the data is optimized, and the abnormal data learning model is output.

5. The data governance method of automatically detecting data outliers based on deep learning according to claim 1, wherein, The specific formula for calculating the probability density is: ； wherein, represents the probability density estimation value at the residual value after kernel density estimation, represents the residual value of the data point currently being evaluated, used to determine the degree of abnormality of the data point, represents the residual values of other data points in the data set, used for comparison with to estimate the density of represents the total number of data points in the data set, is the summation index, representing the data point currently being processed, is the kernel function, used to calculate the relative density contribution of point with respect to point represents the normalized distance, which is the difference between the residual of the current data point and the residual of other points divided by the bandwidth , is the input of the kernel function is the bandwidth parameter of the kernel function, controlling the smoothness of the density estimation, is the weight for each data point , adjusted according to the characteristics of the data point such as frequency, importance or newness.​​​ 6. The data governance method of automatically detecting data outliers based on deep learning according to claim 1, wherein, According to the abnormal score result, according to the data type, adjust the data detection threshold, analyze the input data, identify the data points that exceed the threshold and record as abnormal, and obtain the data detection record The specific steps are as follows: Based on the abnormal score result, according to the abnormal score of multiple data points, adjust the threshold range to match the characteristics of the data type, including continuous and discrete, and perform threshold sensitivity analysis, and obtain the threshold setting adjustment record; According to the threshold setting adjustment record, compare the abnormal scores of multiple data points, identify the data points whose scores exceed the threshold and record as abnormal, and generate an abnormal data point marking record; Based on the abnormal data point marking record, record the numerical value, position in the data set, timestamp and abnormal type information of multiple abnormal data points, and obtain the data detection record.

7. The data governance method of automatically detecting data outliers based on deep learning according to claim 1, wherein, Based on the data detection record, the clustering of abnormal data is calculated, the abnormal type is identified, the development trend of data anomaly is evaluated by combining time series analysis, and the trend evaluation information is obtained The specific steps are as follows: Based on the data detection record, similarity analysis is performed on the abnormal data points, and clustering operation is performed according to the feature vector, the group attribution of the data points is analyzed, and the abnormal data clustering result is obtained; Based on the abnormal data clustering result, time series analysis is performed on multiple clusters, and the time dependence and development trend of multiple clusters are identified and recorded, and the abnormal trend analysis result is obtained; Based on the abnormal trend analysis result, multiple factors affecting the change trend of data anomaly are analyzed and identified, the correlation between multiple factors in the data set and the abnormal trend is calculated, and the trend evaluation information is generated.

8. The data governance method of automatically detecting data outliers based on deep learning according to claim 1, wherein, The specific formula for calculating the correlation between multiple factors in the data set and the abnormal trend is: ； wherein, denotes the observation value of a single factor at a specific time point, used to calculate the individual data points related to abnormal data, denotes the average value of all factor observation values, used to normalize the observation value of each factor, reducing the influence of different magnitudes, denotes the observation value of abnormal data corresponding to , which is the target variable for correlation analysis, denotes the average value of all abnormal data values, used to normalize the observation value of each abnormal data, is the correlation coefficient, used to quantify the strength and direction of the linear relationship between and .

9. The data governance method of automatically detecting data outliers based on deep learning according to claim 1, wherein, According to the trend evaluation information, by comparing the patterns of abnormal data and normal data, the error causes are identified, the data are corrected in combination with the abnormal types, and the specific steps for generating the data processing result are as follows: Based on the trend evaluation information, a plurality of abnormal types are analyzed, key differences between the patterns of abnormal data and normal data are identified, reasons for data abnormality are analyzed, and an abnormal reason analysis result is obtained; Based on the abnormal reason analysis result, logical verification is performed on the identified error causes, whether the data items violate known business rules and logical relationships is checked, data points that need to be adjusted are identified, and a to-be-corrected data mark is obtained; Based on the to-be-corrected data mark, data values and relationships of abnormal data points are adjusted and corrected according to business rules and logical requirements, consistency and accuracy of the data are verified, and a data processing result is generated.

Images