Data governance method for automatically detecting data abnormal value based on deep learning

Abstract

The invention relates to the technical field of data processing, in particular to a data management method for automatically detecting data abnormal values based on deep learning, which comprises the following steps of: based on an input data set, performing synchronization and missing value filling on a plurality of data sources, and performing standardization and normalization processing on data to obtain a preprocessed data set; according to the method, the data preprocessing is executed, the consistency and quality of data input are optimized, feature extraction and model layer-by-layer training are utilized, the recognition capability of the model for abnormal data is enhanced, the precision and efficiency of data anomaly detection are improved, calculation of residual errors and reconstruction errors is combined, the accuracy of abnormal data scoring is improved, and the accuracy of data anomaly detection is improved. The detection threshold value is dynamically adjusted by utilizing data characteristics, the false alarm rate is reduced, and the reliability and the stability of the enhanced data processing flow are ensured by combining time sequence analysis, evaluating the abnormal development trend, performing comparative analysis on abnormal data and normal data modes and performing data correction.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data governance method for automatically detecting data outliers based on deep learning. Background Art

[0002] The field of data processing technology focuses on the systematic conversion, analysis and interpretation of collected raw data through a variety of computer science and information technology means. It processes batches of structured and unstructured data through multiple steps of data collection, verification, storage, cleaning, conversion, analysis and visualization. It combines machine learning and artificial intelligence to improve data processing efficiency and accuracy, support decision-making, strategic planning and operational optimization, solve batch data storage, high-speed data transmission and real-time data analysis, and is applied to multiple industries such as financial services, healthcare, e-commerce and government management.

[0003] Among them, the data governance method for automatic detection of data outliers aims to automatically detect outliers in data sets and perform data governance. Through automated processing, it improves data quality, optimizes data accuracy and reliability, supports data analysis and decision-making, and improves the efficiency and accuracy of data quality management. Through deep learning models, it learns the behavior and patterns of normal data and automatically detects data points that deviate from standard patterns. Through automated anomaly detection, it reduces the need for manual intervention and enhances the speed and reliability of the data processing process. It is applied to financial risk control, network security, and market analysis to help enterprises and organizations ensure the integrity and availability of data.

[0004] Traditional data governance technology that automatically detects data outliers cannot effectively synchronize and unify data formats and standards when processing batch data from multiple sources, resulting in inconsistencies in data processing and accumulated errors. It relies on static rules or simple statistical models for anomaly detection, which limits its effectiveness when processing high-dimensional and complex relational data, resulting in high false positive or false negative rates. In a dynamic data environment, this fixed threshold method is difficult to adapt to the natural fluctuations of the data itself over time, resulting in delayed or incorrect decision-making. There is a lack of effective forecasting tools for predicting the development trend of data anomalies, and potential risks cannot be identified in advance, affecting the timeliness and accuracy of the company's strategy adjustments. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings existing in the prior art and propose a data governance method for automatically detecting data outliers based on deep learning.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a data governance method for automatically detecting data outliers based on deep learning, comprising the following steps:

[0007] S1: Based on the input data set, synchronize multiple data sources, fill missing values, and standardize and normalize the data to obtain the preprocessed data set;

[0008] S2: Based on the preprocessed data set, extract features from the data, train the model by adjusting the parameter settings of the input layer, the hidden layer, and the output layer, and construct an abnormal data learning model;

[0009] S3: Based on the abnormal data learning model, perform anomaly detection on the data set, analyze the residual and reconstruction error of the data point in the model output space, calculate the anomaly score of the data point, and generate anomaly score results by combining probability density estimation and neighborhood similarity analysis;

[0010] S4: According to the anomaly scoring result and the data type, the data detection threshold is adjusted, the input data is analyzed, data points exceeding the threshold are identified and recorded as anomalies, and a data detection record is obtained;

[0011] S5: Based on the data detection records, by calculating the clustering of abnormal data, identifying the abnormal type, and evaluating the development trend of data anomalies in combination with time series analysis, to obtain trend evaluation information;

[0012] S6: According to the trend assessment information, by comparing the patterns of abnormal data and normal data, identifying the cause of the error, and correcting the data in combination with the abnormal type to generate a data processing result.

[0013] As a further solution of the present invention, the preprocessed data set includes missing value filling records, scale parameter standardization processing results, and normalized range parameters; the abnormal data learning model includes weight parameter optimization records, bias parameter adjustment results, and accuracy indicators; the abnormal scoring results include calculated residual values, estimated reconstruction error data, and data anomaly scores; the data detection records include marked abnormal data points, adjusted detection thresholds, and identified data types; the trend evaluation information includes classified anomaly types, calculated cluster centers, and development trend evaluation results; the data processing results include data entity relationship correction records, logical relationship matching results, and data integrity optimization results.

[0014] As a further solution of the present invention, based on the input data set, multiple data sources are synchronized, missing values ​​are filled, and the data is standardized and normalized to obtain the specific steps of the preprocessed data set:

[0015] S101: By inputting a data set, connecting to an API interface, database, and reading files, the data is synchronized to the data warehouse in real time, the integrity of the data is analyzed, and the data synchronization result is obtained;

[0016] S102: Based on the data synchronization result, identify missing items and perform data filling, identify outlier data points and perform data cleaning, and generate missing value processing data;

[0017] S103: Based on the missing value processing data, the numerical features are standardized and the features are normalized to optimize the consistency of the data input and generate a preprocessed data set.

[0018] As a further solution of the present invention, based on the preprocessed data set, feature extraction is performed on the data, and the model is trained by adjusting the parameter settings of the input layer, the hidden layer, and the output layer. The specific steps of constructing the abnormal data learning model are:

[0019] S201: Based on the preprocessed data set, feature extraction and conversion are performed on the input data, and the impact on the target variable is evaluated by calculating the change rate and frequency distribution of multiple features to obtain a feature extraction result;

[0020] S202: Using the feature extraction results, construct a data analysis framework, transform and abstract the input data by stacking multiple processing layers, complete model construction, and obtain a network structure design result;

[0021] S203: Based on the network structure design result, through multiple iterations, the connection weights and bias values ​​of the multi-layer neurons are adjusted, the stability and accuracy of the model are evaluated, the fit of the data is optimized, and the abnormal data learning model is output.

[0022] As a further solution of the present invention, based on the abnormal data learning model, anomaly detection is performed on the data set, residuals and reconstruction errors of data points in the model output space are analyzed, anomaly scores of data points are calculated, and probability density estimation and neighborhood similarity analysis are combined to generate anomaly score results in the following specific steps:

[0023] S301: Based on the abnormal data learning model, for the model input data, the difference between the actual output and the expected output of multiple data points at the model output layer is calculated and recorded as a residual to obtain a data residual calculation result;

[0024] S302: Based on the data residual calculation result, the statistical positions of multiple data points in the data set are evaluated, and the probability density is calculated using the residual value to identify abnormal data points and obtain a probability density estimation result;

[0025] S303: Based on the probability density estimation result and in combination with the neighborhood similarity between data points, the abnormality of the data points is quantitatively evaluated to generate an abnormality scoring result.

[0026] As a further solution of the present invention, the specific formula for calculating the probability density is:

[0027]

[0028] Among them, p(d(x)) represents the probability density estimate at the residual value d(x) after kernel density estimation, d(x) represents the residual value of the data point currently being evaluated, which is used to determine the degree of abnormality of the data point, and d(x) i ) represents the residual value of other data points in the data set, which is used to compare with d(x) to estimate the density of d(x), n represents the total number of data points in the data set, i is the summation index, which represents the i-th data point currently being processed, and K is the kernel function used to calculate the relative density of point d(x) relative to point d(x) i )'s relative density contribution, represents the standardized distance, that is, the difference between the residual of the current data point and the residual of other points divided by the bandwidth h, which is the input of the kernel function K. h is the bandwidth parameter of the kernel function, which controls the smoothness of the density estimation. w i For each data point d(x i ) is adjusted based on characteristics of the data point such as frequency, importance, or recency.

[0029] As a further solution of the present invention, according to the abnormality scoring result, according to the data type, the data detection threshold is adjusted, the input data is analyzed, and the data points exceeding the threshold are identified and recorded as abnormalities. The specific steps of obtaining the data detection record are:

[0030] S401: Based on the anomaly scoring result, according to the anomaly scores of multiple data points, adjusting the threshold range to match the characteristics of the data type, including continuous and discrete types, and performing threshold sensitivity analysis to obtain a threshold setting adjustment record;

[0031] S402: Setting adjustment records according to the threshold, comparing abnormal scores of multiple data points, identifying data points with scores exceeding the threshold and recording them as abnormalities, and generating abnormal data point marking records;

[0032] S403: Based on the abnormal data point marking record, the values ​​of multiple abnormal data points, the positions in the data set, the timestamp and the abnormal type information are recorded to obtain a data detection record.

[0033] As a further solution of the present invention, based on the data detection records, by calculating the clustering of abnormal data, identifying the abnormal type, and combining time series analysis to evaluate the development trend of data anomalies, the specific steps of obtaining trend evaluation information are:

[0034] S501: Based on the data detection record, similarity analysis is performed on the abnormal data points, and clustering operation is performed according to the feature vector to analyze the group attribution of the data points to obtain abnormal data clustering results;

[0035] S502: Based on the abnormal data clustering result, perform time series analysis on multiple clusters, identify and record the time dependency and development trend of multiple clusters, and obtain abnormal trend analysis results;

[0036] S503: Based on the abnormal trend analysis result, analyze and identify various factors that affect the changing trend of data anomalies, calculate the correlation between various factors and abnormal trends in the data set, and generate trend evaluation information.

[0037] As a further solution of the present invention, the specific formula for calculating the correlation between multiple factors and abnormal trends in the data set is:

[0038]

[0039] Among them, x i Refers to the observed value of a single factor at a specific point in time, used to calculate individual data points related to abnormal data, Represents the average value of all factor observations, which is used to standardize the observations of each factor and reduce the impact of different magnitudes. i Refers to x i The observed value of the abnormal data is the target variable for correlation analysis. It represents the average value of all abnormal data values, which is used to standardize the observed value of each abnormal data. r is the correlation coefficient, which is used to quantify the strength and direction of the linear relationship between x and y.

[0040] As a further solution of the present invention, according to the trend assessment information, by comparing the patterns of abnormal data and normal data, identifying the cause of the error, and correcting the data in combination with the abnormal type, the specific steps of generating the data processing result are:

[0041] S601: Based on the trend assessment information, analyze multiple types of anomalies, identify key differences between abnormal data and normal data patterns, analyze causes of data anomalies, and obtain an abnormal cause analysis result;

[0042] S602: Based on the abnormal cause analysis result, logical verification is performed on the identified error cause, and by checking whether the data item violates known business rules and logical relationships, the data points that need to be adjusted are identified to obtain the data mark to be corrected;

[0043] S603: Based on the data mark to be corrected, according to business rules and logical requirements, adjust and correct the data value and relationship of the abnormal data point, verify the consistency and accuracy of the data, and generate a data processing result.

[0044] Compared with the prior art, the advantages and positive effects of the present invention are:

[0045] In the present invention, by performing data preprocessing, the consistency and quality of data input are optimized, the model's ability to recognize abnormal data is enhanced by utilizing feature extraction and layer-by-layer training of the model, the accuracy and efficiency of data anomaly detection are improved, the calculation of residuals and reconstruction errors is combined to improve the accuracy of abnormal data scoring, the detection threshold is dynamically adjusted by utilizing data characteristics to reduce the false alarm rate, time series analysis is combined to evaluate the abnormal development trend, and comparative analysis and data correction of abnormal data and normal data patterns are performed to ensure the reliability and stability of the enhanced data processing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the main steps of the present invention;

[0047] Figure 2 It is a schematic diagram of the refinement of S1 of the present invention;

[0048] Figure 3 It is a schematic diagram of the refinement of S2 of the present invention;

[0049] Figure 4 It is a schematic diagram of the refinement of S3 of the present invention;

[0050] Figure 5 It is a schematic diagram of the refinement of S4 of the present invention;

[0051] Figure 6 It is a detailed schematic diagram of S5 of the present invention;

[0052] Figure 7 It is a detailed schematic diagram of S6 of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0054] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, in the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0055] See also Figure 1 The present invention provides a technical solution, a data governance method for automatically detecting data outliers based on deep learning, comprising the following steps:

[0056] S1: Based on the input data set, synchronize multiple data sources, fill missing values, and standardize and normalize the data to obtain the preprocessed data set;

[0057] S2: Based on the preprocessed data set, extract features from the data, train the model by adjusting the parameter settings of the input layer, hidden layer, and output layer, and build an abnormal data learning model;

[0058] S3: Based on the abnormal data learning model, perform anomaly detection on the data set, analyze the residual and reconstruction error of the data points in the model output space, calculate the anomaly score of the data points, and generate the anomaly score result by combining probability density estimation and neighborhood similarity analysis;

[0059] S4: According to the anomaly scoring result and the data type, the data detection threshold is adjusted, the input data is analyzed, the data points exceeding the threshold are identified and recorded as anomalies, and a data detection record is obtained;

[0060] S5: Based on the data detection records, the abnormal data clusters are calculated to identify the abnormal type, and the development trend of the data anomaly is evaluated by combining time series analysis to obtain trend evaluation information;

[0061] S6: Based on the trend assessment information, the cause of the error is identified by comparing the patterns of abnormal data and normal data, and the data is corrected in combination with the abnormal type to generate a data processing result.

[0062] The preprocessed data set includes missing value filling records, scale parameter standardization processing results, and normalized range parameters. The abnormal data learning model includes weight parameter optimization records, bias parameter adjustment results, and accuracy indicators. The anomaly scoring results include calculated residual values, estimated reconstruction error data, and data anomaly scores. Data detection records include marked abnormal data points, adjusted detection thresholds, and identified data types. Trend assessment information includes classified anomaly types, calculated cluster centers, and development trend assessment results. Data processing results include data entity relationship correction records, logical relationship matching results, and data integrity optimization results.

[0063] See also Figure 2 , based on the input data set, synchronize multiple data sources, fill missing values, and standardize and normalize the data. The specific steps to obtain the preprocessed data set are:

[0064] S101: By inputting a data set, connecting to an API interface, database, and reading files, the data is synchronized to the data warehouse in real time, the integrity of the data is analyzed, and the data synchronization result is obtained;

[0065] In the above content, by connecting the API interface, database and file reading, according to the formula D =

[0066] N / T, calculation data synchronization speed;

[0067] In the formula, D represents the data synchronization speed, N represents the number of synchronized data, and T represents the time taken for synchronization;

[0068] Detailed explanation of the formula and the process of formula calculation and derivation:

[0069] Assume that 500 pieces of data are synchronized and it takes 50 seconds to calculate D:

[0070] D = 500 / 50 = 10 pieces / second

[0071] The result of 10 records per second indicates that the data synchronization efficiency is high. It is used to evaluate the data synchronization performance. The data synchronization result is obtained by synchronizing the data.

[0072] S102: Based on the data synchronization results, identify missing items and perform data filling, identify outlier data points and perform data cleaning, and generate missing value processing data;

[0073] In sub-step S102, based on the data synchronization results, data integrity analysis is performed. Python's Pandas library is used to quantitatively analyze the missing ratio in the dataset and identify fields with high missing rates, such as the common missing current and voltage readings in power company monitoring data. Next, an appropriate missing data interpolation method is selected based on the data type and distribution characteristics. The mean filling method is used for numerical fields, and the K-nearest neighbor method is used for more sensitive or volatile data such as voltage peaks. The missing data is filled by calculating the average of the nearest neighbors to the missing data point. For the processing of outlier data points, the Z-score value of each data point is calculated. Any data point that exceeds ±3 standard deviations is marked as an outlier. The interquartile range is used for verification and removal of extreme values. Outlier data points are evaluated based on their impact on the model and analysis and then decided to be corrected or deleted. After completing data processing, the program is changed to verify its accuracy and consistency through cross-validation and regression testing to ensure that the data repair operation does not introduce new biases. The updated dataset is uploaded back to the data warehouse to support further analysis and decision making.

[0074] S103: Processing data based on missing values, standardizing numerical features, and normalizing features to optimize the consistency of data input and generate a preprocessed data set;

[0075] In sub-step S103, based on the missing value processing data, standardization and normalization of numerical features are performed to optimize the consistency of data input. The standardization of numerical features uses the Z-score standardization method to convert the data into a distribution with a mean of 0 and a standard deviation of 1. In the power industry, standardization and normalization can effectively monitor and analyze grid load anomalies, optimize the energy distribution of the grid, ensure the stability and efficiency of power supply, and help power companies identify potential equipment failures and consumption anomalies in advance, reduce power outages and improve customer satisfaction. Normalization uses MinMaxScaler to scale each feature to a given minimum and maximum value to enhance the prediction accuracy of the model and improve the convergence speed of the algorithm. The processed data passes data quality detection, including consistency check and outlier re-detection, to ensure the quality standards of data processing. The generated preprocessed data set integrates the advantages of various data processing steps, provides high-quality input data for deep learning or machine learning models, and enhances the efficiency and effectiveness of model training.

[0076] See also Figure 3 , based on the preprocessed data set, feature extraction is performed on the data, and the model is trained by adjusting the parameter settings of the input layer, hidden layer, and output layer. The specific steps for constructing the abnormal data learning model are:

[0077] S201: Based on the preprocessed data set, feature extraction and conversion are performed on the input data, and the impact on the target variable is evaluated by calculating the change rate and frequency distribution of multiple features to obtain feature extraction results;

[0078] In sub-step S201, the features in the preprocessed data set are extracted and transformed by using multiple algorithms, and the rate of change of the features is calculated using the time series analysis method. In the power industry, the rate of change and frequency distribution analysis of the features can be used to monitor the periodic changes in power demand and supply, and to promptly discover abnormal patterns that may lead to unstable power supply. This is of great significance for ensuring the reliability of the power grid and optimizing load distribution. Frequency distribution analysis is implemented through Fourier transform to identify and quantify the regularity and abnormality of feature changes over time. The rate of change of the feature provides information on the sensitivity of the data point to the change of time. The frequency distribution reveals the periodicity and abnormal frequency components of the data. For example, in financial data analysis, the rate of change is used to reveal the volatility of asset prices, and the frequency distribution can point out the periodic peaks of trading activities. By evaluating multiple features in the data set, the special positive that has the greatest impact on the target variable is determined. The feature extraction result is a comprehensive feature score that summarizes the influence of each feature on the target.

[0079] S202: Using the feature extraction results, a data analysis framework is constructed, and input data is transformed and abstracted by stacking multiple processing layers to complete model construction and obtain a network structure design result;

[0080] In the above content, by stacking multiple processing layers, according to the formula W new =W old +ηΔW, calculation weight adjustment;

[0081] Where W new Represents the new weight value, W old represents the old weight value, η represents the learning rate, and ΔW represents the gradient change of the weight;

[0082] Detailed explanation of the formula and the process of formula calculation and derivation:

[0083] Assume that the original weight W old is 0.5, the gradient change ΔW is 0.1, the learning rate η is 0.01, and W is calculated new :

[0084] W new =0.5+0.01×0.1=0.501

[0085] The result 0.501 shows that the weights are slightly adjusted to optimize the network weights, and the learning effect and accuracy of the model are improved by adjusting the learning rate and gradient, thus obtaining the network structure design result.

[0086] S203: Based on the network structure design results, through multiple iterations, the connection weights and bias values ​​of the multi-layer neurons are adjusted to evaluate the stability and accuracy of the model, optimize the fit of the data, and output the abnormal data learning model;

[0087] In sub-step S203, based on the network structure design result, the gradient descent algorithm is used to optimize the learning efficiency of the model by adjusting the weights and biases of the multi-layer neural network to obtain an optimized abnormal data learning model. The formula for calculating the weight adjustment is specifically:

[0088]

[0089] Among them, W t represents the weight parameter at training step t, η represents the learning rate, Represents the weight parameter W t The corresponding gradient of the error function E, W t+1 Represents the weight parameter after updating in training step t+1. This formula directly reduces the value of the error function by calculating the weight adjustment amount at each step, thereby improving the model's ability to capture data features and enhancing the model's accuracy in identifying abnormal data.

[0090] formula:

[0091]

[0092] Detailed explanation of the formula and the process of formula calculation and derivation:

[0093] The formula is used to update the weights in the neural network through the gradient descent method to reduce the error function E and improve the model's ability to identify abnormal data;

[0094] Parameter meaning and setting value:

[0095] W t is the weight of the current training step t, obtained from the model initialization stage, and calculated through the process of forward propagation and back propagation of small batches of data. η is the learning rate, which is assumed to be 0.01. It is adjusted based on the model performance in the preliminary test to ensure that the learning progress is neither too fast to cause oscillation nor too slow to cause too long convergence time. is the loss function E with respect to weight W t The gradient of the network is obtained through the back propagation algorithm of the current batch of data. The gradient indicates the rate of change of the error function under the current weight and is used to guide the direction and amplitude of the weight update. Assume that in a certain step, the weight W of the network t is 1.5, the gradient of the loss function The calculated result is 0.05;

[0096] Substitute the parameters into the formula for calculation:

[0097] W t+1 =1.5-0.01×0.05=1.5-0.0005=1.4995

[0098] The result 1.4995 shows that the weights have been slightly adjusted, indicating that the current learning rate and gradient adjustment effectively optimize the weights in the direction of reducing the error, reducing the model's loss and enhancing the model's learning efficiency and accuracy. The results are crucial to confirming the correct adjustment of the model's weights during training, ensuring that the model can continue to learn and improve the recognition of abnormal data.

[0099] See also Figure 4 , based on the abnormal data learning model, anomaly detection is performed on the data set, the residual and reconstruction error of the data points in the model output space are analyzed, the anomaly score of the data points is calculated, and the specific steps of generating the anomaly score result by combining probability density estimation and neighborhood similarity analysis are as follows:

[0100] S301: Based on the abnormal data learning model, for the model input data, the difference between the actual output and the expected output of multiple data points at the model output layer is calculated and recorded as the residual to obtain the data residual calculation result;

[0101] In sub-step S301, the input data is processed by the abnormal data learning model, and the difference between the actual output and the expected output of each data point in the model output layer, i.e., the residual, is calculated to measure the prediction accuracy of the model for the target data point. In the power industry, by monitoring the residual between the output power of the power station and the predicted value, the performance degradation of the equipment or the maintenance requirements can be discovered in time, the power generation efficiency and maintenance scheduling can be optimized, and overload or power loss can be prevented to ensure the stable operation of the power grid. The residual calculation involves the statistical method of the sum of squared errors or absolute errors, which reflects the performance of the model at each data point. A large residual value indicates that the prediction of the model at the data point is far from the actual situation, indicating that the model fails to capture the key characteristics of the data, or the data point itself has abnormal characteristics. The data residual calculation results obtained provide basic data for subsequent steps for analyzing and optimizing the performance of the model.

[0102] S302: Based on the data residual calculation result, the statistical positions of multiple data points in the data set are evaluated, and the probability density is calculated using the residual value to identify abnormal data points and obtain the probability density estimation result;

[0103] The specific formula for calculating probability density is:

[0104]

[0105] Among them, p(d(x)) represents the probability density estimate at the residual value d(x) after kernel density estimation, d(x) represents the residual value of the data point currently being evaluated, which is used to determine the degree of abnormality of the data point, and d(x) i ) represents the residual value of other data points in the data set, which is used to compare with d(x) to estimate the density of d(x), n represents the total number of data points in the data set, i is the summation index, which represents the i-th data point currently being processed, and K is the kernel function used to calculate the relative density of point d(x) relative to point d(x) i )'s relative density contribution, represents the standardized distance, that is, the difference between the residual of the current data point and the residual of other points divided by the bandwidth h, which is the input of the kernel function K. h is the bandwidth parameter of the kernel function, which controls the smoothness of the density estimation. w i For each data point d(x i ) is adjusted based on characteristics of the data point such as frequency, importance, or recency.

[0106] formula:

[0107]

[0108] Parameter meaning and setting value:

[0109] d(x) is the residual value of the current data point, which is the difference between the observed value and the model predicted value. The residual value reflects the degree of deviation of the data point relative to the model prediction and is used for subsequent anomaly scoring. Assume that d(x) = 5;

[0110] d(x i ) is the residual value of other data points in the data set. The same residual calculation as d(x) is performed on each data point in the data set to provide a reference residual value, which is compared with d(x) and kernel density estimation is performed. Assume that the residual set of data points is [3, 7, 5, 9, 6];

[0111] n is the total number of data points, which is obtained by counting the number of data points in the statistical data set and is used to calculate the bandwidth and the sum in the standardized kernel density estimation. It is assumed to be 5;

[0112] h is the kernel bandwidth. The bandwidth is a smoothing parameter in kernel density estimation. It is calculated by Silverman's rule and is used to control the smoothness of the kernel function, affecting the precision and generalization ability of density estimation. The standard deviation based on the residual is set and calculated as h = 1.06 × std(d) × n -1 / 5 , assuming the standard deviation std(d) is 2, h = 1.06 × 2 × 5 -1 / 5 ≈1.87;

[0113] K is a Gaussian kernel function, which is used to calculate the weight contribution between different residual values ​​and estimate the probability density.

[0114] w i is the data point weight, which is used to adjust the importance of each data point contribution when calculating the kernel density estimate. Assuming that the weights are evenly distributed, each w i =1;

[0115] Assuming that the given data point residual set is [3, 7, 5, 9, 6], calculate the contribution of each point;

[0116] For d(x i )=3:

[0117]

[0118] For d(x i )=7:

[0119]

[0120] For d(x i )=5:

[0121]

[0122] For d(x i )=9:

[0123]

[0124] For d(x i )=6:

[0125]

[0126] Substitute the parameters into the formula for calculation:

[0127]

[0128]

[0129]

[0130] The calculation result p(d(x))≈0.133 represents the probability density estimate of the current data point d(x)=5. The value reflects the degree of abnormality of the point relative to other points in the data set. A lower density value indicates that the point is a potential outlier. The density value is used to quickly identify data points that need attention in a large amount of data.

[0131] S303: Based on the probability density estimation result and the neighborhood similarity between the data points, the abnormality of the data points is quantitatively evaluated to generate an abnormality scoring result;

[0132] In sub-step S303, based on the probability density estimation result, the neighborhood similarity between data points is quantitatively evaluated, the abnormality of each data point is scored, and the local anomaly factor algorithm is used to perform anomaly detection to identify local anomalies that are significantly different from other data points in their neighborhoods. It is especially effective for complex data sets containing clusters. At the beginning, the synthetic data set containing normal and abnormal data is standardized, and then the key parameters of the LOF algorithm are set, including the neighborhood size. This parameter determines the number of neighboring points considered when calculating the local density and directly affects the sensitivity of the algorithm. Each data point calculates its local reachable density and local Anomaly factor, where points with low local reachable density are considered potential anomalies. The algorithm generates a LOF score for each data point, which indicates its relative degree as an outlier. Points with scores higher than the set threshold are marked as anomalies. Finally, normal and abnormal data points are visualized through a scatter plot, which intuitively shows the detection effect of the LOF algorithm and recommends response measures based on the results, including system configuration adjustments or security protection enhancements, to provide support for data analysis and decision-making. By identifying and isolating potential abnormal data points, the quality of data processing and the stability of the model are improved. The generated anomaly score results are of great significance for subsequent data monitoring and the formulation of anomaly handling strategies.

[0133] See also Figure 5 ,According to the anomaly scoring results, according to the data type, adjust the data detection threshold, analyze the input data, identify the data points that exceed the threshold and record them as anomalies. The specific steps to obtain the data detection record are:

[0134] S401: Based on the anomaly scoring result, according to the anomaly scores of multiple data points, the threshold range is adjusted to match the characteristics of the data type, including continuous and discrete types, and a threshold sensitivity analysis is performed to obtain a threshold setting adjustment record;

[0135] In sub-step S401, based on the anomaly scoring results, the threshold range is adjusted by analyzing the data type characteristics, including the differentiated processing of continuous and discrete data. The threshold setting of continuous data takes into account the distribution characteristics of the data, including the mean and standard deviation. Discrete data depends on the probability of occurrence of frequency or category. In the power industry, different thresholds are set to monitor abnormal fluctuations in grid load and power consumption, and abnormal consumption caused by demand peaks or equipment failures is identified in a timely manner to optimize energy distribution and prevent grid failures. Threshold sensitivity analysis uses sensitivity analysis and ROC curves to evaluate the impact of different threshold settings on model performance and determine the optimal threshold to maximize the accuracy of anomaly detection and reduce false alarm rates. The adjustment record covers the results of each threshold test and the corresponding model performance indicators. The record provides data support and decision-making basis for subsequent threshold optimization to ensure that the threshold setting matches the data characteristics and detection requirements.

[0136] S402: adjusting the record according to the threshold setting, comparing the abnormal scores of multiple data points, identifying the data points whose scores exceed the threshold and recording them as abnormalities, and generating abnormal data point marking records;

[0137] In the above content, the records are adjusted according to the threshold settings, and the anomaly scores of multiple data points are compared according to the formula The total difference in scores exceeding the threshold was calculated;

[0138] In the formula, S represents the total difference, x i represents the anomaly score of the ith data point, T represents the threshold, and n represents the total number of data points;

[0139] Detailed explanation of the formula and the process of formula calculation and derivation:

[0140] Assume that there are 5 data points with anomaly scores of 72, 78, 65, 80, and 67, respectively, and the threshold T is 75. Calculate S:

[0141] S=(72-75) 2 +(78-75) 2 +(65-75) 2 +(80-75) 2 +(67-75) 2

[0142] S = 9 + 9 + 100 + 25 + 64 = 207

[0143] Result 207 indicates the total amount of difference in scores beyond the threshold, and the formula is used to evaluate the total deviation between the anomaly score and the set threshold, identify and record the data points beyond the threshold as anomalies, and generate a labeled record of the anomaly data points.

[0144] S403: Based on the abnormal data point marking record, the values ​​of multiple abnormal data points, the positions in the data set, the timestamps and the abnormal type information are recorded to obtain a data detection record;

[0145] In sub-step S403, based on the abnormal data point marking record, the value of each abnormal data point, the location in the data set, the timestamp and the abnormal type information are recorded. Data tracking and recording technology, including database logs and time series analysis tools, are used to ensure that the information of each abnormal point is accurately recorded and stored. The abnormal type information includes data noise, deviation anomaly or potential data tampering. The target information is used for subsequent data repair, model adjustment and risk assessment. In the power industry, the marking and governance of abnormal data points are crucial to preventing system failures and optimizing energy distribution. By real-time monitoring of power grid operation data, abnormalities such as overload or voltage instability can be identified and handled in a timely manner to ensure the stability and efficiency of the power supply system, and data points marked as abnormal can be Take corresponding governance measures. For abnormal data of lower importance, apply data elimination algorithms, such as using the Z-Score method to automatically identify and remove data points that are statistically significantly deviated from the normal range. After removing abnormal data points from the data set, recalculate the statistical summary of the data set to verify the consistency of the data. For data of general importance, use data verification and correction tools to automatically correct data format errors and range errors. Record logs after each correction and reload the data model to check the effect of the correction. Conduct cause analysis and impact scope investigation for key data, use decision tree analysis to identify the root cause of the anomaly, collect the input path and historical records of relevant data during the investigation, and confirm the starting point of the data anomaly and the affected business process.

[0146] See also Figure 6 ,Based on the data detection records, by calculating the clustering of abnormal data, identifying the abnormal type, and combining time series analysis to evaluate the development trend of data anomalies, the specific steps to obtain trend evaluation information are:

[0147] S501: Based on the data detection records, similarity analysis is performed on the abnormal data points, and clustering operation is performed according to the feature vectors to analyze the group belonging of the data points and obtain abnormal data clustering results;

[0148] In sub-step S501, by performing similarity analysis on abnormal data points and performing clustering operations based on feature vectors, the K-means clustering algorithm is used to divide the data into several groups based on the feature vectors of the data points, and minimize the distance between the data points and the center point in each group. The clustering process randomly selects K data points as the initial group center, assigns each data point to the nearest group center, and then updates the center position of each group. The process is repeated until the group center no longer changes. Abnormal data points with similar characteristics are grouped, and the common characteristics and behaviors of the data points in each group are analyzed. This is particularly important in the power industry. In power grid monitoring, clustering analysis can effectively distinguish different types of power consumption anomalies, such as peak anomalies or continuous inefficient operation, which helps to optimize energy distribution and prevent power waste. The abnormal data clustering results obtained are used to identify potential patterns in the data and clustered areas of abnormal behavior.

[0149] S502: Based on the abnormal data clustering results, perform time series analysis on multiple clusters, identify and record the time dependencies and development trends of multiple clusters, and obtain abnormal trend analysis results;

[0150] In sub-step S502, based on the abnormal data clustering results, the autoregressive moving average model is used to analyze the time dependence and development trend of each cluster. By combining the autoregressive and moving average models, the time series characteristics of the data are matched. By calculating the linear combination of the data at each time point and the data at the previous time points and the weighted sum of random errors, the dependence and intrinsic trend of the time series are revealed. The process involves selecting the number of matching lag terms, estimating model parameters, and performing model fitting and verification. In the power industry, time series analysis is of great value for monitoring and predicting power demand fluctuations, changes in power generation efficiency, or abnormal grid loads. By analyzing the time series patterns of historical consumption data, future power demand peaks are predicted, and grid operation and energy allocation strategies are optimized. The results of the analysis show the changing laws and trends of different clusters in the time dimension. The abnormal trend analysis results obtained provide a scientific basis for understanding the behavior of each cluster over time and support the prediction and decision-making of future data.

[0151] S503: Based on the abnormal trend analysis results, analyze and identify various factors that affect the change trend of data anomalies, calculate the correlation between various factors and abnormal trends in the data set, and generate trend evaluation information;

[0152] The specific formula for calculating the correlation between multiple factors and abnormal trends in the data set is:

[0153]

[0154] Among them, x iRefers to the observed value of a single factor at a specific point in time, used to calculate individual data points related to abnormal data, Represents the average value of all factor observations, which is used to standardize the observations of each factor and reduce the impact of different magnitudes. i Refers to x i The observed value of the abnormal data is the target variable for correlation analysis. It represents the average value of all abnormal data values, which is used to standardize the observed value of each abnormal data. r is the correlation coefficient, which is used to quantify the strength and direction of the linear relationship between x and y.

[0155] formula:

[0156]

[0157] Detailed explanation of the formula and the process of formula calculation and derivation:

[0158] The formula is used to calculate the correlation between multiple clusters and time in the abnormal data clustering results, and the correlation coefficient r is used to measure the linear correlation between the clustering index and time in the time series;

[0159] Parameter meaning and setting value:

[0160] x i represents the clustering index at the ith time point, which is the dependent variable considered in time series analysis, for example, the abnormal intensity of a particular cluster at different time points;

[0161] y i For the corresponding x i The time points, i.e., the independent variables, are usually continuous time series (such as days, hours, etc.);

[0162] For all x i The average value of represents the average level of clustering index during the analysis period;

[0163] For all y i The mean of , represents the average time point during the analysis period;

[0164] n is the sample size, the total number of days or time points included in the analysis;

[0165] Substitute the parameters into the formula for calculation:

[0166] Assume that the clustering index x of the five time points is [50, 55, 53, 60, 58], and the time series y is [1, 2, 3, 4, 5];

[0167] Calculate the average and

[0168]

[0169]

[0170] Calculate r:

[0171]

[0172]

[0173]

[0174] The result r ≈ 0.008 shows that in this target dataset, the correlation between the clustering index and time is low, indicating that the clustering index hardly changes over time in the selected time period.

[0175] See also Figure 7 ,According to the trend evaluation information, by comparing the patterns of abnormal data and normal data, identifying the cause of the error, and correcting the data according to the abnormal type, the specific steps to generate the data processing results are:

[0176] S601: Based on the trend assessment information, analyze various types of anomalies, identify key differences between abnormal data and normal data patterns, analyze the causes of data anomalies, and obtain an analysis result of the causes of anomalies;

[0177] In the above content, by analyzing various types of anomalies, according to the formula Calculate the proportion of abnormal types;

[0178] In the formula, R represents the percentage of abnormal types, n e The number of data points representing the target anomaly type, n t Represents the total number of data points;

[0179] Detailed explanation of the formula and the process of formula calculation and derivation:

[0180] Assume that the total number of data points is n t is 1000, the number of data points of a specific anomaly type n e is 150, calculate R:

[0181]

[0182] The result 15% shows the proportion of the target anomaly type in the total data. The formula is used to identify and analyze abnormal patterns in the data and key differences from normal data patterns, helping to understand the probability and frequency of abnormal data occurrence and obtain abnormal cause analysis results.

[0183] S602: Based on the abnormal cause analysis result, the identified error cause is logically verified, and the data points that need to be adjusted are identified by checking whether the data items violate known business rules and logical relationships, and obtaining the data mark to be corrected;

[0184] In sub-step S602, based on the results of the abnormal cause analysis, assertion detection and rule engine are used to verify the identified error causes. Assertion detection checks whether data items violate business rules, including value range, data type and logical consistency, by setting assertions during data processing. The rule engine uses preset business rules to automatically check the logical relationship of the data to ensure that the data points meet the business logic and data integrity requirements. This is crucial to ensuring the accuracy and consistency of electricity consumption data in the power industry. The rule engine automatically detects data points that are significantly inconsistent with historical consumption patterns, identifies metering errors and data tampering, and takes timely measures to avoid financial losses and power supply problems. Data points that violate logic are marked, and the data tags to be corrected include problem descriptions and recommended corrective measures for each data point. This information is crucial for data quality management and maintenance, helping the data management team to quickly locate and handle data anomalies.

[0185] S603: Based on the data mark to be corrected, according to business rules and logic requirements, adjust and correct the data value and relationship of the abnormal data point, verify the consistency and accuracy of the data, and generate a data processing result;

[0186] In sub-step S603, according to business rules and logical requirements, necessary adjustments and corrections are made to abnormal data points in the data tags to be corrected, and data cleaning technology and constraint satisfaction programming are used to ensure the correctness of data values ​​and relationships. Data cleaning includes data value replacement, format adjustment and relationship reconstruction. Constraint satisfaction programming restores data consistency by setting and solving data constraint problems. After the adjustment is completed, data consistency and accuracy are verified by running data quality checks again and using data integrity tests to ensure that the corrected data points comply with business logic and rules. Ensuring data accuracy and consistency in the power industry is crucial for monitoring and managing the efficiency of power grids. By correcting power usage data, power companies can accurately predict power demand and adjust power supply plans, reduce power waste and optimize resource allocation. The generated data processing results mark all corrected and verified data points, providing high-quality and reliable data support for business decisions.

[0187] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A data governance method for automatically detecting data outliers based on deep learning, characterized in that: The following steps are involved: Based on the input data set, multiple data sources are synchronized, missing values ​​are filled, and the data is standardized and normalized to obtain a preprocessed data set; Based on the preprocessed data set, feature extraction is performed on the data, and the model is trained by adjusting the parameter settings of the input layer, the hidden layer, and the output layer to construct an abnormal data learning model; Based on the abnormal data learning model, anomaly detection is performed on the data set, residuals and reconstruction errors of data points in the model output space are analyzed, anomaly scores of data points are calculated, and anomaly score results are generated by combining probability density estimation and neighborhood similarity analysis; According to the anomaly scoring result, according to the data type, the data detection threshold is adjusted, the input data is analyzed, the data points exceeding the threshold are identified and recorded as anomalies, and a data detection record is obtained; Based on the data detection records, by calculating the clustering of abnormal data, identifying the abnormal type, and combining time series analysis to evaluate the development trend of data anomalies, trend evaluation information is obtained; According to the trend assessment information, by comparing the patterns of abnormal data and normal data, the cause of the error is identified, and the data is corrected in combination with the abnormal type to generate a data processing result.

2. The data governance method for automatically detecting data outliers based on deep learning according to claim 1 is characterized in that: The preprocessed data set includes missing value filling records, scale parameter standardization processing results, and normalized range parameters. The abnormal data learning model includes weight parameter optimization records, bias parameter adjustment results, and accuracy indicators. The abnormal scoring results include calculated residual values, estimated reconstruction error data, and data anomaly scores. The data detection records include marked abnormal data points, adjusted detection thresholds, and identified data types. The trend assessment information includes classified anomaly types, calculated cluster centers, and development trend assessment results. The data processing results include data entity relationship correction records, logical relationship matching results, and data integrity optimization results.

3. The data governance method for automatically detecting data outliers based on deep learning according to claim 1 is characterized in that: Based on the input data set, multiple data sources are synchronized, missing values ​​are filled, and the data is standardized and normalized. The specific steps to obtain the preprocessed data set are: By inputting data sets, connecting API interfaces, databases, and file reading, data is synchronized to the data warehouse in real time, and the integrity of the data is analyzed to obtain data synchronization results; Based on the data synchronization results, identify missing items and perform data filling, identify outlier data points and perform data cleaning, and generate missing value processing data; Based on the missing value processing data, the numerical features are standardized and the features are normalized to optimize the consistency of data input and generate a preprocessed data set.

4. The data governance method for automatically detecting data outliers based on deep learning according to claim 1 is characterized in that: Based on the preprocessed data set, feature extraction is performed on the data, and the model is trained by adjusting the parameter settings of the input layer, hidden layer, and output layer. The specific steps for constructing the abnormal data learning model are as follows: Based on the preprocessed data set, feature extraction and conversion are performed on the input data, and the impact on the target variable is evaluated by calculating the change rate and frequency distribution of multiple features to obtain feature extraction results; Using the feature extraction results, a data analysis framework is constructed, and input data is transformed and abstracted by stacking multiple processing layers to complete model construction and obtain a network structure design result; Based on the network structure design results, through multiple iterations, the connection weights and bias values ​​of the multi-layer neurons are adjusted, the stability and accuracy of the model are evaluated, the fit of the data is optimized, and the abnormal data learning model is output.

5. The data governance method for automatically detecting data outliers based on deep learning according to claim 1 is characterized in that: Based on the abnormal data learning model, the data set is detected for anomalies, the residual and reconstruction error of the data points in the model output space are analyzed, the anomaly score of the data points is calculated, and the specific steps of generating the anomaly score result by combining probability density estimation and neighborhood similarity analysis are as follows: Based on the abnormal data learning model, for the model input data, the difference between the actual output and the expected output of multiple data points at the model output layer is calculated and recorded as residuals to obtain the data residual calculation result; Based on the data residual calculation results, the statistical positions of multiple data points in the data set are evaluated, and the residual values ​​are used to calculate the probability density, identify abnormal data points, and obtain the probability density estimation results; Based on the probability density estimation result and combined with the neighborhood similarity between data points, the abnormality of the data points is quantitatively evaluated to generate an abnormality scoring result.

6. The data governance method for automatically detecting data outliers based on deep learning according to claim 5 is characterized in that: The specific formula for calculating the probability density is: Among them, p(d(x)) represents the probability density estimate at the residual value d(x) after kernel density estimation, d(x) represents the residual value of the data point currently being evaluated, which is used to determine the degree of abnormality of the data point, and d(x) i ) represents the residual value of other data points in the data set, which is used to compare with d(x) to estimate the density of d(x), n represents the total number of data points in the data set, i is the summation index, which represents the i-th data point currently being processed, and K is the kernel function used to calculate the relative density of point d(x) relative to point d(x) i )'s relative density contribution, represents the standardized distance, that is, the difference between the residual of the current data point and the residual of other points divided by the bandwidth h, which is the input of the kernel function K. h is the bandwidth parameter of the kernel function, which controls the smoothness of the density estimation. w i For each data point d(x i ) is adjusted based on characteristics of the data point such as frequency, importance, or recency.

7. The data governance method for automatically detecting data outliers based on deep learning according to claim 1 is characterized in that: According to the anomaly scoring result, the data detection threshold is adjusted according to the data type, the input data is analyzed, and the data points exceeding the threshold are identified and recorded as anomalies. The specific steps for obtaining the data detection record are as follows: Based on the anomaly scoring result, according to the anomaly scores of multiple data points, the threshold range is adjusted to match the characteristics of the data type, including continuous and discrete types, and a threshold sensitivity analysis is performed to obtain a threshold setting adjustment record; Setting adjustment records according to the threshold, comparing abnormal scores of multiple data points, identifying data points with scores exceeding the threshold and recording them as abnormal, and generating abnormal data point marking records; Based on the abnormal data point marking record, the values ​​of multiple abnormal data points, the positions in the data set, the timestamp and the abnormal type information are recorded to obtain a data detection record.

8. The data governance method for automatically detecting data outliers based on deep learning according to claim 1, characterized in that: Based on the data detection records, by calculating the clustering of abnormal data, identifying the abnormal type, and combining time series analysis to evaluate the development trend of data anomalies, the specific steps of obtaining trend evaluation information are as follows: Based on the data detection records, similarity analysis is performed on the abnormal data points, and clustering operations are performed according to the feature vectors to analyze the group attribution of the data points to obtain abnormal data clustering results; Based on the abnormal data clustering results, time series analysis is performed on multiple clusters to identify and record the time dependencies and development trends of multiple clusters to obtain abnormal trend analysis results; Based on the abnormal trend analysis results, multiple factors that affect the changing trend of data anomalies are analyzed and identified, the correlation between multiple factors and abnormal trends in the data set is calculated, and trend evaluation information is generated.

9. The data governance method for automatically detecting data outliers based on deep learning according to claim 8, characterized in that: The specific formula for calculating the correlation between multiple factors and abnormal trends in the data set is: Among them, x i Refers to the observed value of a single factor at a specific point in time, used to calculate individual data points related to abnormal data, Represents the average value of all factor observations, which is used to standardize the observations of each factor and reduce the impact of different magnitudes. i Refers to x i The observed value of the abnormal data is the target variable for correlation analysis. It represents the average value of all abnormal data values, which is used to standardize the observed value of each abnormal data. r is the correlation coefficient, which is used to quantify the strength and direction of the linear relationship between x and y.

10. The data governance method for automatically detecting data outliers based on deep learning according to claim 1, characterized in that: According to the trend assessment information, by comparing the patterns of abnormal data and normal data, identifying the cause of the error, and correcting the data in combination with the abnormal type, the specific steps of generating the data processing result are: Based on the trend assessment information, multiple types of anomalies are analyzed to identify key differences between abnormal data and normal data patterns, analyze the causes of data anomalies, and obtain an analysis result of the causes of the anomalies; Based on the abnormal cause analysis result, the identified error cause is logically verified, and the data points that need to be adjusted are identified by checking whether the data items violate known business rules and logical relationships, so as to obtain the data mark to be corrected; Based on the data markers to be corrected, according to business rules and logical requirements, the data values ​​and relationships of abnormal data points are adjusted and corrected, and the consistency and accuracy of the data are verified to generate data processing results.

Images