An abnormal data detection method and device

Abstract

Embodiments of the present application relate to an abnormal data detection method and device. The method comprises: inputting W pieces of to-be-detected data distributed in W dimensions into each target model corresponding to each target dimension; the target model corresponding to any target dimension is obtained by removing strong influence point data from sample data; for any target model, if it is determined that the W pieces of to-be-detected data do not satisfy the target model, each dimension contained in the target model is determined as an abnormal dimension; for any abnormal dimension, the abnormal probability of the abnormal dimension being determined as an abnormal dimension in each target model is determined; and whether the to-be-detected data corresponding to the abnormal dimension is abnormal data is determined according to the abnormal probability. Not only the existence of abnormal data in the W pieces of to-be-detected data is detected, but also which dimension of data in the W pieces of to-be-detected data is abnormal data is accurately located. Thus, the automatic and accurate positioning of abnormal data is realized, and manual checking is not needed.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an abnormal data detection method, apparatus, computing device and computer-readable storage medium. Background Technology

[0002] With the development of computer technology, more and more technologies are being applied in the financial field. The traditional financial industry is gradually transforming into financial technology (FiKtech). However, due to the security and real-time requirements of the financial industry, higher demands are also being placed on technology.

[0003] With the development of the internet finance industry and the increasing sophistication of computer technology, the scale of data generated by financial systems per unit of time across different dimensions is growing larger and larger, with these dimensions reaching hundreds or even thousands. Inevitably, anomalous data will exist within this data. The causes of anomalous data are numerous, such as errors in manual data entry or errors in computer processing. The impact of anomalous data on subsequent statistical processing steps cannot be underestimated; therefore, it is essential to detect anomalous data.

[0004] The current methods for detecting abnormal data have low accuracy, and even after abnormal data is detected, manual verification is still required, which incurs high labor and time costs.

[0005] In summary, an anomaly detection method is provided to improve the accuracy of anomaly detection. Summary of the Invention

[0006] This invention provides an abnormal data detection method to improve the accuracy of abnormal data detection.

[0007] In a first aspect, embodiments of the present invention provide an abnormal data detection method, comprising:

[0008] W data points to be detected, distributed across W dimensions, are input into each target model corresponding to each target dimension. The target model corresponding to any target dimension is obtained by removing sample data after removing data with strong influence points. Data with strong influence points refers to sample data whose influence on the accuracy of the target model does not meet the preset conditions.

[0009] For any target model, if it is determined that W data to be detected do not meet the target model, then each dimension contained in the target model is determined as an abnormal dimension.

[0010] For any anomalous dimension, determine the probability that the anomalous dimension is identified as an anomalous dimension in each target model;

[0011] Based on the anomaly probability, determine whether the data to be detected corresponding to the anomaly dimension is an anomaly.

[0012] By inputting W data points distributed across W dimensions into the corresponding target models for each target dimension, the anomalous dimensions are identified. Then, the probability of each anomalous dimension being identified as anomalous in each target model is determined. Based on this probability, it is determined whether the data points corresponding to those anomalous dimensions are indeed anomalous. This not only detects anomalous data among the W data points but also accurately pinpoints which dimension of the data is anomalous. This achieves automated and accurate anomalous data location without requiring manual review.

[0013] In some embodiments, after determining the dimensions included in the target model as anomalous dimensions, the method further includes:

[0014] For any abnormal dimension, obtain the historical data corresponding to that abnormal dimension;

[0015] By clustering the data to be detected corresponding to the anomaly dimension and the historical data, the anomaly score of the data to be detected corresponding to the anomaly dimension is determined.

[0016] Determining whether the data to be detected corresponding to the anomaly dimension is anomaly data based on the anomaly probability includes:

[0017] Based on the anomaly probability and the anomaly score, it is determined whether the data to be detected corresponding to the anomaly dimension is anomaly data.

[0018] By acquiring historical data for the anomaly dimension, the data to be detected corresponding to the anomaly dimension and each historical data point are clustered to obtain anomaly scores for the data to be detected corresponding to the anomaly dimension. Combining the anomaly probability and the anomaly score determines whether the data to be detected corresponding to that anomaly dimension is indeed anomaly. This combination of two methods considers both the probability of the anomaly dimension being identified as an anomaly dimension and the historical data for that anomaly dimension, thus increasing the accuracy of anomaly detection.

[0019] In some embodiments, the anomaly score of the data to be detected corresponding to the anomaly dimension is determined by clustering the data to be detected corresponding to the anomaly dimension and the historical data, including:

[0020] Construct an isolated binary tree for the data to be detected corresponding to the anomaly dimension and for each historical data;

[0021] Calculate the anomaly score of the data to be detected corresponding to the anomaly dimension in the isolated binary tree.

[0022] By constructing an isolated binary tree of the data to be detected corresponding to the anomaly dimension and each historical data, the accuracy of identifying anomaly data is improved.

[0023] In some embodiments, the target model corresponding to any target dimension is determined by the following methods:

[0024] Obtain initial n sets of sample data distributed across M dimensions; where each set of sample data has M dimensions;

[0025] For the target dimension among the M dimensions, K independent variable dimensions that are correlated with the target dimension are selected from the M dimensions based on the correlation coefficients of the initial n sets of sample data; the target dimension can be any one of the M dimensions.

[0026] The target model is determined based on n sets of target sample data distributed across the target dimension and the K independent variable dimensions; the target model is used to characterize the relationship satisfied between the target dimension and the K independent variable dimensions.

[0027] The initial n sets of sample data are distributed across M dimensions, but not all of these M dimensions are necessarily correlated. Therefore, it is necessary to select the dimensions that are correlated. For any target dimension, based on the correlation coefficients of the initial n sets of sample data, K independent variable dimensions with correlations are selected to obtain the corresponding target dimension. In this way, each dimension can be used as a target dimension, and each target dimension and its independent variable dimensions can correspond to a target model. This approach considers a wider range of scenarios and situations, increasing the accuracy of the determined target model.

[0028] In some embodiments, determining the target model based on n target sets of sample data distributed across the target dimension and the K independent variable dimensions includes:

[0029] Obtain n sets of target sample data distributed across the target dimension and the K independent variable dimensions;

[0030] The influence of the candidate sample data on the accuracy of the target model is determined based on the target n sets of sample data and n-1 sets of sample data excluding the candidate sample data; the candidate sample data is any set of sample data from the target n sets of sample data.

[0031] Based on the influence of the n candidate groups of sample data, remove w groups of strongly influential data points from the target n groups of sample data; 1 ≤ w < n;

[0032] The target model is determined based on the retained sample data.

[0033] Based on the target n sets of sample data and n-1 sets of sample data excluding the candidate sets, the influence of the candidate sets of sample data on the accuracy of the target model is determined. Based on this influence, w sets of strongly influencing data points are removed from the target n sets of sample data. The target model is then determined based on the retained sample data. In this way, sample data that significantly impacts the accuracy of the target model is removed, minimizing their influence on the final target model and improving its accuracy. Anomaly detection is then performed on the data to be tested based on the more accurate target model, further improving the accuracy of anomaly detection.

[0034] In some embodiments, determining the influence of the candidate group sample data on the accuracy of the target model based on the target n groups of sample data and n-1 groups of sample data excluding the candidate group sample data includes:

[0035] The first fitting coefficients of the first fitting model are obtained by fitting the target n sets of sample data;

[0036] The second fitting coefficients of the second fitting model are obtained by fitting the n-1 groups of sample data other than the candidate group sample data.

[0037] The influence degree is determined based on the first fitting coefficient, the second fitting coefficient, the number of independent variable dimensions included in the target model, and the mean square error of the first fitting model.

[0038] By using the first fitting coefficient obtained by fitting the data including candidate group samples and the second fitting coefficient obtained by fitting the data excluding candidate group samples, the influence of candidate group samples on the accuracy of the target model is determined, thus improving the accuracy of determining the influence. This leads to a more accurate target model.

[0039] In some embodiments, removing w groups of strongly influential data points from the target n groups of sample data based on the influence of the n candidate groups of sample data includes:

[0040] For any candidate group of sample data, if the influence of the candidate group of sample data on the accuracy of the target model is greater than the first quarter of the F distribution with (p, np-1) degrees of freedom, then the candidate group of sample data is determined to be a strong influence point data; where p is the number of independent variable dimensions included in the target model.

[0041] Remove the data of the strongly influential points from the target n sets of sample data.

[0042] Using the first quarter of the F-distribution with (p, np-1) degrees of freedom to determine the magnitude of influence is more scientific and reasonable, improving the accuracy of identifying data on points of strong influence. This leads to a more accurate target model.

[0043] In some embodiments, after determining the target model based on the retained sample data, the method further includes:

[0044] The test data is input into the target model for testing; the mean absolute error rate of the target model is obtained.

[0045] The goodness-of-fit parameter for fitting the target model to the retained sample data and the mean absolute error rate are determined to satisfy preset thresholds.

[0046] Instead of directly using all target models corresponding to each target dimension for anomaly detection, a further selection process is performed among the target models. Test data is input into any target model for testing, and the average absolute error rate (ARR) of the target model is obtained. If both the model's fit parameter and the ARR meet preset thresholds, it indicates that the target model has high fitting accuracy and can be used for subsequent anomaly detection. This improves the accuracy of anomaly detection.

[0047] Secondly, embodiments of the present invention also provide an abnormal data detection device, comprising:

[0048] Processing unit, used for:

[0049] W data points to be detected, distributed across W dimensions, are input into each target model corresponding to each target dimension. The target model corresponding to any target dimension is obtained by removing sample data after removing data with strong influence points. Data with strong influence points refers to sample data whose influence on the accuracy of the target model does not meet the preset conditions.

[0050] For any target model, if it is determined that W data to be detected do not meet the target model, then each dimension contained in the target model is determined as an abnormal dimension.

[0051] For any anomalous dimension, determine the probability that the anomalous dimension is identified as an anomalous dimension in each target model;

[0052] Based on the anomaly probability, determine whether the data to be detected corresponding to the anomaly dimension is an anomaly.

[0053] In some embodiments, the processing unit is further configured to:

[0054] For any abnormal dimension, obtain the historical data corresponding to that abnormal dimension;

[0055] By clustering the data to be detected corresponding to the anomaly dimension and the historical data, the anomaly score of the data to be detected corresponding to the anomaly dimension is determined.

[0056] The processing unit is specifically used for:

[0057] Based on the anomaly probability and the anomaly score, it is determined whether the data to be detected corresponding to the anomaly dimension is anomaly data.

[0058] In some embodiments, the processing unit is specifically used for:

[0059] Construct an isolated binary tree for the data to be detected corresponding to the anomaly dimension and for each historical data;

[0060] Calculate the anomaly score of the data to be detected corresponding to the anomaly dimension in the isolated binary tree.

[0061] In some embodiments, the processing unit is specifically used for:

[0062] Obtain initial n sets of sample data distributed across M dimensions; where each set of sample data has M dimensions;

[0063] For the target dimension among the M dimensions, K independent variable dimensions that are correlated with the target dimension are selected from the M dimensions based on the correlation coefficients of the initial n sets of sample data; the target dimension can be any one of the M dimensions.

[0064] The target model is determined based on n sets of target sample data distributed across the target dimension and the K independent variable dimensions; the target model is used to characterize the relationship satisfied between the target dimension and the K independent variable dimensions.

[0065] In some embodiments, the processing unit is specifically used for:

[0066] Obtain n sets of target sample data distributed across the target dimension and the K independent variable dimensions;

[0067] The influence of the candidate sample data on the accuracy of the target model is determined based on the target n sets of sample data and n-1 sets of sample data excluding the candidate sample data; the candidate sample data is any set of sample data from the target n sets of sample data.

[0068] Based on the influence of the n candidate groups of sample data, remove w groups of strongly influential data points from the target n groups of sample data; 1 ≤ w < n;

[0069] The target model is determined based on the retained sample data.

[0070] In some embodiments, the processing unit is specifically used for:

[0071] The first fitting coefficients of the first fitting model are obtained by fitting the target n sets of sample data;

[0072] The second fitting coefficients of the second fitting model are obtained by fitting the n-1 groups of sample data other than the candidate group sample data.

[0073] The influence degree is determined based on the first fitting coefficient, the second fitting coefficient, the number of independent variable dimensions included in the target model, and the mean square error of the first fitting model.

[0074] In some embodiments, the processing unit is specifically used for:

[0075] For any candidate group of sample data, if the influence of the candidate group of sample data on the accuracy of the target model is greater than the first quarter of the F distribution with (p, np-1) degrees of freedom, then the candidate group of sample data is determined to be a strong influence point data; where p is the number of independent variable dimensions included in the target model.

[0076] Remove the data of the strongly influential points from the target n sets of sample data.

[0077] In some embodiments, the processing unit is further configured to:

[0078] The test data is input into the target model for testing; the mean absolute error rate of the target model is obtained.

[0079] The goodness-of-fit parameter for fitting the target model to the retained sample data and the mean absolute error rate are determined to satisfy preset thresholds.

[0080] Thirdly, embodiments of the present invention also provide a computing device, comprising:

[0081] Memory, used to store computer programs;

[0082] The processor is configured to invoke a computer program stored in the memory and execute the abnormal data detection method listed in any of the above methods according to the obtained program.

[0083] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer-executable program for causing a computer to perform any of the abnormal data detection methods listed above. Attached Figure Description

[0084] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0085] Figure 1 This is a schematic diagram illustrating a method for determining a target model based on n sets of sample data, provided by an embodiment of the present invention.

[0086] Figure 2 A flowchart illustrating a method for determining a target model provided in an embodiment of the present invention;

[0087] Figure 3 A schematic diagram of a fitted straight line obtained by fitting using the least squares method, provided for an embodiment of the present invention;

[0088] Figure 4 A detailed schematic diagram illustrating the determination of a target model is provided for an embodiment of the present invention;

[0089] Figure 5 A schematic diagram illustrating a possible abnormal data detection method provided in an embodiment of the present invention;

[0090] Figure 6 A schematic diagram of an isolated binary tree constructed according to an embodiment of the present invention;

[0091] Figure 7 This is a schematic diagram of the structure of an abnormal data detection device provided in an embodiment of the present invention;

[0092] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0093] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0094] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation on its own.

[0095] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0096] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms can be used interchangeably where appropriate, for example, in situations where implementation is possible in a sequence other than those given in the embodiments illustrated or described in this application.

[0097] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0098] To better explain this application, the technical terms or terms involved in this application are explained as follows.

[0099] 1. Multiple Linear Regression: In regression analysis, if there are two or more independent variables, it is called multiple regression. In fact, a phenomenon is often related to multiple factors. Predicting or estimating the dependent variable by the optimal combination of multiple independent variables is more effective and more realistic than predicting or estimating it using only one independent variable. Therefore, multiple linear regression is more practical than univariate linear regression.

[0100] 2. Ordinary Least Squares (OLS): This is a mathematical optimization modeling method. It finds the best function match for data by minimizing the sum of squared errors. The least squares method can be used to easily obtain unknown data while minimizing the sum of squared errors between the obtained data and the actual data.

[0101] 3. Degrees of freedom: When using a sample statistic to estimate a population parameter, the number of independent or freely variable data points in the sample is called the degrees of freedom of that statistic. Generally, the degrees of freedom equal the number of independent variables minus the number of derived variables; for example, variance is defined as the sum of the squares of the sample value minus the mean (a derived variable determined by the sample), so for N random samples, the degrees of freedom are N-1.

[0102] 4. Decision Tree: A decision tree is a predictive model that represents a mapping between object attributes and object values. Each node in the tree represents an object, each branch represents a possible attribute value, and each leaf node corresponds to the value of the object represented by the path from the root node to that leaf node. Decision trees have only a single output; if multiple outputs are desired, separate decision trees can be built to handle different outputs. Decision trees are a frequently used technique in data mining, used for both data analysis and prediction.

[0103] 5. Strong Influence Points: These are data points that have a strong impact on the parameter estimation of a multiple linear regression model. Since multiple linear regression uses the least squares method for parameter estimation, all records are treated equally. When records in the database are far removed from the main body of the multidimensional data, they will cause the fitted model to be biased towards those data points. Identifying strong influence points is another important issue to consider when performing multiple linear regression. Strong influence points are data points that significantly affect the stability and accuracy of parameter estimation. In the regression model dataset, strong influence points refer to those points that have a very large influence and impact on the values ​​of statistical measures.

[0104] Research has revealed that in most financial scenarios, linear correlations exist between data across different dimensions, with very few non-linear relationships. Therefore, based on this characteristic, a method can be designed to automatically and deeply mine models that conform to the relationships between data across different dimensions through analysis of sample data. These models can then be used to detect anomalies in the data to be tested, thereby improving the overall quality of the data.

[0105] To ensure the accuracy of anomaly detection, identifying a model that reflects real-world patterns and is business-interpretive across different data dimensions is crucial. Therefore, improving the accuracy of this model has become a key focus of our research.

[0106] Based on this, embodiments of the present invention provide the following method for determining a target model based on n sets of sample data, such as... Figure 1 As shown, it includes:

[0107] Step 101: Obtain the initial n sets of sample data distributed across M dimensions; where each set of sample data has M dimensions.

[0108] Step 102: For the target dimension among the M dimensions, select K independent variable dimensions that are correlated with the target dimension from the M dimensions based on the correlation coefficients of the initial n sets of sample data; the target dimension is any one of the M dimensions.

[0109] Step 103: Determine the target model based on the target n sets of sample data distributed in the target dimension and the K independent variable dimensions; the target model is used to characterize the relationship satisfied between the target dimension and the K independent variable dimensions.

[0110] In step 101, initial n sets of sample data are obtained, and each set of sample data is distributed in M ​​dimensions.

[0111] This invention does not limit the methods and approaches for obtaining sample data. Data can be automatically read from a database or manually imported. For example, training data in Excel format can be read using the pandas library, or the database can be accessed through other means.

[0112] Optionally, the read sample data can be preprocessed, especially by default filling for missing values. Depending on the data characteristics, methods such as zero-value filling or median filling can be selected. The data can also be cleaned to meet the format requirements of the algorithm training.

[0113] Table 1 shows the possible sample data that could be read.

[0114] Table 1

[0115]

[0116]

[0117]

[0118] Table 1 contains a date column and five dimension columns: Dimension A: Interbank Deposits; Dimension B: Domestic Commercial Banks; Dimension C: Other Domestic Banking Financial Institutions; Dimension D: Other Domestic Financial Institutions; and Dimension E: Interest Receivable. Table 1 contains 61 sets of sample data, covering the period from January 2016 to January 2021.

[0119] There are certain interrelationships among the various dimensions. The goal of this invention is to automatically mine the equations or approximate equations between the dimensions from a large amount of sample data, and to improve the accuracy of these equations through subsequent algorithms. Thus, for any target dimension, a corresponding target model is obtained, and anomaly detection is achieved based on this target model.

[0120] One possible implementation is to use the obtained sample data as initial n sets of sample data for subsequent determination of the target model.

[0121] Another possible implementation involves dividing the obtained sample data into a training set and a test set. The training set is used as the initial n sets of sample data for determining the target model, while the test set is used to test and validate the obtained target model to evaluate its accuracy.

[0122] If the sample data is divided into a training set and a test set, the sample data can be divided according to a certain ratio. This embodiment of the invention does not limit the division ratio, for example, 9:1, 8:2, etc. When dividing the training set and the test set, the sample data can be sorted according to certain rules before division, or the sample data can be divided without sorting; there is no restriction on this, because whether or not the sample data is sorted does not affect the accuracy of the determined target model.

[0123] The following detailed example illustrates how the obtained sample data is divided into training and test sets.

[0124] The sample data was sorted by date and divided into a training set:test set ratio of 9:1, with the first 90% of the sample data used as the training set and the last 10% as the test set. In the example in Table 1, a total of 61 sets of sample data were collected. The first 55 sets of sample data (i.e., from January 2016 to July 2020) were used as the training set to determine the target model; the last 6 sets of sample data (i.e., from August 2020 to January 2021) were used as the test set to validate and test the target model to evaluate its accuracy.

[0125] One possible approach is to use K-fold cross-validation to split the sample data if the available sample data is small. For example, if there are only 10 sets of sample data, the sample can be divided into K parts (e.g., 5 parts), with 2 sets in each part. During the target model determination phase, 4 parts (8 sets) can be randomly selected as the training set, and 1 part (2 sets) as the test set. The regression coefficients of the target model are obtained using the training set. This random sampling process is repeated multiple times to generate multiple regression coefficients, and the weighted average of these coefficients yields the final regression coefficients. This approach compensates for the problem of insufficient training due to limited sample data.

[0126] The following section uses the first 55 sets of sample data (i.e., from January 2016 to July 2020) as the training set to determine the target model, and introduces the method for determining the target model.

[0127] The first 55 sets of sample data are used as the training set, that is, as the initial n sets of sample data distributed across M dimensions. In the example above, the initial n sets of sample data are distributed across 5 dimensions.

[0128] In step 102, since the sample data of the 5 dimensions may not all have a linear regression relationship, and there may only be a linear regression relationship between some of the dimensions, it is necessary to determine the independent variable dimension that is correlated with the target dimension for each of the M dimensions.

[0129] For example, for the five dimensions in Table 1, let each dimension be the target dimension, and select the corresponding independent variable dimension for that target dimension. Then, substitute the data from the target dimension and the data from the independent variable dimensions into the linear regression equation. The linear regression equation is y = θ1x1 + θ2x2 + θ3x3 + ... + θ n x n Where y represents the data corresponding to the target dimension, and x1, x2, ... represent the data corresponding to the respective variable dimensions.

[0130] Taking dimension A as the target dimension as an example, this section introduces the method for selecting independent variable dimensions for the target dimension. In this example, dimension A is the target dimension, and dimensions B, C, D, and E are candidate independent variable dimensions. The next step is to select the independent variable dimension corresponding to dimension A from these candidate independent variable dimensions.

[0131] First, construct an initial matrix of n sample data sets (55×5), containing 55 data sets across 5 dimensions. Move one column of the target dimension (dimension A) to the last column of the matrix. Calculate the correlation coefficient matrix r based on the initial matrix of n sample data sets. The correlation coefficient matrix r is obtained using the covariance formula.

[0132] The specific calculation formula is as follows:

[0133]

[0134] Where X i For any candidate independent variable dimension, the data values ​​are for each month. Y is the average of 55 months of data for this candidate independent variable dimension; i For the target dimension, the monthly data values, This represents the average of 55 months of data for the target dimension. Substituting this data into Formula 1 yields the correlation coefficient between the target dimension and any candidate independent variable dimension. For example, Yi represents the monthly data value of dimension A over 55 months. X1 represents the average value of dimension A over 55 months; X2 represents the monthly data value of dimension B over those 55 months. The average value of dimension B over 55 months is used. Substituting this data into Formula 1, we can obtain the correlation coefficient between dimension A and dimension B. Similarly, we can obtain the correlation coefficients between dimension A and dimension C, dimension A and dimension D, and dimension A and dimension E. These will not be listed individually here.

[0135] For example, the correlation coefficients form the following correlation coefficient matrix r.

[0136] [[1. -0.10800406 0.610583 0.4473144 0.9976391]

[0137] [-0.10800406 1. -0.11538687 -0.07851993 -0.07923952]

[0138] [0.610583 -0.11538687 1. 0.51206121 0.63029953]

[0139] [0.4473144 -0.07851993 0.51206121 1. 0.46870661]

[0140] [0.9976391 -0.07923952 0.63029953 0.46870661 1.]]

[0141] The last column is the target dimension column, i.e., dimension A. According to this column, the correlation coefficient between dimension A and dimension B is 0.9976391; the correlation coefficient between dimension A and dimension C is -0.07923952; the correlation coefficient between dimension A and dimension D is 0.63029953; and the correlation coefficient between dimension A and dimension E is 0.46870661. The closer the absolute value of the correlation coefficient is to 1, the stronger the correlation between the two dimensions.

[0142] Then, the variance contribution value for each candidate independent variable dimension is calculated based on the correlation coefficient matrix r. The formula for the variance contribution value is as follows.

[0143]

[0144] Where `columns` represents the total number of columns in matrix `r`, which is 5 in this example. `r(i, i)` represents the value in the i-th row and i-th column of the correlation coefficient matrix. For example, `r(1, 5)`. 2 / r(1,1)=0.9976391 2 =0.99528377. That is, the variance contribution of dimension B to the target model obtained with dimension A as the target dimension is 0.99528377.

[0145] The final matrix of variance contribution values ​​of dimensions B, C, D, and E to the target model obtained with dimension A as the target dimension is [0.99528377 0.0062789 0.3972775 0.21968589]. The larger the variance contribution value, the more beneficial it is to the target model obtained with dimension A as the target dimension.

[0146] The F-value of the F-distribution corresponding to the maximum variance contribution is calculated using Formula 3. The maximum variance contribution is the variance contribution value corresponding to dimension B.

[0147]

[0148] Here, nos is n, and in is the number of dimensions of the candidate independent variables. In this example, n = 55, and in = 4.

[0149] Substituting the values ​​into the formula, the F-value for dimension B is calculated to be 11184.801222455637. Referring to the F-distribution table, the F-value is converted to a probability distribution p-value of 2.449050249153728e-63. In statistics, a p-value < 0.05 generally indicates that the independent variable is significant and can be included in the regression equation. Therefore, dimension B is initially selected as the independent variable dimension of the target model.

[0150] Then, the correlation coefficient matrix r is transformed using the following method:

[0151] i: current row number; j: current column number; k: index of the factor with the largest variance contribution in v, here the value is 1; the transformation formula is as follows:

[0152] When i! = k and j! = k: the new value of r[i,j] = r[k,j] / r[k,k];

[0153] When i! = k and j! = k: the new value of r[i,j] = r[i,j] - r[i,k] * r[k,j] / r[k,k];

[0154] When i! = k and j = k: the new value of r[i,j] = -r[i,k] / r[k,k];

[0155] In other cases, the new value of r[i,j] is 1 / r[k,k];

[0156] The transformed matrix r is:

[0157] [[1. -0.10800406 0.610583 0.4473144 0.9976391]

[0158] [0.10800406 0.98833512 -0.04944142 -0.03020816 0.02850955]

[0159] [-0.610583 -0.04944142 0.6271884 0.23893864 0.02115805]

[0160] [-0.4473144 -0.03020816 0.23893864 0.79990983 0.02244828]

[0161] [-0.9976391 0.02850955 0.02115805 0.02244828 0.00471623]]

[0162] Then, based on the transformed correlation coefficient matrix r, the steps for calculating the maximum variance contribution are repeated, iteratively selecting new independent variable dimensions. Finally, when the target dimension is dimension A, the obtained independent variable dimensions are dimensions B, C, and D.

[0163] Using the same method, we can obtain the independent variable dimensions when the target dimension is dimension B; the independent variable dimensions when the target dimension is dimension C; and the independent variable dimensions when the target dimension is dimension D. These will not be elaborated upon further here.

[0164] It is worth noting that the number of independent variable dimensions may be the same or different for different target dimensions. For example, when dimension A is the target dimension, there are 3 independent variable dimensions: dimension B, dimension C, and dimension D; when dimension B is the target dimension, there are 2 independent variable dimensions: dimension C and dimension D; and when dimension C is the target dimension, there is 1 independent variable dimension: dimension D.

[0165] In step 103, for any target dimension, the process of determining the target model corresponding to that target dimension is described.

[0166] For example, let's take a target dimension as dimension A and the corresponding independent variable dimensions as dimensions B, C, and D.

[0167] The process of determining the target model is as follows Figure 2 As shown, it includes:

[0168] Step 201: Obtain the target n sets of sample data distributed in the target dimension and the K independent variable dimensions.

[0169] Step 202: Determine the influence of the candidate group sample data on the accuracy of the target model based on the target n groups of sample data and n-1 groups of sample data excluding the candidate group sample data; the candidate group sample data is any one of the target n groups of sample data.

[0170] Step 203: Based on the influence of the n candidate group sample data, remove w groups of strongly influential point data from the target n group sample data; 1≤w<n.

[0171] Step 204: Determine the target model based on the retained sample data.

[0172] In step 201, target n sets of sample data are determined from the initial n sets of sample data. For example, when the target dimension is dimension A, the corresponding independent variable dimensions are dimensions B, C, and D, and K = 3. Therefore, the determined target n sets of sample data consist of 55 sets of sample data distributed across four dimensions: A, B, C, and D.

[0173] The following describes the fitting results obtained if these 55 sets of sample data are used for fitting. The least squares method is used as an example.

[0174] The principle of the least squares method is to calculate regression coefficients to make each data point as close as possible to the fitted straight line. Figure 3 A schematic diagram of one possible fitting scenario is shown. Figure 3 In this model, each point is evenly distributed around the fitted line, and the distance between the actual value of each point and the corresponding predicted value on the line is minimized.

[0175] In this example, the objective equation is: y = θ1x1 + θ2x2 + θ3x3 - θ1x1 / θ2x2 / θ3x3. Here, y is dimension A, x1 is dimension B, x2 is dimension C, and x3 is dimension D. We need to calculate the regression coefficient β, which determines the slope of the straight line. The goal is to make the line fit the 55 sets of sample data as closely as possible, which means minimizing the sum of the distances between all points and the equation's straight line. This sum of distances can be defined using the RSS (Residual Squared Sum).

[0176]

[0177] Where y i This is the actual value. These are the predicted values. While ensuring the minimum RSS, the regression coefficients are obtained by solving the standard equation (Formula 5) using the least squares method.

[0178] β=(X T X) -1 X T Y formula 5

[0179] Substituting the 55 sets of sample data distributed across 4 dimensions into the above formula yields the regression coefficient β. The regression coefficient β is also a matrix. The first fitted model obtained after fitting the data from the 55 sets of samples is: A = 1.0053 × B + 0.25 × C + 0.9828 × D. The goodness of fit R0 of this first fitted model is... 2 The p-value of 0.999 indicates that the first fitting model obtained by fitting the data from these 55 sets of samples has a good fit and can reflect the rules between these 55 sets of samples well.

[0180] Superficially, evaluating the first-fit model based on its goodness of fit and significance level leads to the conclusion that it is reasonable and relatively accurate. However, based on business experience and historical data, the relationship that dimensions A, B, C, and D should satisfy is: A = 1 × B + 1 × C + 1 × D. Therefore, the previously obtained first-fit model does not conform to business experience and historical data, and lacks business interpretability. Using such a first-fit model for anomaly detection will inevitably lead to a decrease in detection accuracy.

[0181] Further analysis revealed that outliers may have appeared in the 55 sets of sample data. The presence of outliers caused the initial fitted model to be inconsistent with business experience and historical data. There are many reasons for the appearance of outliers, such as errors during the collection or input process, or errors and anomalies in the sample data itself.

[0182] The above hypothesis will be verified through leverage ratio analysis.

[0183] Based on the previously obtained first fitted model, the leverage ratio of each group of sample data is analyzed. The leverage ratio reflects the degree of influence of each group of sample data on the regression coefficient of the first fitted model. For multiple linear regression, the standard equation for solving the coefficients using the OLS least squares method can be used to derive the formula for calculating the leverage ratio matrix.

[0184] H = X(X) T X) -1 X T Formula 6

[0185] The H matrix reflects the projection of the actual observed value onto the predicted value for each group of sample data. Essentially, the H matrix transforms the actual observed value into the predicted value. The leverage ratio of the i-th sample data group corresponds to the value of the i-th element on the diagonal of the H matrix. In the example above, we calculated the leverage ratio statistics for 55 groups of sample data, as shown in Table 2.

[0186] Table 2

[0187]

[0188]

[0189]

[0190] It can be observed that the leverage ratios corresponding to the first two sets of sample data are 0.365953 and 0.375185, respectively, which are far greater than the average of twice the leverage ratio statistic. Therefore, it can be determined that the first two sets of sample data are relatively extreme. With the existence of such extreme data, the first fitted model is highly likely to be inconsistent with business experience and historical data.

[0191] However, the leverage ratio analysis method is not accurate enough in detecting outliers in the sample data and lacks universality. Therefore, a method is provided to identify the high-influence data points in the target n sets of sample data. Based on the sample data after removing the high-influence data points, the relationship between the target dimension and the independent variable dimensions is determined, which is more accurate. See steps 202-204 for details.

[0192] In step 202, the influence of the candidate sample data on the accuracy of the target model is determined based on the target n sets of sample data and n-1 sets of sample data excluding the candidate sample data; the candidate sample data is any set of sample data from the target n sets of sample data.

[0193] Iterate through each of the n target sample data sets and calculate the impact of removing that sample data set on the accuracy of the target model. For example, using the first sample data set as a candidate sample data set, determine the impact of the first sample data set on the accuracy of the target model based on the first fitting model fitted by 55 sample data sets and the second model fitted by 54 sample data sets excluding the first sample data set; use the second sample data set as a candidate sample data set, determine the impact of the second sample data set on the accuracy of the target model based on the first fitting model fitted by 55 sample data sets and the second model fitted by 54 sample data sets excluding the second sample data set; use the third sample data set as a candidate sample data set, determine the impact of the third sample data set on the accuracy of the target model based on the first fitting model fitted by 55 sample data sets and the second model fitted by 54 sample data sets excluding the third sample data set, and so on, to obtain the impact of each of the 55 sample data sets on the accuracy of the target model.

[0194] The specific method for calculating the influence of any candidate group sample data is as follows: fit the target n groups of sample data to obtain the first fitting coefficient of the first fitting model; fit the n-1 groups of sample data other than the candidate group sample data to obtain the second fitting coefficient of the second fitting model; determine the influence based on the first fitting coefficient, the second fitting coefficient, the number of independent variable dimensions included in the target model, and the mean square error of the first fitting model.

[0195] The specific formula is as follows:

[0196]

[0197] Where p is the number of dimensions of the independent variables in the model; s is the mean square error of the first fitted model; The regression coefficient matrix is ​​obtained by fitting the target n sets of sample data, i.e., the first fitting coefficient; The regression coefficient matrix after removing the i-th group of sample data is the second fitting coefficient. These are the predicted values ​​obtained by fitting the target n sets of sample data; This is the predicted value after removing the i-th group of sample data. The i-th group of sample data is the candidate group sample data. In this example, p = 3. s is calculated using the following formula:

[0198]

[0199] Where n is the number of sample data sets, and np represents the degrees of freedom of the first fitted model. In this example, n = 55.

[0200] Impact reflects the degree to which each set of sample data affects the accuracy of the target model. In principle, for a normal model, the degree of influence of each set of sample data on the model is similar. The greater the degree of influence, the greater the probability that the set of sample data is abnormal. Table 3 shows a possible impact of each set of sample data.

[0201] Table 3

[0202]

[0203]

[0204]

[0205] Table 3 illustrates the impact of each candidate group sample data after removing the sample data from that candidate group.

[0206] By using the first fitting coefficient obtained by fitting the data including candidate group samples and the second fitting coefficient obtained by fitting the data excluding candidate group samples, the influence of candidate group samples on the accuracy of the target model is determined, thus improving the accuracy of determining the influence. This leads to a more accurate target model.

[0207] In step 203, based on the influence of the n candidate group sample data, w groups of strongly influential point data are removed from the target n group sample data; 1≤w<n.

[0208] Data points with strong influence have a significant impact on the accuracy of the target model and should therefore be removed. This embodiment of the invention does not limit the method for determining data points with strong influence.

[0209] One possible approach is to set the threshold based on the experience and needs of the operations and maintenance personnel. If the accuracy requirement for the target model is high, the threshold for strong influence data can be set higher; if the accuracy requirement for the target model is relatively low, the threshold can be set lower. For example, based on experience, the threshold can be set to 4 / n, where n is the number of target n sets of sample data. In this example, n = 55. If the influence of any candidate set of sample data exceeds the threshold, then that candidate set of sample data is identified as strong influence data and is removed.

[0210] Another possible approach is to use the F-distribution to determine the data points with strong influence. Specifically, for any candidate group of sample data, if the influence of the candidate group of sample data on the accuracy of the target model is greater than the first quarter of the F-distribution with (p, np-1) degrees of freedom, then the candidate group of sample data is determined to be data points with strong influence; where p is the number of dimensions of the independent variables included in the target model; the data points with strong influence are then removed from the target n groups of sample data.

[0211] For example, if p = 3 and n = 55, then the data points with strong influence are determined using the F-distribution with (3, 51) degrees of freedom. For any candidate group sample data in Table 3, the influence value is compared with the value of the first quarter digit of the (3, 51) degrees of freedom F-distribution. If it is greater than that value, it is determined as a data point with strong influence.

[0212] Using the first quarter of the F-distribution with (p, np-1) degrees of freedom to determine the magnitude of influence is more scientific and reasonable, improving the accuracy of identifying data on points of strong influence. This leads to a more accurate target model.

[0213] After identifying the data points with strong influence, these data points are removed. For example, using the first method provided in this embodiment to identify the data points with strong influence, the first group of sample data, the second group of sample data, and the 54th group of sample data were ultimately removed. Upon verification, it was found that the first and second groups of sample data were indeed anomalous samples, but the 54th group of sample data was a sample that actually conformed to the model but exhibited significant data fluctuations. Although the above method cannot precisely remove sample data that only contains anomalies, and may remove a small number of samples without anomalies, such as the 54th group of sample data, removing a small number of samples without anomalies will not have a substantial impact on the target model.

[0214] In step 204, the target model is determined based on the retained sample data.

[0215] For example, after removing 3 sets of data points with strong influence in step 203, the target model is determined based on the remaining 52 sets of sample data. The determined target model is: A = 1×B + 1×C + 1×D + 1.31e-10, where 1.31e-10 is the intercept constant and can be ignored.

[0216] It can be observed that after removing data from high-impact points, the resulting target model aligns with business experience and historical data, and possesses business interpretability.

[0217] The above describes the process of determining the target model when the target dimension is dimension A. When the target dimensions are dimensions B, C, D, and E, the respective target models can be determined by following steps 201-204. Different target models may contain different dimensions. Thus, five corresponding target models are determined for the five dimensions.

[0218] The initial n sets of sample data are distributed across M dimensions, but not all of these M dimensions are necessarily correlated. Therefore, it is necessary to select the dimensions that are correlated. For any target dimension, based on the correlation coefficients of the initial n sets of sample data, K independent variable dimensions with correlations are selected to obtain the corresponding target dimension. In this way, each dimension can be used as a target dimension, and each target dimension and its independent variable dimensions can correspond to a target model. This approach considers a wider range of scenarios and situations, increasing the accuracy of the determined target model.

[0219] Based on the target n sets of sample data and n-1 sets of sample data excluding the candidate sets, the influence of the candidate sets of sample data on the accuracy of the target model is determined. Based on this influence, w sets of strongly influencing data points are removed from the target n sets of sample data. The target model is then determined based on the retained sample data. In this way, sample data that significantly impacts the accuracy of the target model is removed, minimizing their influence on the final target model and improving its accuracy. Anomaly detection is then performed on the data to be tested based on the more accurate target model, further improving the accuracy of anomaly detection.

[0220] In some embodiments, sample data from the test set may also be used to test and validate the target models obtained above.

[0221] For example, in step 101, the n sets of sample data were divided into a training set and a test set. The training set serves as the initial n sets of sample data for determining the target model. The test set is used to test and validate the target model. Alternatively, the test set can be obtained through other means, such as highly accurate sample data provided by operations and maintenance personnel that has been confirmed to be free of anomalies.

[0222] After determining the target model based on the retained sample data, the method further includes: inputting test data into the target model for testing; obtaining the average absolute error rate of the target model; and determining that the fit parameter of the target model to the retained sample data and the average absolute error rate respectively meet preset thresholds.

[0223] For example, after step 204, five target models corresponding to the five dimensions are obtained. The fit parameter for each target model can also be obtained to characterize the goodness of fit. The six test data points from the test set are input into the first target model, and the mean absolute error rate is calculated using the following formula:

[0224]

[0225] Where y represents the actual values ​​of the six test data points. This represents the predicted value obtained based on the target model. Thus, the mean absolute error rate of the first target model is obtained.

[0226] The same method can be used to obtain the average absolute error rate of each target model.

[0227] One possible approach is to filter the target models based on their mean absolute error rate and goodness-of-fit parameters. For example, three target models could be selected for subsequent detection of outliers on the target data.

[0228] Another possible approach is to score each target model based on its mean absolute error rate (MAR) and goodness-of-fit parameter, determining absolute equality that meets a first preset condition and approximate equality that meets a second preset condition. When using these target models for anomaly detection, different weights can be assigned to the scores of the absolute equality and approximate equality. For example, a target model with an MAR less than 0.01 and a goodness-of-fit parameter greater than 0.999 can be identified as an absolute equality; a target model with an MAR greater than or equal to 0.01 and less than 0.1 and a goodness-of-fit parameter greater than 0.9 can be identified as an approximate equality.

[0229] Therefore, instead of directly using all target models corresponding to each target dimension for anomaly detection, a further selection process is performed among the target models. Test data is input into any target model for testing, and the average absolute error rate (ARR) of the target model is obtained. If the fit parameter and the ARR of the target model both meet preset thresholds, it indicates that the target model has high fitting accuracy and can be used for subsequent anomaly detection. This improves the accuracy of anomaly detection.

[0230] To better explain the embodiments of the present invention, the process of determining the target model described above will be described below in a specific implementation scenario. Figure 4 A detailed flowchart for determining the target model is shown.

[0231] Step 401: Read sample data.

[0232] Step 402: Preprocess the sample data.

[0233] Step 403: Divide the sample data into a training set and a test set.

[0234] The training set serves as the initial n sets of sample data used to determine the target model.

[0235] Step 404: For any target dimension, select K independent variable dimensions that are correlated with the target dimension from M dimensions based on the correlation coefficients of the initial n sets of sample data.

[0236] Step 405: Obtain the target n sets of sample data distributed across the target dimension and the K independent variable dimensions.

[0237] Step 406: For any candidate group sample data, determine the influence of the candidate group sample data on the accuracy of the target model based on the target n group sample data and the n-1 group sample data other than the candidate group sample data.

[0238] Step 407: Determine if the influence is greater than 4 / n. If it is, proceed to step 408; otherwise, proceed to step 409.

[0239] Step 408: Remove the sample data of this candidate group.

[0240] Step 409: Retain the sample data of this candidate group.

[0241] Step 410: Determine the target model corresponding to the target dimension based on the retained sample data.

[0242] Repeating steps 404-410 yields multiple target models corresponding to multiple dimensions.

[0243] Step 411: Use the test set data to evaluate each target model and obtain the absolute equation and approximate equation.

[0244] The following section introduces how the obtained target models are used to detect abnormal data in the data to be detected.

[0245] Figure 5 One possible method for detecting outlier data is shown, including:

[0246] Step 501: Input W data points to be detected, distributed across W dimensions, into the target models corresponding to each target dimension.

[0247] Step 502: For any target model, if it is determined that W data to be detected do not meet the target model, then each dimension contained in the target model is determined as an abnormal dimension.

[0248] Step 503: For any abnormal dimension, determine the probability that the abnormal dimension is identified as an abnormal dimension in each target model.

[0249] Step 504: Determine whether the data to be detected corresponding to the abnormal dimension is abnormal data based on the abnormal probability.

[0250] In step 501, W data points distributed across W dimensions are input into the target models corresponding to each target dimension. This embodiment of the invention does not limit the number of dimensions to W.

[0251] One possible approach is that the W dimensions include at least the M dimensions that define the target model. For example, the W dimensions could be identical to the M dimensions that define the target model, distributed across the five dimensions A, B, C, D, and E. Alternatively, the W dimensions could include other dimensions besides the M dimensions that define the target model.

[0252] Another possible approach is that the W dimensions may include a subset of the M dimensions. For example, in a given set of target models, some models may be removed during the test set, resulting in the remaining target models not containing certain dimensions. Therefore, the dimensions of the data to be tested do not necessarily need to include those dimensions. For instance, among the five target models corresponding to the previously determined five dimensions, only three target models meet the test set requirements. Therefore, only these three target models are used for testing the data to be detected. These three target models only include dimensions A, B, C, and D. Thus, the W dimensions of the data distribution to be detected could also be only dimensions A, B, C, and D, excluding dimension E.

[0253] An example of possible data to be detected is shown in Table 4.

[0254] Table 4

[0255]

[0256] The W data points to be detected are input into each target model, for example, into three target models. The three target models are:

[0257] Target Model 1: A = B + C + D;

[0258] Target Model 2: B = C + D;

[0259] Target Model 3: C = A + B.

[0260] In step 502, for any target model, if it is determined that W data to be detected do not meet the target model, then each dimension contained in the target model is determined as an abnormal dimension.

[0261] For example, for target model 1, if the data to be tested shown in Table 4 is input into target model 1 and the mean absolute error rate does not meet the preset threshold, then dimensions A, B, C, and D contained in target model 1 are identified as abnormal dimensions. For target model 2, if the data to be tested shown in Table 4 is input into target model 2 and the mean absolute error rate meets the preset threshold, then no operation is performed. For target model 3, if the data to be tested shown in Table 4 is input into target model 3 and the mean absolute error rate does not meet the preset threshold, then dimensions A, B, and C contained in target model 3 are identified as abnormal dimensions.

[0262] In summary, dimensions A, B, C, and D were identified as anomalous dimensions.

[0263] In step 503, for any anomalous dimension, the probability that the anomalous dimension is identified as an anomalous dimension in each target model is determined.

[0264] One possible approach is to determine the ratio of the number of times the anomalous dimension appears in the target model that has been identified as an anomalous dimension to the number of times the anomalous dimension appears in each target model, and then determine the anomalous probability of the anomalous dimension based on the ratio.

[0265] For example, regarding dimension A, it appears twice in each target model, and is identified as an anomalous dimension in two target models (target model 1 and target model 3). Therefore, the anomalous probability of dimension A is 2 / 2 = 1. Regarding dimension B, it appears three times in each target model, and is identified as an anomalous dimension in two target models (target model 1 and target model 3). Therefore, the anomalous probability of dimension B is 2 / 3. Regarding dimension C, it appears three times in each target model, and is identified as an anomalous dimension in two target models (target model 1 and target model 3). Therefore, the anomalous probability of dimension C is 2 / 3. Regarding dimension D, it appears twice in each target model, and is identified as an anomalous dimension in one target model (target model 1). Therefore, the anomalous probability of dimension D is 1 / 2.

[0266] One possible approach is to assign different weights to the probability values ​​obtained from absolute equality and approximate equality, thereby more accurately locating outlier data. For example, if target model 1 is an absolute equality, then in cases where target model 1 is identified as an outlier dimension, the frequency of that outlier dimension appearing in the target model that is identified as an outlier dimension is multiplied by 1; if target model 3 is an approximate equality, then in cases where target model 3 is identified as an outlier dimension, the frequency of that outlier dimension appearing in the target model that is identified as an outlier dimension is multiplied by 0.8.

[0267] In step 504, it is determined whether the data to be detected corresponding to the abnormal dimension is abnormal data based on the abnormal probability.

[0268] The anomaly probability of any anomaly dimension is compared with a preset threshold. If it is greater than the preset threshold, the data to be detected corresponding to that anomaly dimension is determined to be anomalous; if it is not greater than the preset threshold, the data to be detected corresponding to that anomaly dimension is determined to be non-anomaly. The preset threshold can be set based on the experience and needs of those skilled in the art. No restrictions are imposed. Alternatively, the data to be detected corresponding to the top N anomaly dimensions with the highest anomaly probabilities can be determined as anomalous data.

[0269] For example, the data to be detected corresponding to dimension A, 100.5, is identified as abnormal data.

[0270] By inputting W data points distributed across W dimensions into the corresponding target models for each target dimension, the anomalous dimensions are identified. Then, the probability of each anomalous dimension being identified as anomalous in each target model is determined. Based on this probability, it is determined whether the data points corresponding to those anomalous dimensions are indeed anomalous. This not only detects anomalous data among the W data points but also accurately pinpoints which dimension of the data is anomalous. This achieves automated and accurate anomalous data location without requiring manual review.

[0271] This invention also provides another method for detecting abnormal data, which, after determining each dimension in the target model as an abnormal dimension, further includes: for any abnormal dimension, obtaining each historical data corresponding to the abnormal dimension; determining the abnormal score of the data to be detected corresponding to the abnormal dimension by clustering the data to be detected corresponding to the abnormal dimension and the historical data; and determining whether the data to be detected corresponding to the abnormal dimension is abnormal data based on the abnormal probability, including: determining whether the data to be detected corresponding to the abnormal dimension is abnormal data based on the abnormal probability and the abnormal score.

[0272] The above method is for any anomaly dimension, such as the data to be detected in dimension A. At the same time, historical data of dimension A is obtained, and the data to be detected and historical data of dimension A are clustered to obtain the anomaly score of dimension A.

[0273] By acquiring historical data for the anomaly dimension, the data to be detected corresponding to the anomaly dimension and each historical data point are clustered to obtain anomaly scores for the data to be detected corresponding to the anomaly dimension. Combining the anomaly probability and the anomaly score determines whether the data to be detected corresponding to that anomaly dimension is indeed anomaly. This combination of two methods considers both the probability of the anomaly dimension being identified as an anomaly dimension and the historical data for that anomaly dimension, thus increasing the accuracy of anomaly detection.

[0274] The embodiments of the present invention do not impose specific limitations on the method for obtaining abnormal scores through clustering.

[0275] One possible approach is to use k-means clustering, determining the anomaly score by the distance between the data to be detected in any anomaly dimension and all historical data. For example, if the distance between the data to be detected in dimension A and all historical data is large, the similarity is low, and the anomaly score is small.

[0276] Another possible approach is to obtain the anomaly score for any anomaly dimension by constructing an isolated binary tree. Specifically, this involves constructing an isolated binary tree for the data to be detected corresponding to the anomaly dimension and the historical data; and calculating the anomaly score of the data to be detected corresponding to the anomaly dimension in the isolated binary tree.

[0277] For example, for dimension A, the historical data for dimension A are: 19.49, 20.23, 25.34, 49.12, 36.66. Using ensemble learning, the data to be tested and the historical data (19.49, 20.23, 25.34, 49.12, 36.66, 100.5) are iterated N times (e.g., 100). Each iteration constructs an isolated binary tree. Based on the decision tree algorithm, the data to be tested and the historical data are randomly cut, and each cut produces an independent leaf node. This process continues to create new leaf nodes until the tree reaches a specified height or can no longer be cut, at which point the algorithm terminates.

[0278] The specific steps for constructing an isolated binary tree are as follows: (1) First, randomly select a split point between the minimum and maximum values ​​(19.49 and 100.5) of all sample data, assuming the random value is 60.2. (2) Place data nodes in the sample that are greater than the split point value of 60.2 on the right branch of the tree, and place data nodes that are less than or equal to 60.2 on the left branch of the tree. (3) Repeat steps (1) and (2) based on the branches until all data nodes are randomly split to form isolated leaf nodes or the tree reaches the specified height.

[0279] The first isolated tree is randomly constructed following the steps described above. Figure 6 As shown, the 5 random split points are (60.2, 34, 42.2, 22.5, 20).

[0280] Calculate the PathLength of each leaf node as h(x):

[0281] h(x) = e + c(T.size) Formula 10

[0282] Where e is the number of edges traversed by a leaf node from the root node to the leaf node, i.e., the number of splits, T.size represents the number of samples that share a leaf node with sample x, and C(T.size) can be seen as a correction value, representing the average path length of a binary tree constructed from T.size samples.

[0283]

[0284] Where o is Euler's constant 0.5772156649; taking the calculation of the PathLength of the sample 100.5 as an example, e is the number of edges from the root node to the node 100.5 = 1, T.size = 1, so the PathLength of the point 100.5 = 1 + c(1), which can be substituted into the formula for calculating c(n) above.

[0285] To ensure randomness and thus accuracy of anomaly scores, 100 random isolated binary trees are constructed by iterating N times (default 100) using the method described above. The PathLength (h(x)) of the node 100.5 is calculated for each tree. Then, the anomaly score for the isolated forest is calculated using the following formula:

[0286]

[0287] Where n is the number of samples, F(h(x)) is the average PathLength of the sample on 100 isolated trees, and c(n) is the average path length of the tree constructed from n samples. In the example above, n is the number of samples, which is 6. The result of c(6) is calculated using the formula for calculating c(n) above. Finally, the anomaly score of the node corresponding to dimension A, i.e., sample data 100.5, is calculated.

[0288] Then, based on the anomaly probability corresponding to the anomaly dimension and the anomaly score, it is determined whether the data to be detected corresponding to the anomaly dimension is anomaly data.

[0289] By constructing an isolated binary tree of the data to be detected corresponding to the anomaly dimension and each historical data, the accuracy of identifying anomaly data is improved.

[0290] Based on the same technological concept Figure 7 An exemplary embodiment of the present invention provides the structure of an abnormal data detection device, which can perform an abnormal data detection process.

[0291] like Figure 7 As shown, the device specifically includes:

[0292] Processing unit 701 is used for:

[0293] W data points to be detected, distributed across W dimensions, are input into each target model corresponding to each target dimension. The target model corresponding to any target dimension is obtained by removing sample data after removing data with strong influence points. Data with strong influence points refers to sample data whose influence on the accuracy of the target model does not meet the preset conditions.

[0294] For any target model, if it is determined that W data to be detected do not meet the target model, then each dimension contained in the target model is determined as an abnormal dimension.

[0295] For any anomalous dimension, determine the probability that the anomalous dimension is identified as an anomalous dimension in each target model;

[0296] Based on the anomaly probability, determine whether the data to be detected corresponding to the anomaly dimension is an anomaly.

[0297] In some embodiments, the processing unit 701 is further configured to:

[0298] For any abnormal dimension, obtain the historical data corresponding to that abnormal dimension;

[0299] By clustering the data to be detected corresponding to the anomaly dimension and the historical data, the anomaly score of the data to be detected corresponding to the anomaly dimension is determined.

[0300] The processing unit 701 is specifically used for:

[0301] Based on the anomaly probability and the anomaly score, it is determined whether the data to be detected corresponding to the anomaly dimension is anomaly data.

[0302] In some embodiments, the processing unit 701 is specifically used for:

[0303] Construct an isolated binary tree for the data to be detected corresponding to the anomaly dimension and for each historical data;

[0304] Calculate the anomaly score of the data to be detected corresponding to the anomaly dimension in the isolated binary tree.

[0305] In some embodiments, the processing unit 701 is specifically used for:

[0306] Obtain initial n sets of sample data distributed across M dimensions; where each set of sample data has M dimensions;

[0307] For the target dimension among the M dimensions, K independent variable dimensions that are correlated with the target dimension are selected from the M dimensions based on the correlation coefficients of the initial n sets of sample data; the target dimension can be any one of the M dimensions.

[0308] The target model is determined based on n sets of target sample data distributed across the target dimension and the K independent variable dimensions; the target model is used to characterize the relationship satisfied between the target dimension and the K independent variable dimensions.

[0309] In some embodiments, the processing unit 701 is specifically used for:

[0310] Obtain n sets of target sample data distributed across the target dimension and the K independent variable dimensions;

[0311] The influence of the candidate sample data on the accuracy of the target model is determined based on the target n sets of sample data and n-1 sets of sample data excluding the candidate sample data; the candidate sample data is any set of sample data from the target n sets of sample data.

[0312] Based on the influence of the n candidate groups of sample data, remove w groups of strongly influential data points from the target n groups of sample data; 1 ≤ w < n;

[0313] The target model is determined based on the retained sample data.

[0314] In some embodiments, the processing unit 701 is specifically used for:

[0315] The first fitting coefficients of the first fitting model are obtained by fitting the target n sets of sample data;

[0316] The second fitting coefficients of the second fitting model are obtained by fitting the n-1 groups of sample data other than the candidate group sample data.

[0317] The influence degree is determined based on the first fitting coefficient, the second fitting coefficient, the number of independent variable dimensions included in the target model, and the mean square error of the first fitting model.

[0318] In some embodiments, the processing unit 701 is specifically used for:

[0319] For any candidate group of sample data, if the influence of the candidate group of sample data on the accuracy of the target model is greater than the first quarter of the F distribution with (p, np-1) degrees of freedom, then the candidate group of sample data is determined to be a strong influence point data; where p is the number of independent variable dimensions included in the target model.

[0320] Remove the data of the strongly influential points from the target n sets of sample data.

[0321] In some embodiments, the processing unit 701 is further configured to:

[0322] The test data is input into the target model for testing; the mean absolute error rate of the target model is obtained.

[0323] The goodness-of-fit parameter for fitting the target model to the retained sample data and the mean absolute error rate are determined to satisfy preset thresholds.

[0324] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 8 As shown, it includes at least one processor 801 and a memory 802 connected to at least one processor. In this embodiment, the specific connection medium between the processor 801 and the memory 802 is not limited. Figure 8 Taking the connection between the processor 801 and the memory 802 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0325] In this embodiment of the application, the memory 802 stores instructions that can be executed by at least one processor 801. By executing the instructions stored in the memory 802, at least one processor 801 can perform the steps of the above-described abnormal data detection method.

[0326] The processor 801 is the control center of the computer device, capable of connecting to various parts of the device via various interfaces and lines. It performs abnormal data detection by running or executing instructions stored in the memory 802 and accessing data stored in the memory 802. In some embodiments, the processor 801 may include one or more processing units. The processor 801 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the processor 801. In some embodiments, the processor 801 and the memory 802 may be implemented on the same chip; in some embodiments, they may be implemented on separate chips.

[0327] The processor 801 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0328] Memory 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 802 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 802 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 802 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0329] Based on the same technical concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer-executable program, the computer-executable program being used to cause a computer to perform the abnormal data detection method listed in any of the above methods.

[0330] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0331] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0332] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0333] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0334] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An abnormal data detection method, characterized in that, include: W data points to be detected, distributed across W dimensions, are input into the target models corresponding to each target dimension. The target model corresponding to any target dimension is obtained by removing sample data after removing data from highly influential points; the data from highly influential points refers to sample data whose influence on the accuracy of the target model does not meet the preset conditions. For any target model, if it is determined that W data to be detected do not meet the target model, then each dimension contained in the target model is determined as an abnormal dimension. For any anomalous dimension, determine the probability that the anomalous dimension is identified as an anomalous dimension in each target model; and determine whether the data to be detected corresponding to the anomalous dimension is anomalous data based on the anomalous probability. After determining each dimension contained in the target model as an anomalous dimension, the process further includes: For any anomaly dimension, obtain the historical data corresponding to the anomaly dimension; by clustering the data to be detected corresponding to the anomaly dimension and the historical data, determine the anomaly score of the data to be detected corresponding to the anomaly dimension. Determining whether the data to be detected corresponding to the anomaly dimension is anomaly data based on the anomaly probability includes: Based on the anomaly probability and the anomaly score, it is determined whether the data to be detected corresponding to the anomaly dimension is anomaly data.

2. The method as described in claim 1, characterized in that, By clustering the data to be detected corresponding to the anomaly dimension and the historical data, the anomaly score of the data to be detected corresponding to the anomaly dimension is determined, including: Construct an isolated binary tree for the data to be detected corresponding to the anomaly dimension and for each historical data; Calculate the anomaly score of the data to be detected corresponding to the anomaly dimension in the isolated binary tree.

3. The method as described in claim 1, characterized in that, The target model corresponding to any target dimension is determined in the following ways: Obtain initial n sets of sample data distributed across M dimensions; where each set of sample data has M dimensions; For the target dimension among the M dimensions, K independent variable dimensions that are correlated with the target dimension are selected from the M dimensions based on the correlation coefficients of the initial n sets of sample data; the target dimension can be any one of the M dimensions. The target model is determined based on n sets of target sample data distributed across the target dimension and the K independent variable dimensions; the target model is used to characterize the relationship satisfied between the target dimension and the K independent variable dimensions.

4. The method as described in claim 3, characterized in that, The target model is determined based on n sets of target sample data distributed across the target dimension and the K independent variable dimensions, including: Obtain n sets of target sample data distributed across the target dimension and the K independent variable dimensions; The influence of the candidate sample data on the accuracy of the target model is determined based on the target n sets of sample data and n-1 sets of sample data excluding the candidate sample data; the candidate sample data is any set of sample data from the target n sets of sample data. Based on the influence of the n candidate groups of sample data, remove w groups of strongly influential data points from the target n groups of sample data; 1 ≤ w < n; The target model is determined based on the retained sample data.

5. The method as described in claim 4, characterized in that, Determining the influence of the candidate sample data on the accuracy of the target model based on the target n sets of sample data and n-1 sets of sample data excluding the candidate sample data includes: The first fitting coefficients of the first fitting model are obtained by fitting the target n sets of sample data; The second fitting coefficients of the second fitting model are obtained by fitting the n-1 groups of sample data other than the candidate group sample data. The influence degree is determined based on the first fitting coefficient, the second fitting coefficient, the number of independent variable dimensions included in the target model, and the mean square error of the first fitting model.

6. The method as described in claim 4, characterized in that, Based on the influence of the n candidate groups of sample data, w groups of strongly influential data points are removed from the target n groups of sample data, including: For any candidate group of sample data, if the influence of the candidate group of sample data on the accuracy of the target model is greater than the first quarter of the F distribution with (p, np-1) degrees of freedom, then the candidate group of sample data is determined to be a strong influence point data; where p is the number of independent variable dimensions included in the target model. Remove the data of the strongly influential points from the target n sets of sample data.

7. The method as described in claim 4, characterized in that, After determining the target model based on the retained sample data, the process also includes: The test data is input into the target model for testing; the mean absolute error rate of the target model is obtained. The goodness-of-fit parameter for fitting the target model to the retained sample data and the mean absolute error rate are determined to satisfy preset thresholds.

8. An abnormal data detection device, characterized in that, include: Processing unit, used for: W data points distributed across W dimensions are input into the target models corresponding to each target dimension. The target model for any target dimension is obtained by removing data from samples after removing data with strong influence points. Data with strong influence points refers to sample data whose impact on the accuracy of the target model does not meet preset conditions. For any target model, if it is determined that W data points do not satisfy the target model, then each dimension contained in the target model is identified as an anomalous dimension. For any anomalous dimension, the probability of that anomalous dimension being identified as an anomalous dimension in each target model is determined. Based on the probability of anomalous dimension, it is determined whether the data points corresponding to the anomalous dimension are anomalous data. The processing unit is specifically configured to: acquire historical data corresponding to any abnormal dimension; cluster the data to be detected corresponding to the abnormal dimension and the historical data to determine the abnormal score of the data to be detected corresponding to the abnormal dimension; and determine whether the data to be detected corresponding to the abnormal dimension is abnormal data based on the abnormal probability and the abnormal score.

9. A computing device, characterized in that, include: Memory, used to store computer programs; A processor is configured to invoke a computer program stored in the memory and execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer-executable program for causing a computer to perform the method according to any one of claims 1 to 7.

Images