Outlier detection method and device, computer equipment and readable storage medium

Abstract

The invention relates to an outlier detection method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a to-be-detected data set and a sample data set corresponding to the to-be-detected data set; extracting attribute features of the to-be-detected data set, and screening each initial target detection algorithm from each detection algorithm based on the attribute features; the attribute features are detection constraint conditions of outlier detection results; training each initial target detection algorithm according to the sample data set to obtain each target detection algorithm; each target detection algorithm is used for determining an outlier detection result of the to-be-detected data set. By adopting the method, the accuracy of outlier detection results can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data detection, and in particular to an outlier detection method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Outlier detection (also known as anomaly detection) is an important data mining method. Outlier detection methods can be used to find outliers (outliers) that are different from the "main data distribution". For example, a subset of credit card transaction data that is a fraud case can be found from a credit card transaction data set, and a subset of network data that is an intrusion can be found from a network data set.

[0003] In the current outlier detection method, a target detection algorithm is manually set, and each to-be-detected data subset in the to-be-detected data set is detected according to the target detection algorithm to obtain the outlier detection result of each to-be-detected data subset.

[0004] However, the current outlier detection method manually selects a single target detection algorithm based on experience, which will cause the target detection algorithm to be unable to adapt to the data set to be detected, thereby resulting in low accuracy of the outlier detection results of the data subset to be detected. Summary of the invention

[0005] Based on this, it is necessary to provide an outlier detection method, apparatus, computer device, computer readable storage medium and computer program product to address the above technical problems.

[0006] In a first aspect, the present application provides an outlier detection method, comprising:

[0007] Acquire a data set to be detected and a sample data set corresponding to the data set to be detected;

[0008] Extracting attribute features of the data set to be detected, and screening each initial target detection algorithm from each detection algorithm based on the attribute features; the attribute features are detection constraints for outlier detection results;

[0009] Each of the initial target detection algorithms is trained according to the sample data set to obtain each target detection algorithm; each of the target detection algorithms is used to determine the outlier detection result of the data set to be detected.

[0010] In one embodiment, the attribute features include distribution features, data volume features, and dimensional features; and extracting the attribute features of the data set to be detected includes:

[0011] Draw a distribution map of the data set to be detected, and determine the distribution characteristics of the data set to be detected based on the distribution map;

[0012] Determining the dimensional characteristics and data volume characteristics of the data set to be detected based on the data labels in the data set to be detected and the number of data subsets to be detected in the data set to be detected;

[0013] The attribute characteristics of the data set to be detected are constructed according to the distribution characteristics, the data volume characteristics and the dimensional characteristics.

[0014] In one embodiment, the screening of each initial target detection algorithm from each detection algorithm based on the attribute feature includes:

[0015] Selecting a detection algorithm for each target type from among various types of detection algorithms according to the attribute characteristics;

[0016] Determine an initial target detection algorithm in the detection algorithm of each target type, and obtain a parameter set of the initial target detection algorithm;

[0017] The initial object detection algorithm is updated based on the parameter set.

[0018] In one embodiment, the data set to be detected includes each data subset to be detected, and after each initial target detection algorithm is trained according to the sample data set to obtain each target detection algorithm, the method further includes:

[0019] Performing data processing on each of the to-be-detected data subsets based on each of the target detection algorithms to obtain an initial detection score of the detection data subset under each of the target detection algorithms;

[0020] Performing standardization processing on each of the initial detection scores to obtain a detection score;

[0021] The outlier detection result of the data subset to be detected is determined according to each of the detection scores and the comprehensive detection score threshold; the comprehensive detection score threshold is obtained by each of the target detection algorithms performing data processing on the sample data set.

[0022] In one embodiment, the determining the outlier detection result of the to-be-detected data subset according to each of the detection scores and the comprehensive detection score threshold comprises:

[0023] For each of the data subsets to be detected, weighted processing is performed on the detection scores corresponding to the data subsets to be detected according to the weights corresponding to the target detection algorithms to obtain a comprehensive detection score corresponding to the data subsets to be detected;

[0024] Determine whether the comprehensive detection score is greater than a comprehensive detection score threshold, and obtain a determination result;

[0025] The outlier detection result of the to-be-detected data subset is determined according to the judgment result.

[0026] In one embodiment, the sample data set includes sample data subsets, and after the initial target detection algorithms are trained according to the sample data set to obtain the target detection algorithms, the method further includes:

[0027] Performing data processing on each of the sample data subsets based on each of the target detection algorithms to obtain an initial sample detection score for each of the sample data subsets under each of the target detection algorithms;

[0028] The initial sample detection scores are standardized to obtain the detection scores of the samples, and the comprehensive detection score threshold is determined according to the detection scores of the samples.

[0029] In one embodiment, determining the comprehensive detection score threshold according to the detection scores of each sample includes:

[0030] According to the weight of each target detection algorithm, weighted processing is performed on each sample detection score corresponding to each sample data subset to obtain a sample comprehensive detection score corresponding to the sample data subset;

[0031] Sorting the sample comprehensive detection scores corresponding to each of the sample data subsets in order from largest to smallest to obtain a sample comprehensive detection score sequence;

[0032] A comprehensive detection score threshold is determined in the sample comprehensive detection score sequence based on a preset ratio threshold.

[0033] In a second aspect, the present application also provides an outlier detection device, comprising:

[0034] An acquisition module, used to acquire a data set to be detected and a sample data set corresponding to the data set to be detected;

[0035] A screening module, used to extract attribute features of the data set to be detected, and screen each initial target detection algorithm among various detection algorithms based on the attribute features; the attribute features are detection constraints of outlier detection results;

[0036] The training module is used to train each of the initial target detection algorithms according to the sample data set to obtain each target detection algorithm; each of the target detection algorithms is used to determine the outlier detection result of the data set to be detected.

[0037] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0038] Acquire a data set to be detected and a sample data set corresponding to the data set to be detected;

[0039] Extracting attribute features of the data set to be detected, and screening each initial target detection algorithm from each detection algorithm based on the attribute features; the attribute features are detection constraints for outlier detection results;

[0040] Each of the initial target detection algorithms is trained according to the sample data set to obtain each target detection algorithm; each of the target detection algorithms is used to determine the outlier detection result of the data set to be detected.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0042] Acquire a data set to be detected and a sample data set corresponding to the data set to be detected;

[0043] Extracting attribute features of the data set to be detected, and screening each initial target detection algorithm from each detection algorithm based on the attribute features; the attribute features are detection constraints for outlier detection results;

[0044] Each of the initial target detection algorithms is trained according to the sample data set to obtain each target detection algorithm; each of the target detection algorithms is used to determine the outlier detection result of the data set to be detected.

[0045] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0046] Acquire a data set to be detected and a sample data set corresponding to the data set to be detected;

[0047] Extracting attribute features of the data set to be detected, and screening each initial target detection algorithm from each detection algorithm based on the attribute features; the attribute features are detection constraints for outlier detection results;

[0048] Each of the initial target detection algorithms is trained according to the sample data set to obtain each target detection algorithm; each of the target detection algorithms is used to determine the outlier detection result of the data set to be detected.

[0049] The above-mentioned outlier detection method, device, computer equipment, computer-readable storage medium and computer program product obtain a data set to be detected and a sample data set corresponding to the data set to be detected; extract the attribute features of the data set to be detected, and select each initial target detection algorithm in each detection algorithm based on the attribute features; the attribute features are the detection constraints of the outlier detection results; train each initial target detection algorithm according to the sample data set to obtain each target detection algorithm; each target detection algorithm is used to determine the outlier detection result of the data set to be detected. By adopting this method, each initial target detection algorithm is selected in each detection algorithm by the attribute features of the data set to be detected, and the initial target detection algorithm adapted to the data set to be detected is obtained, thereby improving the objectivity of the selected target detection algorithm. Then, each initial target detection algorithm is trained by the sample data subset corresponding to the data subset to be detected, and each target detection algorithm is obtained, thereby improving the accuracy of each target detection algorithm. Furthermore, the outlier detection result of the data set to be detected is determined by each target detection algorithm, thereby avoiding the defects of a single target detection algorithm and improving the accuracy of the outlier detection result. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 1 is a flow chart of an outlier detection method in one embodiment;

[0052] Figure 2 A schematic diagram of a process for extracting attribute features in one embodiment;

[0053] Figure 3 A schematic diagram of a process for selecting an initial target detection algorithm in one embodiment;

[0054] Figure 4 A schematic diagram of a process for determining an outlier detection result in one embodiment;

[0055] Figure 5 A schematic diagram of a process for determining an outlier detection result based on each detection score in one embodiment;

[0056] Figure 6 A schematic diagram of a process for determining a comprehensive detection score threshold in one embodiment;

[0057] Figure 7A schematic diagram of a process for determining a comprehensive detection score threshold based on the detection scores of each sample in one embodiment;

[0058] Figure 8 is a flow chart of an outlier detection method in another embodiment;

[0059] Fig. 9 is a structural block diagram of an outlier detection device in one embodiment;

[0060] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] Outlier detection (also known as outlier detection, anomaly detection) is an important data mining method. Outlier detection methods can be used to find outliers (deviant from the general data distribution) that are different from the "main data distribution". For example, a subset of credit card transaction data that represents fraud cases can be found from a credit card transaction data set. Or, a subset of network data that represents network intrusion can be found from a normal network data set. At the same time, outlier detection methods are used in the preprocessing process of machine learning tasks. Outlier detection methods are used to find outliers in the training data set to prevent training or prediction failures caused by the presence of a small number of outliers.

[0063] In the current outlier detection method, a target detection algorithm is manually set, and each to-be-detected data subset in the to-be-detected data set is detected according to the target detection algorithm to obtain the outlier detection result of each to-be-detected data subset.

[0064] For example, the data set to be detected is a credit card transaction data set. The credit card transaction data subset contains a normal credit card transaction data subset for normal transactions and an abnormal credit card transaction data subset for abnormal transactions. The abnormal credit card transaction data subset is an outlier. Each credit card transaction data subset contains the transaction users, transaction amount, transaction time, and transaction frequency of both parties. The KNN (K-NearestNeighbor) algorithm is manually selected to determine the abnormal credit card transaction data subset in the credit card transaction data set. Specifically, each credit card transaction data subset in the credit card transaction data subset is vectorized to obtain a transaction data vector, and each transaction data vector is clustered to obtain each cluster cluster and scattered points. The credit card transaction data subset represented by the scattered points is determined as an outlier.

[0065] However, the current outlier detection method manually selects a single target detection algorithm based on experience, which will cause the target detection algorithm to be unable to adapt to the data set to be detected, thereby resulting in low accuracy of the outlier detection results of the data subset to be detected.

[0066] If the target detection algorithm is a distribution-based anomaly detection method, the disadvantages of the target detection algorithm are as follows:

[0067] (1) The occurrence of low-probability events in statistical distribution is defined as outliers, which mainly depends on a large amount of data. When the amount of data is small, the accuracy of identification through statistical distribution methods is very low.

[0068] (2) Statistical testing requires that the data in the dataset to be tested obey a certain distribution. However, real data will not conform to a certain distribution, such as a normal distribution. Therefore, the applicability of statistical testing methods will be greatly reduced.

[0069] If the target detection algorithm is an algorithm based on distance, density and clustering, the disadvantages of the target detection algorithm are as follows:

[0070] (1) When the data set to be detected is a high-dimensional data set, the similarity or distance measurement between the data becomes difficult, the accuracy of the clustering results will decrease, and the accuracy of the outlier detection results will be lower.

[0071] (2) Since clustering algorithms are sensitive to noise and outliers, abnormal data in the dataset to be detected will have a negative impact on the clustering results, leading to incorrect cluster assignments, which in turn leads to low accuracy of outlier detection results.

[0072] If the target detection algorithm is a dimensionality reduction algorithm, the disadvantages of the target detection algorithm are as follows:

[0073] (1) In the process of dimensionality reduction of the dataset to be detected, certain errors will be introduced due to information loss or imperfect mapping, thereby reducing the accuracy of outlier detection results.

[0074] (2) During the dimensionality reduction process, some important information will be lost due to the reduction in the number of features, which will affect the performance of subsequent tasks and thus reduce the efficiency of the outlier detection algorithm.

[0075] If the target detection algorithm is a machine learning algorithm based on trees, classification, and prediction, the disadvantages of the target detection algorithm are as follows:

[0076] (1) If the data set to be detected is a high-dimensional data set, due to the increase in data sparsity, the machine learning algorithm will face the problem of "dimensionality curse", that is, the performance of the algorithm is limited by the increase in dimension, thereby reducing the efficiency of the outlier detection algorithm.

[0077] (2) When the anomalies in the dataset to be detected are complex or there is overlap between the abnormal data and the normal data, the performance of the anomaly detection method will decrease, thereby reducing the efficiency of the outlier detection algorithm.

[0078] (3) Anomaly detection methods may have false positives (incorrectly labeling normal data as anomalies) or false negatives (failing to identify true anomalies), so machine learning algorithms need to be tuned and evaluated to improve accuracy and reliability.

[0079] In one embodiment, Figure 1 As shown, a method for detecting outliers is provided. The embodiment of the present application takes the method applied to a computer device as an example for explanation. The embodiment of the present application does not limit the execution device of the method for detecting outliers, and includes the following steps 102 to 106:

[0080] Step 102: Obtain a data set to be detected and a sample data set corresponding to the data set to be detected.

[0081] In implementation, the computer device obtains the data set to be tested. At the same time, the computer device obtains the sample data set corresponding to the data set to be tested. The data set to be tested and the sample data set are data sets of the same attribute. The data set to be tested contains each data subset to be tested, each transaction data subset has each data tag, and each data tag has a corresponding tag value.

[0082] Specifically, the dataset to be detected is a dataset of multi-dimensional feature combination. If the dataset to be detected is a fault dataset, the sample dataset is a fault sample dataset. If the dataset to be detected is a transaction dataset, the sample dataset is a transaction sample dataset.

[0083] In an exemplary embodiment, a computer device obtains a transaction data set. The transaction data set contains multiple transaction data subsets, each of which represents a transaction behavior. Each data tag in the transaction data subset includes a transaction ID, transaction frequency, number of transactions, transaction channel type, transaction amount, number of customers involved in the transaction, etc. The computer device needs to determine a transaction data subset representing abnormal transaction behavior from the transaction data set. Since there are differences in transaction data generated by normal transaction behavior and abnormal transaction behavior, the transaction data subset representing abnormal transaction behavior is an abnormal point of the transaction data set, and is also an outlier of the transaction data set. The computer device obtains a transaction data sample set corresponding to the transaction data set.

[0084] Optionally, the data set to be detected may be, but is not limited to, a fault data set or a transaction data set, etc. This application does not limit the data set to be detected.

[0085] Step 104 , extracting attribute features of the data set to be detected, and screening each initial target detection algorithm from each detection algorithm based on the attribute features.

[0086] Among them, attribute features include distribution features, data volume features and dimension features. Attribute features are detection constraints for outlier detection results.

[0087] In the implementation, the computer device extracts features of the data set to be detected to obtain the distribution features, data volume features and dimensional features of the data set to be detected. Then, the computer device determines each initial target detection algorithm in each detection algorithm based on the distribution features, data volume features and dimensional features. The computer device obtains a parameter set of each target detection algorithm, and updates the initial target detection algorithm based on the parameter set to obtain an updated initial target detection algorithm.

[0088] Step 106: Train each initial target detection algorithm according to the sample data set to obtain each target detection algorithm.

[0089] Among them, each target detection algorithm is used to determine the outlier detection result of the data set to be detected.

[0090] In implementation, a training stop condition is pre-set in the computer device. The computer device simultaneously trains each initial target detection algorithm based on the sample training set to obtain each initial target detection algorithm after training. Then, the computer device determines whether each initial target detection algorithm after training meets the preset training stop condition. If each initial target detection algorithm after training meets the training stop condition, the computer device determines each initial target detection algorithm as each target detection algorithm. Then, the computer device performs data processing on the data set to be detected based on each target detection algorithm to obtain the outlier detection result of the data to be detected. In particular, since the target detection algorithm is an unsupervised algorithm, the abnormal detection result of the training data is directly output after the training is completed. At the same time, the computer device saves the target detection algorithm. For example, model_KNN (KNN model), model_SOS (SOS model), model_ABOD (ABOD model) and model_LODA (LODA model).

[0091] In an optional embodiment, if each initial target detection algorithm after training does not meet the training stop condition, the computer device executes the step of training each initial target detection algorithm based on the sample training set training at the same time until each initial target detection algorithm after training meets the training stop condition.

[0092] In an exemplary embodiment, the training stop condition is that the number of training rounds reaches a preset training round number threshold. The number of training rounds is preset in the computer device, and the number of training rounds is set to 0. The computer device simultaneously inputs the sample training set into each initial target detection algorithm, and performs data processing on the sample training set through the initial target detection algorithm to obtain the trained initial target detection algorithm. Then, the computer device updates the number of training rounds, and determines whether the updated number of training rounds reaches the preset training round number threshold. If the updated number of training rounds reaches the training round number threshold, the computer device determines that each initial target detection algorithm after training meets the training stop condition. Then, the computer device determines each initial target detection algorithm after training as each target detection algorithm. If the updated number of training rounds does not reach the training round number threshold, the computer device executes the step of simultaneously inputting the sample training set into each initial target detection algorithm until the updated number of training rounds reaches the training round number threshold.

[0093] Optionally, the training stop condition may be, but is not limited to, that the number of training rounds reaches a training round threshold. The training round threshold is determined according to training requirements. The embodiment of the present application does not limit the training stop condition and the training round threshold.

[0094] In the above-mentioned outlier detection method, each initial target detection algorithm is screened in each detection algorithm by the attribute characteristics of the data set to be detected, and the initial target detection algorithm adapted to the data set to be detected is obtained, thereby improving the objectivity of screening the initial target detection algorithm. Then, each initial target detection algorithm is trained by the sample data subset corresponding to the data subset to be detected, and each target detection algorithm is obtained, thereby improving the accuracy of each target detection algorithm. Furthermore, the outlier detection result of the data set to be detected is determined by each target detection algorithm, thereby avoiding the defects of a single target detection algorithm and improving the accuracy of the outlier detection result.

[0095] In an exemplary embodiment, the attribute features include distribution features, data volume features, and dimension features; Figure 2 As shown, the specific processing process of extracting the attribute features of the data set to be detected in step 104 includes steps 202 to 206. Among them:

[0096] Step 202: draw a distribution graph of the data set to be detected, and determine the distribution characteristics of the data set to be detected based on the distribution graph.

[0097] Among them, the distribution characteristics represent the degree of correlation of the data in the data set to be detected.

[0098] In implementation, a distribution algorithm is pre-set in the computer device. The computer device draws a distribution map of the data set to be detected based on the distribution algorithm. If the data points in the distribution map are scattered, the computer device determines that the distribution feature of the data set to be detected is sparsely distributed. If the data points in some maps are relatively concentrated, the computer device determines that the distribution feature of the data set to be detected is densely distributed.

[0099] In an exemplary embodiment, the data set to be detected is a transaction data set, and the distribution algorithm is a kernel density estimation algorithm. The computer device draws a density map of the transaction data set based on the kernel density estimation (KDE) algorithm. The density map represents the overall distribution of data in the transaction data set. If the data points of the transaction data in the density map are scattered, the computer device determines that the distribution feature of the transaction data set is sparse distribution. The sparse distribution represents that the degree of association between each transaction data subset is low. If the data points of the transaction data in the density map are concentrated, the computer device determines that the dense distribution is the distribution feature of the transaction data set. The dense distribution represents that the degree of association between the transaction data subsets is high.

[0100] In an optional embodiment, the computer device draws each distribution graph of the data set to be detected based on each distribution algorithm. For example, the computer device draws a histogram of the data set to be detected according to the histogram algorithm, and draws a box plot of the data set to be detected according to the box plot algorithm. Then, the computer device determines the distribution characteristics of the data set to be detected based on the distribution of data in the histogram and the box plot.

[0101] Optionally, the distribution algorithm may be, but is not limited to, a histogram algorithm, a box plot algorithm, and a kernel density estimation algorithm. The embodiments of the present application do not limit the distribution algorithm.

[0102] Step 204 , determining the dimensional characteristics and data volume characteristics of the data set to be detected based on the data labels in the data set to be detected and the number of data subsets to be detected in the data set to be detected.

[0103] The data set to be detected includes various data subsets to be detected.

[0104] In implementation, a data label threshold and a data volume threshold are pre-set in the computer device. The computer device determines the number of labels of the data labels in the data set to be detected, and determines the dimensional characteristics of the data set to be detected based on the number of labels and the data label threshold. Then, the computer device determines the number of data subsets to be detected in the data set to be detected, and determines the data volume characteristics of the data set to be detected based on the number of data subsets to be detected and the data volume threshold.

[0105] Specifically, the computer device determines any subset of data to be detected in the data set to be detected as a target subset of data to be detected, and determines the number of labels of the data labels in the target subset of data to be detected. Then, the computer device determines whether the number of labels is greater than or equal to the data label threshold. If the number of labels is greater than or equal to the data label threshold, the computer device determines that the dimensional feature of the data set to be detected is a high dimension. If the number of labels is less than the data label threshold, the computer device determines that the dimensional feature of the data set to be detected is a low dimension. Then, the computer device counts the subsets of data to be detected in the data set to be detected to obtain the number of subsets of data to be detected. The computer device determines whether the number of subsets of data to be detected is greater than or equal to the data volume threshold. If the number of subsets of data to be detected is greater than or equal to the data volume threshold, the computer device determines that the data volume feature of the data set to be detected is a large data volume. If the number of subsets of data to be detected is less than the data volume threshold, the computer device determines that the data volume feature of the data set to be detected is a small data volume.

[0106] In an optional embodiment, a correspondence between a label quantity interval and a dimensional feature is pre-set in a computer device. Furthermore, an association between a quantity interval and a data volume feature is pre-set in a computer device. The computer device determines a target label quantity interval where the label quantity is located, and determines the dimensional feature corresponding to the target label quantity interval as the dimensional feature corresponding to the data set to be detected. The computer device determines a target quantity interval where the number of data subsets to be detected is located, and determines the data volume feature corresponding to the target quantity interval as the data volume feature of the data set to be detected.

[0107] For example, the corresponding relationship between the label quantity interval and the dimensional feature is that the label quantity is [0, 200) for low dimension, [200, 500) for medium dimension, and [500, +∞) for high dimension. The corresponding relationship between the quantity interval and the data volume feature is [0, 500) for small data volume, [500, 1000) for medium data volume, and [1000, +∞) for large data volume. If the number of labels is 400, the computer device determines that the target label quantity interval where the label quantity is located is [200, 500), and determines the middle latitude corresponding to the target label quantity interval [200, 500) as the dimensional feature corresponding to the data set to be tested. If the number of data subsets to be tested is 1800, the computer device determines the target quantity interval where the number of data subsets to be tested is [1000, +∞), and determines the large data volume corresponding to the target quantity interval as the data volume feature of the data set to be tested.

[0108] Optionally, the data label threshold and the data volume threshold are set according to the detection requirements, the correspondence between the label data volume interval and the dimensional characteristics is also set according to the detection requirements, and the correspondence between the quantity interval and the data volume characteristics is set according to the detection requirements. The embodiments of the present application do not limit the data label threshold, the data volume threshold, the correspondence between the label data volume interval and the dimensional characteristics, and the correspondence between the quantity interval and the data volume characteristics.

[0109] Step 206: construct attribute features of the data set to be detected based on the distribution features, data volume features, and dimensional features.

[0110] In implementation, the computer device combines the distribution characteristics, data volume characteristics and dimension characteristics of the data set to be detected to obtain the attribute characteristics of the data set to be detected.

[0111] In this embodiment, the attribute characteristics of the data set to be detected are obtained by extracting the distribution characteristics, data volume characteristics and dimensional characteristics of the data set to be detected, which facilitates the subsequent screening of detection algorithms that adapt to the data set to be detected based on the attribute characteristics.

[0112] In an exemplary embodiment, Figure 3 As shown, the specific process of selecting each initial target detection algorithm from each detection algorithm based on the attribute feature in step 104 includes steps 302 to 306. Among them:

[0113] Step 302 , screening detection algorithms for each target type from detection algorithms of various types according to attribute characteristics.

[0114] Among them, attribute characteristics include data volume characteristics, distribution characteristics and dimension characteristics.

[0115] In implementation, various types of detection algorithms are pre-set in the computer device, and the computer device selects the detection algorithm of each target type from the various types of detection algorithms according to the data volume characteristics, distribution characteristics and dimensional characteristics.

[0116] Specifically, the computer device is pre-set with various types of detection algorithms, as shown in the following Table 1. Table 1 is a table of various types of detection algorithms.

[0117] Table 1

[0118]

[0119] As shown in Table 1 above, eight types of detection algorithms are pre-set in the computer device, namely, distribution-based, distance-based, density-based, clustering-based, tree-based, dimension reduction-based, classification-based, and prediction-based. If the data volume feature is a small data volume, the computer device filters the detection algorithms based on distribution and prediction types among the various types of detection algorithms. If the dimensional feature is a low dimension, the computer device filters the detection algorithms based on dimension reduction types among the various types of detection algorithms. If the distribution feature is a sparse distribution, the computer device filters the detection algorithms based on distance, distribution, and prediction types among the various types of detection algorithms to obtain the detection algorithms of the target type. In particular, the computer device needs to determine at least two detection algorithms for the target type.

[0120] In an optional embodiment, the target object can re-screen the detection algorithm of each target type. Specifically, the computer device displays each target type through a configuration interface. The target object determines a first type in each target type and triggers the first type. In response to the triggering operation of each first type in each target type, the computer device obtains the first type and updates the first type to the target type.

[0121] Optionally, the target object may be, but is not limited to, a detection person, and the embodiment of the present application does not limit the target object.

[0122] Step 304: determine an initial target detection algorithm in the detection algorithms for each target type, and obtain a parameter set of the initial target detection algorithm.

[0123] In implementation, the computer device determines the initial target detection algorithm for each target type based on the number of detection algorithms for the target type. Then, the computer device obtains a parameter set of the initial target detection algorithm. The parameter set includes the weight of the initial target detection algorithm. The sum of the weights of the initial target detection algorithms is 1.

[0124] Specifically, the computer device displays the detection algorithm of each target type through a configuration page. If the number of detection algorithms of the target type is 1, the computer device determines the detection algorithm of the target type as the initial target detection algorithm. If the number of detection algorithms of the target type is greater than 1, the target object triggers the initial target detection algorithm in each detection algorithm of the target type. The computer device obtains the initial target detection algorithm in response to the triggering operation of the initial target detection algorithm. Then, for each initial target detection algorithm, the computer device obtains a parameter template corresponding to the initial target detection algorithm, and obtains a parameter set corresponding to the initial target detection algorithm based on the parameter template.

[0125] In an exemplary embodiment, the target type includes a tree-based type, a density-based type, and a dimension reduction-based type. Since the tree-based detection algorithm has only the iForest algorithm, the computer device determines the iForest algorithm as the initial target detection algorithm. Since the density-based detection algorithm includes the LOF algorithm, the COF algorithm, and the SOS algorithm. The target object triggers the LOF algorithm in each density-based detection algorithm. In response to the triggering operation of the LOF algorithm, the computer device obtains the LOF algorithm and determines the LOF algorithm as the initial target detection algorithm. Since the dimension reduction-based detection algorithm includes two algorithms, the PCA algorithm and the AutoEncoder algorithm. The target object triggers the PCA algorithm in each dimension reduction-based detection algorithm. In response to the triggering operation of the PCA algorithm, the computer device obtains the PCA algorithm and determines the PCA algorithm as the initial target detection algorithm. Then, the computer device obtains the parameter template corresponding to the initial target detection algorithm for each initial target detection algorithm, and obtains the parameter set corresponding to the initial target detection algorithm based on the parameter template.

[0126] In an exemplary embodiment, for each initial target detection algorithm, the computer device obtains a parameter template of the initial target detection algorithm. The parameter template includes the algorithm key parameters and weights of the initial target detection algorithm. Different initial target detection algorithms have different algorithm key parameters. Table 2 is a table of algorithm key parameters of the initial target detection algorithm, as shown in Table 2 below:

[0127] Table 2

[0128]

[0129]

[0130] The computer device obtains target algorithm key parameters and target weights of the initial target detection algorithm based on the parameter template of the initial target detection algorithm, and constructs a parameter set of the initial target detection algorithm according to the target algorithm key parameters and the target weights.

[0131] Optionally, each target type includes one or more initial target detection algorithms, and the embodiment of the present application does not limit the number of initial target detection algorithms under a target type.

[0132] Step 306: Update the initial target detection algorithm based on the parameter set.

[0133] In implementation, the computer device updates the default parameter set of the initial detection algorithm based on the parameter set to obtain an updated initial target detection algorithm.

[0134] In an optional embodiment, the computer device displays the initial ratio threshold based on the configuration page. In response to the triggering operation of the initial ratio threshold, the computer device obtains the ratio threshold input by the detection personnel and saves the ratio threshold.

[0135] In this embodiment, the initial target detection algorithms are screened from various detection algorithms through the attribute features of the data set to be detected, thereby avoiding direct manual setting of the initial target detection algorithm, improving the objectivity of screening the initial target detection algorithm, and further improving the accuracy of the initial target detection algorithm.

[0136] In an exemplary embodiment, the data set to be detected includes various data subsets to be detected. After obtaining various target detection algorithms, it is necessary to determine the outlier detection results of the data set to be detected based on the various target detection algorithms. Figure 4 As shown, after step 106 is executed, the specific processing process of the outlier detection method further includes steps 402 to 406. Among them:

[0137] Step 402 : performing data processing on each to-be-detected data subset based on each target detection algorithm to obtain an initial detection score of the detection data subset under each target detection algorithm.

[0138] In implementation, the computer device inputs each data subset to be detected in the data set to be detected into each target detection algorithm, processes the data subset to be detected by the target detection algorithm, and obtains the initial detection score of the data subset to be detected under the target detection algorithm. The initial detection score represents the abnormality degree of the data subset to be detected.

[0139] In an exemplary embodiment, each target detection algorithm is a target PCA algorithm, a target LOF algorithm, and a target iForest algorithm. For each of the data subsets to be detected, the computer device inputs the data subset to be detected into the target PCA algorithm, the target LOF algorithm, and the target iForest algorithm, performs data processing on the data subset to be detected by the target PCA algorithm, the target LOF algorithm, and the target iForest algorithm, and obtains the initial detection score of the data subset to be detected under the target PCA algorithm, the target LOF algorithm, and the target iForest algorithm.

[0140] Step 404: normalize each initial detection score to obtain a detection score.

[0141] In implementation, the computer device performs standardization processing on each initial detection score based on the Z_score algorithm (standard score algorithm) to obtain a detection score. The Z_score algorithm is applicable to the case where the data set contains outliers. The Z_score algorithm is shown in the following formula (1):

[0142]

[0143] In the above formula (1), *′ is the normalized detection score, x is the initial detection score calculated by the target detection algorithm before normalization, is the mean of the initial detection scores calculated by the target detection algorithm before normalization; σ is the standard deviation of the initial detection scores calculated by the target detection algorithm before normalization.

[0144] Specifically, the computer device performs mean processing on each initial detection score under each target detection algorithm to obtain the mean of the initial detection scores under the target detection algorithm. Then, the computer device calculates the standard deviation of the initial detection scores under the target detection algorithm based on the mean of the initial detection scores and each initial detection score under the target detection algorithm. The computer device performs standard processing on the initial detection scores according to the mean of the initial detection scores and the standard deviation of the initial detection scores for each initial detection score under the target detection algorithm to obtain a detection score.

[0145] Step 406: determine the outlier detection result of the to-be-detected data subset according to each detection score and the comprehensive detection score threshold.

[0146] Among them, the comprehensive detection score threshold is obtained by each target detection algorithm through data processing on the sample data set.

[0147] In implementation, the computer device performs weighted processing on each detection score corresponding to each data subset to be detected to obtain a comprehensive detection score of the data subset to be detected. Then, the computer device determines whether the data subset to be detected is an outlier based on the comprehensive detection score and the comprehensive detection score threshold, and obtains the outlier detection result of the data subset to be detected.

[0148] In this embodiment, the data set to be detected is processed by each target detection algorithm to obtain each initial detection score of the data set to be detected under each target detection algorithm, thereby obtaining the initial outlier detection results of the data set to be detected under different evaluation criteria. Then, each initial detection score is standardized to unify each initial detection score, thereby avoiding the difference in evaluation criteria between the detection scores. Furthermore, the outlier detection results are determined based on each detection score, and multiple target detection algorithms are implemented to detect the data set to be detected, thereby avoiding the limitations and disadvantages of a single target detection algorithm and improving the accuracy of the outlier detection results.

[0149] In an exemplary embodiment, Figure 5 As shown, the specific processing process of step 506 includes steps 502 to 506. Among them:

[0150] Step 502 : for each subset of data to be detected, weighted processing is performed on each detection score corresponding to the subset of data to be detected according to the weights corresponding to each target detection algorithm, so as to obtain a comprehensive detection score corresponding to the subset of data to be detected.

[0151] In implementation, the computer device determines the weight of the target detection algorithm in the parameter set of each target detection algorithm. Then, for each data subset to be detected, the computer device performs weighted processing on each detection score corresponding to the data subset to be detected according to the weight of each target detection algorithm, and obtains the comprehensive detection score of the data subset to be detected. Table 3 is a comprehensive detection score table of the data subset to be detected in an exemplary embodiment. As shown in Table 3 below:

[0152] Table 3

[0153]

[0154] Among them, in the above Table 3, ID is the serial number of each subset of data to be detected. ABOD_decision_scores is the detection score obtained by the target ABOD algorithm; KNN_decision_scores is the detection score obtained by the target KNN algorithm; SOS_decision_scores is the detection score obtained by the target SOS algorithm; LODA_decision_scores is the detection score obtained by the target LODA algorithm. Average_scores is the comprehensive detection score.

[0155] In an exemplary embodiment, the weights of the target detection algorithms are 0.4 for the target PCA algorithm, 0.4 for the target LOF algorithm, and 0.2 for the target iForest algorithm. The detection scores of the data subset to be detected under each target detection algorithm are 20 for the target PCA algorithm, 30 for the target LOF algorithm, and 10 for the target iForest algorithm. The computer device performs weighted processing on each detection score under the target detection algorithm, and obtains a comprehensive detection score of the data subset to be detected of 20*0.4+30*0.4+10*0.2=22.

[0156] In an optional embodiment, if the weights of the target detection algorithms are the same, the computer device performs mean processing on the detection scores corresponding to each subset of the data to be detected based on an arithmetic mean algorithm to obtain a comprehensive detection score.

[0157] Step 504, determine whether the comprehensive detection score is greater than the comprehensive detection score threshold, and obtain a determination result.

[0158] In implementation, the computer device determines whether the comprehensive detection score is greater than the comprehensive detection score threshold and obtains a judgment result. If the comprehensive detection score is greater than the comprehensive detection score threshold, it indicates that the probability of abnormality of the data subset to be detected is relatively high; if the comprehensive detection score is less than or equal to the comprehensive detection score threshold, it indicates that the probability of abnormality of the data subset to be detected is relatively low.

[0159] Step 506: Determine the outlier detection result of the to-be-detected data subset according to the judgment result.

[0160] In implementation, if the judgment result is that the comprehensive detection score is greater than the comprehensive detection score threshold, the computer device determines the outlier detection result of the data subset to be detected as the data subset to be detected is an outlier. If the judgment result is that the comprehensive detection score is less than or equal to the comprehensive detection score threshold, the computer device determines the data subset to be detected as normal data, that is, non-outliers.

[0161] Specifically, if the judgment result is that the comprehensive detection score Average_scores is greater than the comprehensive detection score threshold Threshold_a, the computer device determines the outlier detection result of the data subset to be detected as the data subset to be detected is an outlier, and marks the data subset to be detected as 1. If the judgment result is that the comprehensive detection score Average_scores is less than or equal to the comprehensive detection score threshold Threshold_a, the computer device determines the data subset to be detected as normal data, that is, non-outliers. At the same time, the computer device marks the data subset to be detected as 0. Table 4 is a table of outlier detection results for each data subset to be detected in an exemplary embodiment.

[0162] Table 4

[0163] ID Average_scores Threshold_a Outlier 20001 -0.107315801 0.05 0 20002 -0.993041973 0.05 0 20003 0.221617397 0.05 1 20004 0.422916623 0.05 1 ..... 0.039782851 0.05 0

[0164] In Table 4 above, Threshold_a is the comprehensive detection score threshold. Average_scores is the comprehensive detection score. ID is the serial number of each data subset to be detected.

[0165] In an exemplary embodiment, the data subset to be detected is a transaction data subset. If the comprehensive detection score is greater than the comprehensive detection score threshold, the computer device determines the transaction data subset as an outlier as an outlier detection result. At the same time, the computer device determines the transaction data subset as an abnormal transaction data subset. If the comprehensive detection score is less than or equal to the comprehensive detection score threshold, the computer device determines the transaction data subset as normal data as an outlier detection result.

[0166] In this embodiment, by weighting the detection scores of each subset of data to be detected, a comprehensive detection score is obtained, and the outlier detection result is determined based on the comprehensive detection score, thereby reducing or avoiding the limitations and disadvantages of the single target detection algorithm and improving the accuracy of the outlier detection result.

[0167] In an exemplary embodiment, after obtaining each target detection algorithm, it is necessary to determine a comprehensive detection score threshold based on each target detection algorithm and a sample data set. Figure 6 As shown, after step 106 is executed, the specific processing process of the outlier detection method further includes steps 602 to 604. Among them:

[0168] Step 602 : performing data processing on each sample data subset based on each target detection algorithm to obtain an initial sample detection score of each sample data subset under each target detection algorithm.

[0169] In implementation, the computer device inputs each sample data subset in the sample data set into each target detection algorithm, processes the sample data subset through the target detection algorithm, and obtains the initial sample detection score of the sample data subset under each sample data subset.

[0170] The process of each target detection algorithm processing the sample data subset is consistent with the process of each target detection algorithm processing the to-be-detected data subset. For the specific processing process, please refer to the above step 402. The embodiment of the present application will not be described in detail here.

[0171] Step 604 , normalize the detection scores of the initial samples to obtain the detection scores of the samples, and determine the comprehensive detection score threshold according to the detection scores of the samples.

[0172] In implementation, the computer device performs normalization processing on each initial sample detection score according to the Z_score algorithm to obtain each sample detection score. The process of normalizing the initial sample detection score is the same as the process of normalizing the initial detection score. For details, please refer to the embodiment of step 404 above, which will not be described in detail in this embodiment of the present application. Table 5 is a detailed table of sample detection scores in an exemplary embodiment.

[0173] Table 5

[0174]

[0175] In the above Table 5, ID is the sample serial number of each sample data subset. ABOD_decision_scores is the sample detection score obtained by the target ABOD algorithm; KNN_decision_scores is the sample detection score obtained by the target KNN algorithm; SOS_decision_scores is the sample detection score obtained by the target SOS algorithm; LODA_decision_scores is the sample detection score obtained by the target LODA algorithm.

[0176] Then, the computer device performs weighted processing on each sample detection score under each sample data subset to obtain a comprehensive detection score of the sample data subset. The computer device determines a comprehensive detection score threshold based on each comprehensive detection score.

[0177] In this embodiment, a comprehensive detection score threshold is determined by each target detection algorithm and a sample data set, and a judgment standard for outliers is obtained, which facilitates the subsequent determination of outlier detection results for the data subset to be detected based on the comprehensive detection score threshold.

[0178] In an exemplary embodiment, Figure 7 As shown, the specific processing process of determining the comprehensive detection score threshold according to the detection score of each sample in step 604 includes steps 702 to 706. Among them:

[0179] Step 702 , weighted processing is performed on each sample detection score corresponding to each sample data subset according to the weight of each target detection algorithm to obtain a sample comprehensive detection score corresponding to the sample data subset.

[0180] In implementation, the computer device performs weighted processing on each sample data subset according to the weight of each target detection, and obtains the comprehensive detection score of the sample data subset. The process of weighted processing of each sample detection score is the same as the process of weighted processing of each detection score. For details, please refer to the embodiment of step 406 above, which will not be repeated in detail in this embodiment of the present application. Table 6 is a detailed table of comprehensive detection scores of samples in an exemplary embodiment.

[0181] Table 6

[0182]

[0183] In the above Table 4, ID is the sample serial number of each sample data subset. ABOD_decision_scores is the sample detection score obtained by the target ABOD algorithm; KNN_decision_scores is the sample detection score obtained by the target KNN algorithm; SOS_decision_scores is the sample detection score obtained by the target SOS algorithm; LODA_decision_scores is the sample detection score obtained by the target LODA algorithm. Average_scores is the comprehensive sample detection score.

[0184] Step 704, sorting the sample comprehensive detection scores corresponding to each sample data subset in order from large to small, to obtain a sample comprehensive detection score sequence.

[0185] During implementation, the computer device sorts the sample comprehensive detection scores of each sample data subset in descending order according to the order of the sample comprehensive detection scores from large to small, and obtains a sample comprehensive detection score sequence.

[0186] Step 706, determining a comprehensive detection score threshold in the sample comprehensive detection score sequence based on a preset ratio threshold.

[0187] In implementation, the computer device is pre-set with a ratio threshold. The computer device determines the target number of digits according to the ratio threshold and the number of sample comprehensive detection scores. Then, the computer device determines the target number of sample comprehensive detection scores in the sample comprehensive detection score sequence as the comprehensive detection score threshold.

[0188] In an exemplary embodiment, the ratio threshold is 0.1. The sample comprehensive detection score sequence includes 1000 sample comprehensive detection scores. The computer device multiplies the ratio threshold 0.1 and the number of sample comprehensive detection scores 1000 to obtain a target number of 100. Then, the computer device determines the 100th sample comprehensive detection score in the sample comprehensive detection score sequence as the comprehensive detection score threshold.

[0189] Optionally, the ratio threshold can be set to, but is not limited to, 0.1. The range of the ratio threshold is (0, 0.5), which is determined according to detection requirements. The embodiment of the present application does not limit the ratio threshold.

[0190] In this embodiment, each target detection algorithm is used to process each sample data subset, and an initial sample detection score of each sample data under different evaluation criteria is obtained. Then, a comprehensive detection score threshold is determined based on each initial sample detection score, and an outlier judgment criterion under each target detection algorithm is obtained, thereby avoiding the limitations of the evaluation criterion under a single target detection algorithm, and then determining the outlier detection result of the data subset to be detected based on the comprehensive detection score threshold, thereby improving the accuracy of the outlier detection result.

[0191] In an exemplary embodiment, Figure 8 FIG. 1 is a flow chart of an outlier detection method in an exemplary embodiment. Figure 8 As shown, the outlier detection method includes:

[0192] Step 801: Obtain a data set to be detected and a sample data set.

[0193] Step 802, configure the ratio threshold a.

[0194] Step 803: extract attribute features of the data set to be detected and select an initial target detection algorithm according to the attribute features.

[0195] Step 804: configure each initial target detection algorithm, and train each initial target detection algorithm based on the sample data set to obtain each target detection algorithm.

[0196] Step 805, save each trained target detection algorithm.

[0197] Step 806 , determining each initial sample detection score based on each target detection algorithm and the sample data subset.

[0198] Step 807: standardize the detection scores of each initial sample to obtain the detection scores of each sample.

[0199] Step 808: Use an arithmetic mean algorithm to perform weighted processing on the detection scores of each sample corresponding to each sample data subset to obtain a comprehensive detection score for each sample.

[0200] Step 809, sorting the comprehensive detection scores of each sample in descending order to obtain a sample comprehensive detection score sequence.

[0201] Step 810, determining a comprehensive detection score threshold in the sample comprehensive detection score sequence based on the ratio threshold.

[0202] Step 811 , determining each initial detection score of each subset of the data to be detected in the data set to be detected based on each target detection algorithm.

[0203] Step 812, normalize and weight each initial detection score to obtain a comprehensive detection score for each subset of data to be detected.

[0204] Step 813, comparing the comprehensive detection score of each to-be-detected data subset with the comprehensive detection score threshold.

[0205] Step 814: if the comprehensive detection score is greater than the comprehensive detection score threshold, the data subset to be detected is determined as an outlier and marked as 1.

[0206] Step 815: If the comprehensive detection score is less than or equal to the comprehensive detection score threshold, the data subset to be detected is determined as normal data and marked as 0.

[0207] Step 816, output abnormal outlier records.

[0208] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0209] Based on the same inventive concept, the embodiment of the present application also provides an outlier detection device for implementing the outlier detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more outlier detection device embodiments provided below can refer to the limitations of the outlier detection method above, and will not be repeated here.

[0210] In an exemplary embodiment, Fig. 9 As shown, an outlier detection device 900 is provided, comprising: an acquisition module 901, a screening module 902 and a training module 903, wherein:

[0211] The acquisition module 901 is used to acquire the data set to be detected and the sample data set corresponding to the data set to be detected.

[0212] The screening module 902 is used to extract the attribute features of the data set to be detected, and screen each initial target detection algorithm among various detection algorithms based on the attribute features; the attribute features are the detection constraints of the outlier detection results.

[0213] The training module 903 is used to train each initial target detection algorithm according to the sample data set to obtain each target detection algorithm; each target detection algorithm is used to determine the outlier detection result of the data set to be detected.

[0214] In an exemplary embodiment, the attribute features include distribution features, data volume features and dimension features; the screening module 902 includes a first extraction submodule and a first screening submodule. The first extraction submodule includes:

[0215] The drawing submodule is used to draw a distribution map of the data set to be detected, and determine the distribution characteristics of the data set to be detected based on the distribution map.

[0216] The first determination submodule is used to determine the dimensional characteristics and data volume characteristics of the data set to be detected based on the data labels in the data set to be detected and the number of data subsets to be detected in the data set to be detected.

[0217] The first construction submodule is used to construct attribute features of the data set to be detected based on distribution features, data volume features and dimensional features.

[0218] In an exemplary embodiment, the screening module 902 includes a first extraction submodule and a first screening submodule. The first screening submodule includes:

[0219] The second screening submodule is used to screen the detection algorithm of each target type from among the detection algorithms of various types according to the attribute characteristics.

[0220] The first acquisition submodule is used to determine an initial target detection algorithm in the detection algorithm of each target type and acquire a parameter set of the initial target detection algorithm.

[0221] The first updating submodule is used to update the initial target detection algorithm based on the parameter set.

[0222] In an exemplary embodiment, the data set to be detected includes various data subsets to be detected, and the outlier detection device 900 further includes:

[0223] The first processing module is used to perform data processing on each to-be-detected data subset based on each target detection algorithm to obtain an initial detection score of the detection data subset under each target detection algorithm.

[0224] The second processing module is used to perform standardization processing on each initial detection score to obtain a detection score.

[0225] The first determination module is used to determine the outlier detection result of the data subset to be detected according to each detection score and the comprehensive detection score threshold; the comprehensive detection score threshold is obtained by each target detection algorithm through data processing on the sample data set.

[0226] In an exemplary embodiment, the first determining module includes:

[0227] The first processing submodule is used to perform weighted processing on each detection score corresponding to the data subset to be detected according to the weights corresponding to each target detection algorithm for each data subset to be detected, so as to obtain a comprehensive detection score corresponding to the data subset to be detected.

[0228] The first judgment submodule is used to judge whether the comprehensive detection score is greater than the comprehensive detection score threshold and obtain a judgment result.

[0229] The second determination submodule is used to determine the outlier detection result of the to-be-detected data subset according to the judgment result.

[0230] In an exemplary embodiment, the sample data set includes sample data subsets, and the outlier detection device 900 further includes:

[0231] The third processing module is used to perform data processing on each sample data subset based on each target detection algorithm to obtain an initial sample detection score of each sample data subset under each target detection algorithm.

[0232] The fourth processing module is used to perform standardization processing on each initial sample detection score to obtain each sample detection score, and determine a comprehensive detection score threshold based on each sample detection score.

[0233] In an exemplary embodiment, the fourth processing module includes a second processing submodule and a third determining submodule. The third determining submodule includes:

[0234] The second processing submodule is used to perform weighted processing on each sample detection score corresponding to each sample data subset according to the weight of each target detection algorithm, so as to obtain a sample comprehensive detection score corresponding to the sample data subset.

[0235] The sorting submodule is used to sort the sample comprehensive detection scores corresponding to each sample data subset in order from large to small to obtain a sample comprehensive detection score sequence.

[0236] The fourth determination submodule is used to determine a comprehensive detection score threshold in the sample comprehensive detection score sequence based on a preset ratio threshold.

[0237] Each module in the above-mentioned outlier detection device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0238] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Wherein, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Wherein, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, an outlier detection method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0239] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0240] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0241] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0242] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0243] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0244] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0245] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for detecting outliers, characterized in that: The method comprises: Acquire a data set to be detected and a sample data set corresponding to the data set to be detected; Extracting attribute features of the data set to be detected, and screening each initial target detection algorithm from each detection algorithm based on the attribute features; the attribute features are detection constraints for outlier detection results; Each of the initial target detection algorithms is trained according to the sample data set to obtain each target detection algorithm; each of the target detection algorithms is used to determine the outlier detection result of the data set to be detected.

2. The method according to claim 1, characterized in that The attribute features include distribution features, data volume features and dimensional features; the extracting the attribute features of the data set to be detected includes: Draw a distribution map of the data set to be detected, and determine the distribution characteristics of the data set to be detected based on the distribution map; Determining the dimensional characteristics and data volume characteristics of the data set to be detected based on the data labels in the data set to be detected and the number of data subsets to be detected in the data set to be detected; The attribute characteristics of the data set to be detected are constructed according to the distribution characteristics, the data volume characteristics and the dimensional characteristics.

3. The method according to claim 1, characterized in that The selecting each initial target detection algorithm from each detection algorithm based on the attribute feature includes: Selecting a detection algorithm for each target type from among various types of detection algorithms according to the attribute characteristics; Determine an initial target detection algorithm in the detection algorithm of each target type, and obtain a parameter set of the initial target detection algorithm; The initial object detection algorithm is updated based on the parameter set.

4. The method according to claim 1, characterized in that: The data set to be detected includes each subset of data to be detected, and after each initial target detection algorithm is trained according to the sample data set to obtain each target detection algorithm, the method further includes: Performing data processing on each of the to-be-detected data subsets based on each of the target detection algorithms to obtain an initial detection score of the detection data subset under each of the target detection algorithms; Performing standardization processing on each of the initial detection scores to obtain a detection score; The outlier detection result of the data subset to be detected is determined according to each of the detection scores and the comprehensive detection score threshold; the comprehensive detection score threshold is obtained by each of the target detection algorithms performing data processing on the sample data set.

5. The method according to claim 4, characterized in that The determining of the outlier detection result of the to-be-detected data subset according to each of the detection scores and the comprehensive detection score threshold comprises: For each of the data subsets to be detected, weighted processing is performed on the detection scores corresponding to the data subsets to be detected according to the weights corresponding to the target detection algorithms to obtain a comprehensive detection score corresponding to the data subsets to be detected; Determine whether the comprehensive detection score is greater than a comprehensive detection score threshold, and obtain a determination result; The outlier detection result of the to-be-detected data subset is determined according to the judgment result.

6. The method according to claim 1, characterized in that The sample data set includes each sample data subset, and after each initial target detection algorithm is trained according to the sample data set to obtain each target detection algorithm, the method further includes: Performing data processing on each of the sample data subsets based on each of the target detection algorithms to obtain an initial sample detection score for each of the sample data subsets under each of the target detection algorithms; The initial sample detection scores are standardized to obtain the detection scores of the samples, and the comprehensive detection score threshold is determined according to the detection scores of the samples.

7. The method according to claim 6, characterized in that Determining the comprehensive detection score threshold according to the detection scores of each sample includes: According to the weight of each target detection algorithm, weighted processing is performed on each sample detection score corresponding to each sample data subset to obtain a sample comprehensive detection score corresponding to the sample data subset; Sorting the sample comprehensive detection scores corresponding to each of the sample data subsets in order from largest to smallest to obtain a sample comprehensive detection score sequence; A comprehensive detection score threshold is determined in the sample comprehensive detection score sequence based on a preset ratio threshold.

8. An outlier detection device, characterized in that: The device comprises: An acquisition module, used to acquire a data set to be detected and a sample data set corresponding to the data set to be detected; A screening module, used to extract attribute features of the data set to be detected, and screen each initial target detection algorithm among various detection algorithms based on the attribute features; the attribute features are detection constraints of outlier detection results; The training module is used to train each of the initial target detection algorithms according to the sample data set to obtain each target detection algorithm; each of the target detection algorithms is used to determine the outlier detection result of the data set to be detected.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images