Abnormal data detection method and device, storage medium and electronic equipment

Abstract

The application discloses an abnormal data detection method and device, a storage medium and an electronic device. The method comprises the following steps: obtaining a first data set to be detected, wherein the first data set comprises at least one first structured data; detecting and processing the first data set by using a plurality of pre-trained first layer base models respectively, so as to obtain a plurality of first abnormal score sets corresponding to the first structured data; fusing and processing the plurality of first abnormal score sets corresponding to the plurality of first structured data by using a pre-trained second layer base model, so as to obtain a plurality of second abnormal score sets; determining an absolute ranking score and a relative ranking score of each first structured data based on the plurality of second abnormal score sets; and determining that the first structured data is abnormal data when the average value of the absolute ranking score and the relative ranking score of the first structured data is greater than a preset score threshold. The application solves the technical problem that the cost of manually detecting abnormal data is high and the accuracy is low.

Description

Technical Field

[0001] This application relates to the field of data detection technology, and more specifically, to an abnormal data detection method, apparatus, storage medium, and electronic device. Background Technology

[0002] Anomaly detection is a method of identifying data problems that do not conform to expected behavioral patterns. It involves various technologies such as machine learning and data mining for intelligent data, and therefore has wide applications in various aspects of daily life and production. Examples include risk detection in the telecommunications industry, fraud prevention and loan default detection in the financial industry, disease detection in the medical industry, and network security intrusion detection. However, because telecommunications operational information data, financial fraud user behavior data, disease detection user symptom characteristic data, and network security detection network user traffic and behavior data are essentially structured data (i.e., data that can be represented using relational data structures or logically expressed using two-dimensional tables), the quality of anomaly detection results for structured data can significantly impact the healthy development of related businesses.

[0003] Currently, anomaly detection for structured data mostly employs statistical or model-based artificial intelligence machine learning methods. However, statistical methods, such as the 3σ method and Z-score method, lack effective data learning and are insufficiently accurate, making them unsuitable for most production requirements. Model-based methods, such as KNN (K-Nearest Neighbor), COF (Connectivity-Based Outlier Factor), LOF (Local Outlier Factor), CBLOF (Cluster-based Local Outlier Factor), GMM (Gaussian Mixture Model), ALAD (Adversarial Learned Anomaly Detection), and LUNAR (Labeled and Unlabeled Nodes and Relations, graph neural network-based anomaly detection algorithms, etc., however, these model-based methods build models for data samples from a single aspect, without using model fusion to effectively integrate the results of multiple models in a hierarchical manner. This fails to leverage the characteristics of multiple model methods. At the same time, the screening characteristics are often relatively simple, the ability to identify abnormal samples is limited, and the accuracy of anomaly detection results for structured data is difficult to guarantee.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides an abnormal data detection method, apparatus, storage medium, and electronic device to at least solve the technical problems of high cost and low accuracy of manual abnormal data detection.

[0006] According to one aspect of the embodiments of this application, an abnormal data detection method is provided, comprising: acquiring a first dataset to be detected, wherein the first dataset includes at least one first structured data; performing detection processing on the first dataset respectively through multiple pre-trained first-layer base models to obtain multiple sets of first abnormal scores corresponding to each first structured data, wherein each set of first-layer base models includes multiple base models with different model characteristics; fusing the multiple sets of first abnormal scores corresponding to the multiple first structured data through a pre-trained second-layer base model to obtain multiple sets of second abnormal scores, wherein each set of second abnormal scores includes: for... A second subset of abnormal scores reflects the degree of abnormality of the first structured data; based on multiple sets of second abnormal score sets, an absolute ranking score and a relative ranking score are determined for each first structured data, wherein the absolute ranking score reflects the absolute ranking of the second abnormal score subset corresponding to the first structured data within the corresponding second abnormal score set, and the relative ranking score reflects the relative ranking of the second abnormal score subset corresponding to the first structured data relative to other second abnormal score subsets within the second abnormal score set; when the average of the absolute ranking score and the relative ranking score of the first structured data is greater than a preset score threshold, the first structured data is determined to be abnormal data.

[0007] Optionally, obtaining the first dataset to be detected includes: obtaining the second dataset to be detected; preprocessing the second dataset to obtain the first dataset, wherein the preprocessing includes at least one of the following: data cleaning, feature encoding, and standard normalization.

[0008] Optionally, the model characteristics include at least one of the following: distance characteristics, density characteristics, clustering characteristics, and neural network characteristics. The method further includes: when the model characteristic is a distance characteristic, the base model includes at least one of the following: a first base model based on the K-nearest neighbor algorithm, a second base model based on the isolated forest algorithm; when the model characteristic is a density characteristic, the base model includes at least one of the following: a third base model based on local anomaly factors, a fourth base model based on the connectivity anomaly factor algorithm; when the model characteristic is a clustering characteristic, the base model includes at least one of the following: a Gaussian mixture model, a fifth base model based on the local anomaly factor algorithm of clustering; when the model characteristic is a neural network characteristic, the base model includes at least one of the following: a sixth base model based on graph neural networks, a seventh base model based on generative adversarial networks.

[0009] Optionally, the first dataset is processed by multiple pre-trained first-layer base models to obtain multiple sets of first anomaly scores corresponding to each first structured data, including: for each set of first-layer base models, the first structured data is processed by each base model in the first-layer base model to obtain multiple first anomaly scores corresponding to the first structured data; the multiple first anomaly scores of the first structured data form the first anomaly score set of the first structured data.

[0010] Optionally, multiple sets of first anomaly scores corresponding to multiple first structured data are fused using a pre-trained second-layer base model to obtain multiple sets of second anomaly scores. This includes: normalizing the multiple sets of first anomaly scores corresponding to each first structured data to obtain normalized sets of first anomaly scores; and sequentially inputting the normalized sets of first anomaly scores corresponding to each first structured data as multiple input features into the second-layer base model to obtain the multiple sets of second anomaly scores output by the second-layer base model.

[0011] Optionally, determining the absolute and relative ranking scores for each first structured data based on multiple sets of second abnormal score subsets includes: for each first structured data, determining the absolute ranking sub-score of each second abnormal score subset using the Borda Count algorithm based on its ranking within the corresponding second abnormal score subset, and determining the absolute ranking score of the first structured data based on the absolute ranking sub-scores of multiple sets of second abnormal score subsets; for each first structured data, determining multiple winning and multiple negative scores for each first structured data using the Copeland algorithm based on the size relationship between the second abnormal score subset corresponding to the first structured data and other second abnormal score subsets within the second abnormal score subset, and determining the relative ranking score of the first structured data based on the multiple winning and multiple negative scores of the first structured data.

[0012] Optionally, when the average score of the absolute ranking score and the relative ranking score corresponding to the first structured data is greater than a preset score threshold, the first structured data is determined to be abnormal data, including: normalizing the absolute ranking score and the relative ranking score of the first structured data; calculating the average score of the normalized absolute ranking score and the relative ranking score, and determining the first structured data to be abnormal data when the average score is not less than the score threshold.

[0013] According to another aspect of the embodiments of this application, an abnormal data detection apparatus is also provided, comprising: an acquisition module, configured to acquire a first dataset to be detected, wherein the first dataset includes at least one first structured data; an anomaly detection module, configured to perform detection processing on the first dataset respectively through multiple pre-trained first-layer base models to obtain multiple sets of first anomaly score sets corresponding to each first structured data, wherein each set of first-layer base models includes multiple base models with different model characteristics; and a fusion processing module, configured to perform fusion processing on the multiple sets of first anomaly score sets corresponding to the multiple first structured data through pre-trained second-layer base models to obtain multiple sets of second anomaly score sets, wherein each set of second anomaly score sets includes The system comprises: a second subset of abnormal scores used to reflect the degree of abnormality of the first structured data; a determination module used to determine the absolute ranking score and relative ranking score corresponding to each first structured data based on multiple sets of second abnormal score sets, wherein the absolute ranking score reflects the absolute ranking of the second abnormal score subset corresponding to the first structured data within the corresponding second abnormal score set, and the relative ranking score reflects the relative ranking of the second abnormal score subset corresponding to the first structured data relative to other second abnormal score subsets within the second abnormal score set; and an anomaly determination module used to determine that the first structured data is abnormal data when the average of the absolute ranking score and the relative ranking score corresponding to the first structured data is greater than a preset score threshold.

[0014] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-described abnormal data detection method by running the computer program.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described abnormal data detection method through the computer program.

[0016] In this embodiment, a first dataset to be detected is obtained, wherein the first dataset includes at least one first structured data; the first dataset is processed by multiple pre-trained first-layer base models to obtain multiple sets of first anomaly scores corresponding to each first structured data, wherein each set of first-layer base models includes multiple base models with different model characteristics; the multiple sets of first anomaly scores corresponding to multiple first structured data are fused by a pre-trained second-layer base model to obtain multiple sets of second anomaly scores, wherein each set of second anomaly scores includes a subset of second anomaly scores used to reflect the degree of anomaly of the first structured data; based on the multiple sets of second anomaly scores, an absolute ranking score and a relative ranking score are determined for each first structured data, wherein the absolute ranking score is used to reflect the absolute ranking of the subset of second anomaly scores corresponding to the first structured data within the corresponding second anomaly score set, and the relative ranking score is used to reflect the relative ranking of the subset of second anomaly scores corresponding to the first structured data relative to other subsets of second anomaly scores within the second anomaly score set; when the average of the absolute ranking score and the relative ranking score of the first structured data is greater than a preset score threshold, the first structured data is determined to be anomaly data.

[0017] By using the aforementioned multi-layered fusion model for anomaly detection, the technical problems of high cost and low accuracy of manual anomaly detection are effectively solved, improving the efficiency and accuracy of anomaly detection for structured data and ensuring the effectiveness of anomaly detection for structured data. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a hardware structure block diagram of an optional computer terminal (or mobile device) for implementing an abnormal data detection method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating an optional abnormal data detection method according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of an optional abnormal data detection device according to an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Furthermore, all information and data (including but not limited to user device information, user personal information, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties. For example, this system has an interface with the relevant user or organization. Before obtaining relevant information, it needs to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving consent from the aforementioned user or organization.

[0025] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:

[0026] The KNN algorithm is a basic classification and regression algorithm. Its basic idea is to find the k nearest training samples to the sample to be classified and determine the class of the sample based on the classification of these k samples. The steps of the KNN algorithm are as follows: calculate the distance between the sample to be classified and the training samples (usually using Euclidean distance or Manhattan distance); select the k nearest training samples; vote on the class of these k samples and assign the sample to the class with the most votes; output the class of the sample to be classified. In summary, the advantages of the KNN algorithm include its simplicity, ease of implementation, and suitability for multi-class classification problems. However, the KNN algorithm also has some disadvantages, including requiring a significant amount of computation time, potentially producing misleading results on imbalanced datasets, and possibly suffering from the curse of characteristics with high-dimensional data.

[0027] The LOF (Location of Found) algorithm is an unsupervised learning method for anomaly detection, used to identify outliers in a dataset. Based on the concept of density, LOF determines the degree of anomaly by calculating the ratio of the local density of each data point to the local density of its neighbors. Specifically, LOF first calculates the k-distance (K-Distance) of each data point, which is the distance between the data point and its k-th nearest neighbor. Then, for each data point, it calculates its reachability distance (RD), representing the maximum distance from that point to a neighboring point. Next, it measures the local density of each data point by calculating its local reachability density (LRD), which is the ratio of its reachability distance to the average reachability distance of its k nearest neighbors. Finally, it determines the degree of anomaly of each data point by calculating its local outlier factor, which is the ratio of its local reachability density to the local reachability density of its neighbors. The core idea of ​​LOF is that if a data point has a relatively low local density and its neighbors have relatively high densities, then that point is likely an outlier. Therefore, outliers in a dataset can be identified by calculating the local anomaly factor. In summary, the advantage of the LOF algorithm is its ability to adaptively adjust the density threshold, making it suitable for datasets with various density distributions. However, the LOF algorithm has high computational complexity, especially when dealing with large-scale datasets, requiring numerous distance calculations and sorting operations, thus resulting in low efficiency.

[0028] The COF algorithm is a density-based connectivity model that determines the degree of anomalousness by calculating the connectivity factor of each point. Specifically, the COF algorithm is similar to the LOF method mentioned above. It first obtains the local reachability density of each point; then, for each point, it calculates the sum of the local reachability densities of its neighboring points, and then calculates the connectivity-based outlier factor (COF) of that point, which is the ratio of the sum of the local reachability densities of its neighboring points to the local reachability density of that point, to determine the degree of anomalousness of that point.

[0029] The CBLOF algorithm is used to detect outliers. It combines the ideas of LOF (List of Facts) and clustering algorithms. It divides the dataset into multiple clusters and calculates the LOF value for each data point to determine if it is an outlier. The steps of the CBLOF algorithm are as follows: First, the dataset is clustered to obtain multiple clusters; then, for each data point, its LOF value to its cluster is calculated, where the LOF value represents the degree of anomalousness of the data point compared to its neighbors; finally, based on the LOF value, it is determined whether the data point is an outlier. Generally, if the LOF value is greater than a certain threshold, it can be identified as an outlier. In summary, the advantages of the CBLOF algorithm are that it can handle high-dimensional data and large-scale datasets, and it can effectively detect outliers for clusters of different shapes and densities. However, its disadvantage is that it may produce false positives when the dataset contains noise.

[0030] The Gaussian Mixture Model (GMM) is a probabilistic clustering algorithm. It assumes the dataset consists of several Gaussian distributions, each representing a cluster, and the probability of a data point belonging to a cluster is determined by its distance to each cluster center. The main steps of the GMM algorithm are as follows: First, randomly select K data points as initial cluster centers, and then estimate the probability of each data point belonging to each cluster. Next, based on the current cluster centers, calculate the probability of each data point belonging to each cluster, and based on the probabilities of each cluster, re-estimate the cluster centers and the covariance matrix of each cluster. Repeat this step until the change in the cluster centers and covariance matrix is ​​less than a certain threshold or the maximum number of iterations is reached. Finally, based on the final cluster centers, assign each data point to the nearest cluster center. In summary, the advantages of the GMM algorithm are that it can approximate complex data distributions using multiple Gaussian distributions, improving clustering accuracy. Furthermore, for noisy datasets, the GMM algorithm can better adapt to the data by adjusting the weights of the cluster centers and covariance matrix. However, the algorithm is sensitive to the choice of initial cluster centers. Different initial values ​​may lead to different clustering results, and for non-convex datasets, it may produce unsatisfactory clustering results.

[0031] The ALAD algorithm is an anomaly detection algorithm based on Generative Adversarial Networks (GANs). Its basic idea is to learn the distribution of normal samples by training an autoencoder (AE) and a discriminator, and then use the discriminator to determine whether an input sample is an anomaly. Specifically, the ALAD algorithm consists of two main networks: a generator and a discriminator. The generator is an autoencoder that maps input samples to a low-dimensional hidden representation and attempts to reconstruct the input sample from this hidden representation. The discriminator is a binary classifier that attempts to distinguish between the hidden representation output by the generator and the hidden representation extracted from normal samples. Typically, the training process of the ALAD algorithm is divided into two phases: pre-training and adversarial training. In the pre-training phase, the generator and discriminator are trained using the loss functions of the autoencoder and binary classifier, respectively. In the adversarial training phase, the generator and discriminator are trained through alternating updates, where the generator attempts to minimize the reconstruction error of the autoencoder, and the discriminator attempts to maximize the accuracy of the binary classifier. During the testing phase, for a new input sample, the ALAD algorithm determines whether it is an anomaly by calculating the difference between the hidden representation output by the generator and the hidden representation extracted from normal samples. If the difference exceeds a predefined threshold, it is considered an anomaly. The ALAD algorithm has certain advantages, such as its ability to capture complex anomaly patterns and its robustness to different types of anomalies. However, the algorithm may face some challenges when handling high-dimensional and large-scale data, requiring further improvement and optimization.

[0032] The LUNAR algorithm is a graph neural network-based model used for node classification and relationship prediction tasks. Graph neural networks are neural network models capable of processing graph data; they can learn features of nodes and edges and use these features for node classification and relationship prediction. Therefore, the LUNAR model divides graph data into labeled and unlabeled nodes, trains the model using the label information of labeled nodes, and performs semi-supervised learning using the feature information of unlabeled nodes. The main idea of ​​the LUNAR model is to learn node representations through an autoencoder. First, the model maps the features of nodes and edges to a low-dimensional representation space using the encoder. Then, the decoder remaps the low-dimensional representation back to the original feature space. The model's goal is to minimize the difference between the original features and the reconstructed features, thereby learning the node representation. In the LUNAR model, the label information of labeled nodes is used for supervised training, while the feature information of unlabeled nodes is used for semi-supervised learning. In this way, the LUNAR model can perform node classification and relationship prediction using the label information of labeled nodes and the feature information of unlabeled nodes. Experiments demonstrate that the LUNAR model achieves good performance on some graph datasets, showing its potential and advantages in graph data analysis tasks.

[0033] The Minimum Covariance Determinant (MCD) algorithm is a method for anomaly detection and outlier identification. The basic idea is to find the subset with the minimum covariance by minimizing the determinant of the covariance matrix. Specifically, the MCD model first samples a subset from the original dataset and then calculates the covariance matrix of that subset. Next, the model calculates the determinant of the covariance matrix composed of samples from this subset. If the determinant value is small, the subset is considered to have the minimum covariance. The specific steps of the algorithm are: first, randomly sample a subset from the original dataset; then calculate the covariance matrix of this subset; then calculate the determinant of the covariance matrix as the discriminant criterion for this subset; repeat these steps multiple times, selecting the subset with the minimum determinant value as the final normal sample subset; finally, determine whether it is an outlier by calculating the Mahalanobis distance between it and the final normal sample subset.

[0034] The Borda Count algorithm is a method for calculating election results. It determines the final score based on a candidate's ranking among voters. The basic idea is to assign each candidate a score according to their ranking among voters, then sum all the scores, and the candidate with the highest score is selected. Specifically, the algorithm's execution steps are as follows: First, the ranking of each voter is transformed, assigning n-1 points to the first-ranked candidate, n-2 points to the second-ranked candidate, and so on, until the last candidate is assigned 0 points, where n is the number of candidates. Next, the scores assigned to each candidate by each voter are summed to obtain each candidate's total score. Finally, the candidates are ranked according to their total scores, and the candidate with the highest score is selected. In summary, the Borda Count algorithm is simple and easy to understand, and it takes into account the overall ranking of candidates among voters. However, it also has some drawbacks, such as being susceptible to strategic voting by voters, and having high computational complexity when there are many candidates.

[0035] Copeland's algorithm is an algorithm for solving election problems. Election problems involve selecting a winner from a set of candidates. This algorithm calculates a comparison score for each candidate against the others and then determines the winner based on these scores. Specifically, Copeland's algorithm compares each candidate pairwise with the others and calculates the number of wins for each candidate. For example, if candidate A is compared with candidate B, a win for A is recorded as 1, a win for B as -1, and a tie as 0. Then, the number of wins for each candidate is summed to obtain their comparison score. Finally, the winner is determined based on these scores. If only one candidate has the highest comparison score, that candidate is the winner. If multiple candidates have the same highest comparison score, these candidates are all considered the winners. Copeland's algorithm is simple to understand and computationally efficient. However, it also has some limitations, such as the high time complexity of calculating comparison scores when the number of candidates is large.

[0036] Min-max normalization, also known as deviation normalization, is a commonly used data standardization method used to scale numerical data to a specific range. Specifically, min-max normalization performs a linear transformation on the original data, mapping data values ​​to the range [0,1]. Assuming the minimum value of the original data is min and the maximum value is max, then any value x in the original data will have the following value after min-max normalization:

[0037] x ′ =(xx) min ) / (x max -x min )

[0038] Where, x ′ This represents the normalized value. Besides min-max normalization, other commonly used data standardization methods include z-score standardization and decimal scaling. Different data standardization methods are suitable for different data situations; the appropriate method should be selected based on specific needs.

[0039] Example 1

[0040] With the widespread availability of the internet and the complexity of telecommunications services, frontline staff in telecommunications service centers handle multiple business scenarios, including top-ups, sales, refunds, and SIM card issuance. This results in an exceptionally large volume of operational data. Furthermore, the behavioral information generated by staff during these transactions may contain anomalies. Using traditional manual annotation methods to detect anomalies would be prohibitively expensive and inconsistently accurate, hindering timely mitigation of losses and hindering the company's development. Therefore, the quality of anomaly detection results for structured data significantly impacts the healthy development of related businesses.

[0041] Currently, the relevant patent (CN202210222958) "Data Anomaly Detection Method, Device, Medium and Electronic Equipment" proposes a data anomaly detection method based on a fusion model. It adopts a stacking machine learning model fusion method, but the fusion level is only two layers and one fusion. It can be regarded as the integration and fusion of traditional single stacking machine learning models. Moreover, the integrated and fused models are all homogeneous models based on trees. Therefore, this scheme lacks the integration and fusion of other heterogeneous model methods, and it is difficult to guarantee the accuracy of anomaly detection results for structured data.

[0042] The relevant patent (CN202210718289) "Data Detection Method, Apparatus, Computer Equipment and Storage Medium" proposes a model-based data anomaly detection method, which adopts the more classic tree-based isolated forest method. Therefore, this method only uses a single tree model for anomaly detection, resulting in limited anomaly identification ability and difficulty in ensuring the accuracy of anomaly detection results for structured data.

[0043] The relevant patent (CN202210632838) "Method for Detecting Data Anomalies" proposes an anomaly detection method based on the confidence level of a statistical probability distribution function. It uses the statistical probability distribution function to calculate the confidence level to obtain abnormal samples. However, based on the relevant principles of statistics, this type of method considers data that does not meet certain statistical rules as abnormal data. Therefore, it lacks effective learning of the data and is often not accurate enough.

[0044] In addition, the relevant patent (CN202210741524) "A data anomaly detection method, system, electronic device and readable storage medium" also proposes an anomaly detection model method based on multi-feature data. In this scheme, it is pointed out that the multi-feature data is obtained through different acquisition methods, and the original data is directly subjected to multi-angle deep encoder and clustering feature calculation. Similarly, it does not involve model integration and fusion. Therefore, this scheme is also difficult to guarantee the accuracy of anomaly detection results for structured data.

[0045] Therefore, none of the existing technologies described above disclose anomaly monitoring methods based on multi-model fusion, making it impossible for existing solutions to guarantee the accuracy of anomaly monitoring results for structured data. To address this issue, this application provides relevant solutions, which are detailed below.

[0046] According to an embodiment of this application, an embodiment of an abnormal data detection method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0047] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing an anomaly data detection method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0048] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0049] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the abnormal data detection method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the abnormal data detection method of the application described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0050] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0051] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0052] Under the above operating environment, Figure 2 This is a flowchart illustrating an optional abnormal data detection method according to an embodiment of this application, as shown below. Figure 2 As shown, the method includes at least steps S201-S205, wherein:

[0053] Step S201: Obtain the first dataset to be detected.

[0054] The first dataset includes at least one first structured data set. Structured data refers to data stored and processed according to a specific format and organization. It typically exists in the form of tables, databases, XML, JSON, etc., and has clearly defined relationships between fields and values. Therefore, it can also be understood as regular data that can be represented using relational data structures or logically expressed in the form of two-dimensional tables. Structured data generally includes, but is not limited to: telecommunications business behavior information data, user financial behavior data related to financial fraud, user disease characteristic data for disease detection, and network user traffic for network security detection.

[0055] As an optional implementation method, the first dataset can be obtained as follows: first, obtain the second dataset to be detected; then preprocess the second dataset to obtain the first dataset, wherein the preprocessing includes at least one of the following: data cleaning, feature encoding, and standardization.

[0056] In this embodiment, firstly, the original structured dataset (i.e., the second dataset to be detected) is obtained. Since the original structured dataset includes abnormal characters, missing values, and non-numerical features, it can be preprocessed. Preprocessing operations include, but are not limited to, removing abnormal characters, handling missing values, and feature encoding. Then, the preprocessed original structured dataset is standardized. For example, the features of each original structured data point (i.e., the first structured data) within the original structured dataset can be normalized to a standard normal distribution (mean 0, variance 1). The formula for standardization is as follows:

[0057]

[0058] Among them, the above σ(x) represents the mean of the original structured dataset, and σ(x) represents the standard deviation of the original structured dataset.

[0059] Specifically, taking a monthly salesperson behavior dataset obtained from a telecommunications big data platform as the structured dataset to be detected as an example, 32 feature fields are first obtained, such as employee ID, BSS channel ID, agent channel ID, agent name, number of low-price broadband transfer work orders, number of card issuances at night, rebate amount, number of card issuances at abnormal ages, and number of small-value sales items ordered. These feature fields are all numerical fields. Next, 25,841 second structured data entries corresponding to these 32 feature fields are determined. To ensure the accuracy of subsequent abnormal data detection, the 25,841 second structured data entries need to be processed by removing abnormal characters, filling missing values, and feature encoding. Feature encoding can be one-hot encoding to obtain the processed first structured data. The first dataset is composed of these processed first structured data entries and the corresponding feature fields.

[0060] Step S202: The first dataset is processed by multiple pre-trained first-layer base models to obtain multiple sets of first anomaly score sets corresponding to each first structured data.

[0061] Each group of first-layer base models includes multiple base models with different model characteristics, and the aforementioned model characteristics include at least one of the following: distance characteristics, density characteristics, clustering characteristics, and neural network characteristics.

[0062] Optionally, when the model characteristic is a distance characteristic, the base model includes at least one of the following: a first base model based on the K-nearest neighbor algorithm, and a second base model based on the isolated forest algorithm;

[0063] When the model property is a density property, the base model includes at least one of the following: a third base model based on local anomaly factors, or a fourth base model based on a connectivity anomaly factor algorithm;

[0064] When the model property is a clustering property, the base model includes at least one of the following: Gaussian mixture model, or the fifth base model based on the clustering-based local anomaly factor algorithm;

[0065] When the model characteristics are neural network characteristics, the base model is at least one of the following: the sixth base model based on graph neural networks, or the seventh base model based on generative adversarial networks.

[0066] As an optional implementation, in the technical solution provided in step S202 above, the method may include the following steps S2021-S2022, wherein,

[0067] Step S2021: For each group of first-layer base models, the first structured data is detected and processed by each base model in the first-layer base model to obtain multiple first anomaly scores corresponding to the first structured data.

[0068] Step S2022: The first anomaly score set is composed of multiple first anomaly scores from the first structured data.

[0069] The above steps can be understood as dividing the first-layer base model into the following groups according to the characteristics of the model.

[0070] For example, the first basis model of the KNN algorithm based on Minkowski distance, the third basis model based on local anomaly factor, the fifth basis model of the local anomaly factor algorithm based on clustering, and the seventh basis model based on generative adversarial networks are used as a set of first-layer basis models. In the first basis model KNN based on Minkowski distance, n_neighbors = 10 and leaf_size = 30 can be set; in the third basis model LOF based on density, n_neighbors = 20 can be set; in the fifth basis model CBLOF based on clustering, n_clusters = 8, α = 0.9, and β = 5 can be set; and in the seventh basis model ALAD based on generative adversarial neural networks, epochs = 10 can be set.

[0071] The first base model of the KNN algorithm based on Manhattan distance, the fourth base model based on the connectivity anomaly factor algorithm, the Gaussian mixture model, and the sixth base model based on the graph neural network are used as another set of first-layer base models. In the first base model KNN based on Manhattan distance, n_neighbors = 10 can be set; in the fourth base model COF based on the connectivity anomaly factor algorithm, n_neighbors = 10 can be set; in the clustering-based Gaussian mixture model GMM, n_components = 1 and init_params = 'kmeans' can be set; and in the sixth base model LUNAR based on the graph neural network, n_neighbors = 5 and n_epochs = 10 can be set.

[0072] These two sets of first-layer base models can perform anomaly detection on each first-structured data (i.e., each first-structured data is used as an input feature) in the first dataset in parallel and independently, and obtain two sets of anomaly score sets corresponding to the first-structured data. Each set of anomaly score sets includes four anomaly scores. These anomaly scores can form the first anomaly score set corresponding to the first-structured data. These data will be used as the input features of the second-layer base model and processed in the next step.

[0073] Step S203: The multiple sets of first anomaly scores corresponding to multiple first structured data are fused using the pre-trained second-layer base model to obtain multiple sets of second anomaly scores.

[0074] Each set of second anomaly scores includes a subset of second anomaly scores used to reflect the degree of anomaly in the first structured data.

[0075] As an optional implementation, the second-layer basis model can be a minimum covariance determinant model. Therefore, the method may include the following steps S2031-S2032, wherein...

[0076] Step S2031: Normalize the multiple sets of first anomaly scores corresponding to each first structured data to obtain normalized multiple sets of first anomaly scores.

[0077] Step S2032: The multiple sets of first anomaly score sets corresponding to each of the normalized first structured data are sequentially input into the second base model as multiple input features to obtain the multiple sets of second anomaly score sets output by the second base model.

[0078] In this embodiment, the first anomaly score set corresponding to the first structured data calculated in step S202 above is first normalized according to the following formula:

[0079]

[0080] Among them, y i This represents the normalized value of the first outlier score of the first structured data output by each base model; min(x) i ) represents the minimum outlier score of the first structured data output by each base model; max(x) i The value of the first outlier score in the first structured data output by each base model represents the maximum value. The MIN-MAX normalization method described above can normalize the first outlier score to between 0 and 1.

[0081] Next, the normalized values ​​of the multiple sets of first anomaly scores corresponding to the first structured data output by the first base model are used as input features and sequentially input into the minimum covariance algorithm model based on the linear model to obtain the multiple sets of second anomaly scores output by the model after fusion. The multiple sets of second anomaly scores include multiple subsets of second anomaly scores corresponding to the first structured data.

[0082] The purpose of the above fusion steps is to sort the multiple subsets of second abnormal scores within each set of second abnormal scores in order to more quickly locate and identify the most obvious outliers.

[0083] Step S204: Determine the absolute ranking score and relative ranking score of each first structured data based on multiple sets of second abnormal score sets.

[0084] The absolute ranking score reflects the absolute rank of the subset of second outlier scores corresponding to the first structured data within the corresponding second outlier score set. Therefore, the absolute ranking score of the first structured data can be determined as follows:

[0085] For each first structured data set, the Borda Count algorithm is used to determine the absolute ranking sub-score of each second outlier score subset based on its ranking within the corresponding second outlier score subset. The absolute ranking score of the first structured data set is then determined based on the absolute ranking sub-scores of multiple sets of second outlier score subsets.

[0086] Specifically, taking the output of the second-layer base model as an example of two sets of second abnormal score sets, with each set containing four abnormal scores, the determination of the above absolute ranking value is explained simply.

[0087] The two sets of outlier scores output from the second-layer base model are sorted group by group, and the outlier score sets corresponding to the first dataset are ranked according to their scores. A score is assigned to each of the two outlier score subsets of the first structured data based on the ranking order. Within each outlier score subset, the highest score is given to the highest-ranked subset, the second-highest score is the first-ranked score minus 1, and so on, with the lowest score being the last-ranked subset. Finally, the absolute ranking sub-value of each second outlier score subset within its corresponding second outlier score set can be obtained through this absolute ranking. Then, the absolute ranking score corresponding to the first structured data is determined using the calculated absolute ranking sub-value.

[0088] For example, in a dataset consisting of N (N=25) structured data points, a subset of outlier scores for a certain structured data point i, ranked from highest to lowest, are 53, 60, 71, and 73. Using the Borda Count algorithm, these subsets are assigned corresponding scores of 47, 40, 29, and 27 respectively. The absolute ranking sub-value obtained by summing these scores is denoted as Bim1. i Another subset of outlier scores in the structured data X, ranked from highest to lowest, are 46, 51, 53, and 62. Using the Borda Count algorithm, these are assigned corresponding scores of 54, 49, 47, and 38 respectively. The absolute ranking value obtained by summing these scores is denoted as Bim2. i Combining the two scores calculated above, the absolute ranking score corresponding to structured data i is calculated using the following formula:

[0089]

[0090] Where N represents the number of data points in the dataset containing structured data i.

[0091] Furthermore, the relative ranking score reflects the relative ranking of the second outlier subset corresponding to the first structured data relative to other second outlier subsets within the second outlier subset. Therefore, the relative ranking score of the first structured data can be determined as follows:

[0092] For each first structured data, based on the size relationship between the second abnormal score subset corresponding to the first structured data and other second abnormal score subsets within the second abnormal score subset, the Copeland algorithm is used to determine multiple winning scores and multiple negative scores for each first structured data, and the relative ranking score of the first structured data is determined based on the multiple winning scores and multiple negative scores of the first structured data.

[0093] Similarly, taking the example of the second-layer base model outputting two sets of second abnormal score sets, with each set containing four abnormal scores, we will briefly explain the process of determining the relative ranking values.

[0094] First, two zero-based matrices, wins and losses, are initialized to store the win / loss relationships (i.e., relative ranking results) between datasets. Then, the two sets of outlier score sets output by the second-layer base model are processed as groups. For each outlier score set, it is determined whether each outlier score in subset i of the first structured data is higher or lower than any other outlier score in subset j of the same group. If higher, matrices wins[i,j] and losses[j,i] are incremented by 1; if lower, the opposite is true. Through this process, matrices wins and losses are obtained, and multiple wins and losses for the first structured data are calculated using the following formula:

[0095] Win_Score i =∑wins[i,:]

[0096] Loss_Score i =∑losses[i,:]

[0097] Furthermore, using the multiple wins and losses from the first structured data obtained above, their relative ranking scores are calculated according to the following formula:

[0098]

[0099] Step S205: When the average of the absolute ranking score and the relative ranking score of the first structured data is greater than a preset score threshold, the first structured data is determined to be abnormal data.

[0100] As an optional implementation, after obtaining the absolute and relative ranking scores of the first structured data, the absolute and relative ranking scores of the first structured data can first be normalized; then, the average score of the normalized absolute and relative ranking scores is calculated, and the first structured data is determined to be abnormal data when the average score is not less than the score threshold.

[0101] In this embodiment, the absolute ranking score rank_scores_borda calculated in step S204 above is first... i and relative ranking score rank_scores_copeland i The MIN-MAX normalization formula was used for normalization, and the average score of each first structured data point was calculated according to the following formula:

[0102] Si =α×scores_borda i +(1-α)×scores_copeland i

[0103] Where α represents the average parameter, and in this embodiment, α is preferably 0.7, scores_borda i and scores_copeland i These represent the absolute ranking scores: rank_scores_borda i and relative ranking score rank_scores_copeland i The result after MIN-MAX normalization (the result is between 0 and 1). Finally, based on the average score S calculated above... i The relationship between the average score S and the preset score threshold is as follows: i If the score exceeds the threshold, the first structured data is determined to be abnormal data; otherwise, it is normal data.

[0104] Alternatively, after obtaining the average score of each first structured data, a preset number or preset proportion of the first structured data with the highest average score can be selected as the abnormal data.

[0105] For example, when performing anomaly detection on structured data of salesperson behavior, the 3,000 data points with the highest average scores in the first structured data can be selected to form risk point assignment data for abnormal salesperson operation behavior, based on actual business needs. These data points can be used as risk points for "salesperson abnormal operation risk mining" and assigned for verification on the risk prevention and control big data platform, so that relevant technical personnel can find the cause of the risk in a timely manner and provide feedback.

[0106] Based on the scheme defined in steps S201 to S205 above, it can be understood that in this embodiment, a multi-layer model is used to mine structured data, fusing the anomaly detection capabilities of multiple anomaly detection models. Specifically, the first-layer base model selects an anomaly detection model from distance-based, density-based, clustering-based, and neural network-based models respectively as the base model, outputting anomaly scores for each structured data point to obtain a first set of anomaly scores. In the second layer, the minimum covariance determinant model of the ensemble learning algorithm is used to perform a first fusion of the meta-features learned in the first-layer base model (i.e., the first set of anomaly scores for each structured data point), effectively integrating the information from the first-layer base model to obtain multiple sets of second anomaly scores. Then, in the third layer, the multiple sets of second anomaly scores are fused and sorted to obtain the mean anomaly score of the structured data, which is then used to determine the anomaly data. This multi-layer fusion method avoids the limitations of existing single-type models or single-time fusion, improves the overall stability and generalization ability of the three-layer model's two-time fusion, and enhances the ability to identify anomalies in structured data.

[0107] Secondly, in the third layer, instead of directly adding or subtracting the multiple sets of second anomaly scores output by the second-layer base model, the average result of two ranking fusion methods is used: the α-average of the absolute ranking score obtained by the Borda Count method and the relative ranking score obtained by the Copeland method. This is because the Borda Count method focuses on the absolute position ranking (level) of the second anomaly score subsets within each set of second anomaly scores, but ignores their relative differences; while the Copeland method focuses on the relative position relationship (comparison) of the second anomaly score subsets within each set of second anomaly scores, but ignores their absolute differences. In this embodiment, by combining these two ranking methods, the characteristics of both methods can be taken into account, avoiding the limitations of a single ranking method. It makes full use of the different perspectives and calculation methods of these two ranking methods in anomaly detection, improves the generalization ability, and enhances the overall accuracy and robustness, thus achieving more effective anomaly identification.

[0108] Furthermore, the calculated anomaly scores are normalized multiple times in this embodiment to avoid large fluctuations in anomaly scores for a single model on structured data, which could excessively affect the final anomaly detection results. Therefore, the technical solution of this embodiment effectively solves the technical problems of high cost and low accuracy associated with manual anomaly detection.

[0109] For example, taking the authoritative public datasets thyroid (the Garavan Institute's thyroid disease detection dataset published by the University of California, Irvine) and personal loan default detection dataset (the Zhongyuan Bank personal loan default detection dataset published by the China Computer Federation) as examples, we compare the single-element single-fusion models, such as the tree-based IFOREST forest model, the distance-based KNN model, the density-based LOF and COF models, the clustering-based GMM model, and the neural network-based ALAD and LUNAR models, with the methods provided in the embodiments of this application. The results are shown in Table 1 below (AUC value comparison table) and Table 2 below (anomaly data accuracy (recall rate) comparison table).

[0110] Table 1

[0111]

[0112]

[0113] Table 2

[0114]

[0115] As can be seen from Tables 1 and 2 above, for the two authoritative public datasets, the AUC value and outlier sample accuracy (recall) of the multi-quality multi-fusion method provided in this application are significantly improved compared to single-model methods based on tree models, distance, density, clustering, and neural networks, as well as methods using MCD that only perform a single blending fusion. In other words, the ensemble learning method with three-layer model and two-fold fusion provided in the embodiments of this application can show good results.

[0116] Example 2

[0117] Based on Embodiment 1 of this application, an embodiment of an abnormal data detection device is also provided, which executes the abnormal data detection method described in the above embodiment when running. Figure 3 This is a schematic diagram of an optional abnormal data detection device according to an embodiment of this application, such as... Figure 3 As shown, the abnormal data detection device includes at least an acquisition module 31, an anomaly detection module 32, a fusion processing module 33, a determination module 34, and an anomaly judgment module 35, wherein:

[0118] The acquisition module 31 is used to acquire the first dataset to be detected, wherein the first dataset includes at least one first structured data.

[0119] Anomaly detection module 32 is used to detect and process the first dataset through multiple pre-trained first-layer base models to obtain multiple sets of first anomaly scores corresponding to each first structured data. Each set of first-layer base models includes multiple base models with different model characteristics.

[0120] The fusion processing module 33 is used to fuse multiple sets of first anomaly scores corresponding to multiple first structured data through a pre-trained second-layer base model to obtain multiple sets of second anomaly scores. Each set of second anomaly scores includes a subset of second anomaly scores that reflects the degree of anomaly of the first structured data.

[0121] The determination module 34 is used to determine the absolute ranking score and relative ranking score corresponding to each first structured data based on multiple sets of second abnormal score sets. The absolute ranking score is used to reflect the absolute ranking of the second abnormal score subset corresponding to the first structured data within the corresponding second abnormal score set, and the relative ranking score is used to reflect the relative ranking of the second abnormal score subset corresponding to the first structured data relative to other second abnormal score subsets within the second abnormal score set.

[0122] The anomaly detection module 35 is used to determine that the first structured data is abnormal data when the average of the absolute sort score and the relative sort score corresponding to the first structured data is greater than a preset score threshold.

[0123] It should be noted that each module in the above-mentioned abnormal data detection device can be a program module (for example, a set of program instructions that implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0124] Example 3

[0125] According to an embodiment of this application, a non-volatile storage medium is also provided, which stores a program, wherein when the program runs, it controls the device where the non-volatile storage medium is located to execute the abnormal data detection method in Embodiment 1.

[0126] Optionally, the device containing the non-volatile storage medium performs the following steps by running this program:

[0127] Step S201: Obtain the first dataset to be detected, wherein the first dataset includes at least one first structured data;

[0128] Step S202: The first dataset is processed by multiple pre-trained first-layer base models to obtain multiple sets of first anomaly score sets corresponding to each first structured data. Each set of first-layer base models includes multiple base models with different model characteristics.

[0129] Step S203: The multiple sets of first anomaly scores corresponding to multiple first structured data are fused through the pre-trained second-layer base model to obtain multiple sets of second anomaly scores. Each set of second anomaly scores includes a subset of second anomaly scores that reflects the degree of anomaly of the first structured data.

[0130] Step S204: Determine the absolute ranking score and relative ranking score of each first structured data based on multiple sets of second abnormal score sets. The absolute ranking score is used to reflect the absolute ranking of the second abnormal score subset corresponding to the first structured data within the corresponding second abnormal score set, and the relative ranking score is used to reflect the relative ranking of the second abnormal score subset corresponding to the first structured data relative to other second abnormal score subsets within the second abnormal score set.

[0131] Step S205: When the average of the absolute ranking score and the relative ranking score of the first structured data is greater than a preset score threshold, the first structured data is determined to be abnormal data.

[0132] According to an embodiment of this application, a processor is also provided for running a program, wherein the abnormal data detection method in embodiment 1 is executed during program execution.

[0133] Optionally, the program executes the following steps during runtime:

[0134] Step S201: Obtain the first dataset to be detected, wherein the first dataset includes at least one first structured data;

[0135] Step S202: The first dataset is processed by multiple pre-trained first-layer base models to obtain multiple sets of first anomaly score sets corresponding to each first structured data. Each set of first-layer base models includes multiple base models with different model characteristics.

[0136] Step S203: The multiple sets of first anomaly scores corresponding to multiple first structured data are fused through the pre-trained second-layer base model to obtain multiple sets of second anomaly scores. Each set of second anomaly scores includes a subset of second anomaly scores that reflects the degree of anomaly of the first structured data.

[0137] Step S204: Determine the absolute ranking score and relative ranking score of each first structured data based on multiple sets of second abnormal score sets. The absolute ranking score is used to reflect the absolute ranking of the second abnormal score subset corresponding to the first structured data within the corresponding second abnormal score set, and the relative ranking score is used to reflect the relative ranking of the second abnormal score subset corresponding to the first structured data relative to other second abnormal score subsets within the second abnormal score set.

[0138] Step S205: When the average of the absolute ranking score and the relative ranking score of the first structured data is greater than a preset score threshold, the first structured data is determined to be abnormal data.

[0139] According to an embodiment of this application, an electronic device is also provided, wherein the electronic device includes one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are configured to run the programs, wherein the programs are configured to execute the abnormal data detection method in Embodiment 1 above during runtime.

[0140] Optionally, the processor is configured to execute the following steps via a computer program:

[0141] Step S201: Obtain the first dataset to be detected, wherein the first dataset includes at least one first structured data;

[0142] Step S202: The first dataset is processed by multiple pre-trained first-layer base models to obtain multiple sets of first anomaly score sets corresponding to each first structured data. Each set of first-layer base models includes multiple base models with different model characteristics.

[0143] Step S203: The multiple sets of first anomaly scores corresponding to multiple first structured data are fused through the pre-trained second-layer base model to obtain multiple sets of second anomaly scores. Each set of second anomaly scores includes a subset of second anomaly scores that reflects the degree of anomaly of the first structured data.

[0144] Step S204: Determine the absolute ranking score and relative ranking score of each first structured data based on multiple sets of second abnormal score sets. The absolute ranking score is used to reflect the absolute ranking of the second abnormal score subset corresponding to the first structured data within the corresponding second abnormal score set, and the relative ranking score is used to reflect the relative ranking of the second abnormal score subset corresponding to the first structured data relative to other second abnormal score subsets within the second abnormal score set.

[0145] Step S205: When the average of the absolute ranking score and the relative ranking score of the first structured data is greater than a preset score threshold, the first structured data is determined to be abnormal data.

[0146] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0147] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0149] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0151] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0152] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An abnormal data detection method, characterized in that, include: Obtain a first dataset to be detected, wherein the first dataset includes at least one first structured data; The first dataset is processed by multiple pre-trained first-layer base models to obtain multiple sets of first anomaly scores corresponding to each first structured data. Each set of first-layer base models includes multiple base models with different model characteristics, and the model characteristics include at least one of the following: distance characteristics, density characteristics, clustering characteristics, and neural network characteristics. The pre-trained second-layer base model is used to fuse multiple sets of first anomaly scores corresponding to multiple first structured data to obtain multiple sets of second anomaly scores. Each set of second anomaly scores includes a subset of second anomaly scores that reflects the degree of anomaly of the first structured data. The second-layer base model is a minimum covariance determinant model. The absolute ranking score and relative ranking score of each first structured data are determined based on multiple sets of second abnormal score sets. The absolute ranking score is used to reflect the absolute ranking of the second abnormal score subset corresponding to the first structured data within the corresponding second abnormal score set, and the relative ranking score is used to reflect the relative ranking of the second abnormal score subset corresponding to the first structured data relative to other second abnormal score subsets within the second abnormal score set. When the average of the absolute ranking score and the relative ranking score of the first structured data is greater than a preset score threshold, the first structured data is determined to be abnormal data. Wherein, when the model characteristic is the distance characteristic, the base model includes at least one of the following: a first base model based on the K-nearest neighbor algorithm, a second base model based on the isolated forest algorithm; when the model characteristic is the density characteristic, the base model includes at least one of the following: a third base model based on the local anomaly factor, a fourth base model based on the connectivity anomaly factor algorithm; when the model characteristic is the clustering characteristic, the base model includes at least one of the following: a Gaussian mixture model, a fifth base model based on the local anomaly factor algorithm of clustering; when the model characteristic is the neural network characteristic, the base model includes at least one of the following: a sixth base model based on a graph neural network, a seventh base model based on a generative adversarial network.

2. The method according to claim 1, characterized in that, Obtain the first dataset to be detected, including: Obtain the second dataset to be detected; The second dataset is preprocessed to obtain the first dataset, wherein the preprocessing includes at least one of the following: data cleaning, feature encoding, and standardization.

3. The method according to claim 1, characterized in that, The first dataset is processed by pre-trained multiple sets of first-layer base models to obtain multiple sets of first anomaly score sets corresponding to each of the first structured data, including: For each group of the first layer base models, the first structured data is detected and processed by each of the base models in the first layer base models to obtain multiple first anomaly scores corresponding to the first structured data; The first anomaly score set of the first structured data is composed of multiple first anomaly scores of the first structured data.

4. The method according to claim 1, characterized in that, Multiple sets of first anomaly scores corresponding to multiple sets of the first structured data are fused using a pre-trained second-layer base model to obtain multiple sets of second anomaly scores, including: Normalize the multiple sets of the first anomaly scores corresponding to each of the first structured data to obtain the normalized multiple sets of the first anomaly scores; The multiple sets of the first anomaly scores corresponding to each of the normalized first structured data are sequentially input into the second base model as multiple input features to obtain multiple sets of the second anomaly scores output by the second base model.

5. The method according to claim 1, characterized in that, Based on multiple sets of the second anomaly score sets, the absolute ranking score and relative ranking score corresponding to each of the first structured data are determined, including: For each of the first structured data, the Borda Count algorithm is used to determine the absolute ranking sub-score of each of the second abnormal score subsets based on the ranking of the second abnormal score subset corresponding to the first structured data within the corresponding second abnormal score subset, and the absolute ranking score of the first structured data is determined based on the absolute ranking sub-scores of multiple sets of the second abnormal score subsets. For each of the first structured data, based on the size relationship between the second abnormal score subset corresponding to the first structured data and other second abnormal score subsets within the second abnormal score subset, the Copeland algorithm is used to determine multiple winning scores and multiple negative scores for each of the first structured data, and the relative ranking score of the first structured data is determined based on the multiple winning scores and multiple negative scores of the first structured data.

6. The method according to claim 5, characterized in that, When the average score of the absolute ranking score and the relative ranking score corresponding to the first structured data is greater than a preset score threshold, the first structured data is determined to be abnormal data, including: The absolute and relative ranking scores of the first structured data are normalized. Calculate the average score of the normalized absolute ranking score and the relative ranking score, and determine the first structured data as the abnormal data when the average score is not less than the score threshold.

7. An abnormal data detection device, characterized in that, include: An acquisition module is used to acquire a first dataset to be detected, wherein the first dataset includes at least one first structured data. An anomaly detection module is used to detect and process the first dataset by using multiple pre-trained first-layer base models to obtain multiple sets of first anomaly scores corresponding to each first structured data. Each set of first-layer base models includes multiple base models with different model characteristics, and the model characteristics include at least one of the following: distance characteristics, density characteristics, clustering characteristics, and neural network characteristics. The fusion processing module is used to fuse multiple sets of first anomaly scores corresponding to multiple first structured data through a pre-trained second-layer base model to obtain multiple sets of second anomaly scores. Each set of second anomaly scores includes a subset of second anomaly scores that reflects the degree of anomaly of the first structured data. The second-layer base model is a minimum covariance determinant model. The determination module is used to determine the absolute ranking score and relative ranking score corresponding to each of the first structured data based on multiple sets of the second abnormal score sets, wherein the absolute ranking score is used to reflect the absolute ranking of the second abnormal score subset corresponding to the first structured data within the corresponding second abnormal score set, and the relative ranking score is used to reflect the relative ranking of the second abnormal score subset corresponding to the first structured data relative to other second abnormal score subsets within the second abnormal score set. The anomaly detection module is used to determine that the first structured data is abnormal data when the average of the absolute ranking score and the relative ranking score corresponding to the first structured data is greater than a preset score threshold. Wherein, when the model characteristic is the distance characteristic, the base model includes at least one of the following: a first base model based on the K-nearest neighbor algorithm, a second base model based on the isolated forest algorithm; when the model characteristic is the density characteristic, the base model includes at least one of the following: a third base model based on the local anomaly factor, a fourth base model based on the connectivity anomaly factor algorithm; when the model characteristic is the clustering characteristic, the base model includes at least one of the following: a Gaussian mixture model, a fifth base model based on the local anomaly factor algorithm of clustering; when the model characteristic is the neural network characteristic, the base model includes at least one of the following: a sixth base model based on a graph neural network, a seventh base model based on a generative adversarial network.

8. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a computer program, wherein the device containing the non-volatile storage medium executes the abnormal data detection method according to any one of claims 1 to 6 by running the computer program.

9. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, executes the abnormal data detection method according to any one of claims 1 to 6.

Images