Structured data anomaly detection method and device

Abstract

The invention discloses a structured data anomaly detection method and device. The method comprises the steps that multiple pieces of structured data in a to-be-detected data set are acquired; for each piece of structured data, detecting the structured data by using multiple groups of first-layer base models to obtain a first anomaly detection score output by each group of first-layer base models, each group of first-layer base models including a base model based on distance, density and a neural network; the first anomaly detection score output by each group of first-layer basis models is analyzed by using a second-layer basis model to obtain a corresponding second anomaly detection score, and the second-layer basis model comprises a basis model based on a support vector machine and a basis model based on a local outlier factor algorithm; and determining abnormal structured data in the to-be-detected data set according to the second abnormal detection score. According to the method and the device, the technical problems of relatively high manpower and time cost and relatively low accuracy when the abnormal structured data is detected in a manual mode are solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a method and device for detecting anomalies in structured data. Background Art

[0002] With the continuous development of science and technology, the scale of data is getting larger and larger, and the data is becoming more and more complex. Data is no longer just unstructured data such as files, images and texts, but also includes a large amount of unstructured data. Therefore, anomaly detection of structured data is an important research direction in the field of data mining.

[0003] At present, most of the anomaly detection of structured data adopts the method of artificial intelligence machine learning, and this method is generally based on statistics or models, among which:

[0004] Statistical anomaly detection methods include but are not limited to: 3σ method based on normal distribution, Z-score method based on standard score, Figure 4 The boxplot method of quantile range and the Grubbs method based on hypothesis testing, etc. However, these methods are based on the relevant principles of statistics, and consider the data distribution that does not meet certain statistical rules as abnormal. They lack effective learning of data and are often not accurate enough to meet most production requirements.

[0005] Model-based anomaly detection methods include but are not limited to: models based on distance, density, neural network, tree model, etc. However, these model-based anomaly detection methods all build models for data samples from a single aspect, without using ensemble learning to effectively fuse multiple fusion results, unable to give full play to the characteristics of multiple model methods, and often have a single screening dimension. Even if ensemble learning is used, it is often a single ensemble learning (such as stacking fusion). Single ensemble or single fusion cannot avoid the limitations of the model. The generalization ability of ensemble fusion is limited, and the ability to identify abnormal samples is limited.

[0006] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0007] The embodiments of the present application provide a structured data anomaly detection method and device to at least solve the technical problems of high manpower and time costs and low accuracy when detecting abnormal structured data manually.

[0008] According to one aspect of an embodiment of the present application, a method for detecting anomalies in structured data is provided, comprising: obtaining multiple structured data in a data set to be detected; for each structured data, using multiple groups of pre-trained first-layer base models to detect and analyze the structured data, and obtaining a first anomaly detection score output by each group of first-layer base models, wherein each group of first-layer base models includes a distance-based base model, a density-based base model, and a neural network-based base model; using a pre-trained second-layer base model to analyze and process the first anomaly detection score output by each group of first-layer base models, and obtain a corresponding second anomaly detection score, wherein the second-layer base model includes a support vector machine-based base model and a local outlier factor algorithm-based base model; and determining the abnormal structured data in the data set to be detected based on the second anomaly detection score of each structured data.

[0009] Optionally, obtaining multiple structured data in the data set to be detected includes: obtaining the data set to be detected; preprocessing the data set to be detected to obtain multiple initial structured data, wherein the preprocessing includes at least one of the following: data cleaning, feature encoding; performing Gaussian rank normalization on the multiple initial structured data to obtain corresponding multiple structured data.

[0010] Optionally, multiple initial structured data are subjected to Gaussian rank normalization processing to obtain corresponding multiple structured data, including: counting the values ​​of data in the multiple initial structured data according to a preset column dimension to determine the element type under each column dimension; when the element type is a single-value type, all element values ​​under the column dimension are converted to a first fixed value to obtain normalized element values; when the element type is a binary type, two types of element values ​​under the column dimension are respectively converted to a second fixed value or a third fixed value to obtain two types of normalized element values; when the element type is a multi-value type, the value type of each type of element value under the column dimension is determined, and each type of element value is converted according to the value type to obtain the normalized multiple types of element values, wherein the value type includes at least: numerical data, ordered category data, non-numerical data, empty data or unordered category data; each data in the multiple initial structured data is converted according to the processed element values ​​of each type to obtain multiple structured data.

[0011] Optionally, each category of element values ​​is converted according to the value type to obtain normalized multiple categories of element values, including: for each category of element values ​​under the column dimension, when the value type is numerical data or ordered categorical data, the first occurrence number of each category of element values ​​and the second occurrence number of element values ​​under the corresponding column dimension are counted, and the linear division vector of the element value is determined based on the first occurrence number and the second occurrence number; the linear division vector is transformed according to the Chebyshev polynomial, and in combination with a preset scaling factor, the quantile of the Gaussian distribution of the transformed linear division vector is inversely solved using the inverse function of the error function; the quotient of the product of the second occurrence number and the quantile and the first occurrence number is used as the weighted mean of the element value, and the quantile minus the weighted mean is used as the normalized element value; when the value type is non-numerical data or empty data or unordered categorical data, the element value is mapped using the cubic spline interpolation algorithm to obtain the normalized element value.

[0012] Optionally, each group of first-layer base models also includes a random forest network, wherein the pre-trained multiple groups of first-layer base models are used to detect and analyze the structured data to obtain a first anomaly detection score output by each group of first-layer base models, including: for each group of first-layer base models, each base model in the first-layer base model is used to detect and analyze the structured data to obtain anomaly detection scores output by each base model; the anomaly detection scores output by each base model are normalized, and the normalized anomaly detection scores are analyzed using a random forest network to obtain a corresponding first anomaly detection score.

[0013] Optionally, the base model based on the distance algorithm includes at least one of the following: a base model based on the K-nearest neighbor algorithm, and a base model based on the angle outlier detection algorithm; the base model based on the density algorithm includes at least one of the following: a base model based on the local anomaly factor algorithm, and a base model based on the local benefit factor algorithm; the base model based on the neural network includes at least one of the following: a base model based on the anomaly ranking algorithm based on the learnable unified neighbor, and a base model based on the bad learning anomaly detection algorithm.

[0014] Optionally, the second-layer base model also includes a random forest network, wherein the pre-trained second-layer base model is used to analyze and process the first anomaly detection scores output by each group of first-layer base models to obtain a corresponding second anomaly detection score, including: inputting the first anomaly detection scores output by each group of first-layer base models into the base model based on a support vector machine and the base model based on a local outlier factor algorithm in the second-layer base model, and obtaining the anomaly detection scores output by the base model based on the support vector machine and the base model based on the local outlier factor algorithm; using the random forest network to analyze the anomaly detection scores output by the base model based on the support vector machine and the base model based on the local outlier factor algorithm to obtain the corresponding second anomaly detection score.

[0015] Optionally, abnormal structured data in the data set to be detected is determined based on the second anomaly detection score of each structured data, including: traversing each structured data in the data set to be detected, and judging the relationship between the second anomaly detection score of the structured data and a preset threshold value; when the second anomaly detection score of the structured data is not lower than the threshold value, determining that the structured data is abnormal structured data.

[0016] According to another aspect of an embodiment of the present application, a structured data anomaly detection device is also provided, including: an acquisition module, used to acquire multiple structured data in a data set to be detected; an analysis module, used to detect and analyze the structured data for each structured data using multiple pre-trained first-layer base models, to obtain a first anomaly detection score output by each group of first-layer base models, wherein each group of first-layer base models includes a distance-based base model, a density-based base model, and a neural network-based base model; using a pre-trained second-layer base model to analyze and process the first anomaly detection score output by each group of first-layer base models to obtain a corresponding second anomaly detection score, wherein the second-layer base model includes a support vector machine-based base model and a local outlier factor algorithm-based base model; a detection module, used to determine the abnormal structured data in the data set to be detected based on the second anomaly detection score of each structured data.

[0017] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned structured data anomaly detection method by running the computer program.

[0018] In an embodiment of the present application, multiple structured data in a data set to be detected are obtained; for each structured data, multiple groups of pre-trained first-layer base models are used to detect and analyze the structured data to obtain the first anomaly detection score output by each group of first-layer base models, wherein each group of first-layer base models includes a base model based on distance, a base model based on density, and a base model based on a neural network; the first anomaly detection score output by each group of first-layer base models is analyzed and processed using a pre-trained second-layer base model to obtain a corresponding second anomaly detection score, wherein the second-layer base model includes a base model based on a support vector machine and a base model based on a local outlier factor algorithm; the abnormal structured data in the data set to be detected is determined based on the second anomaly detection score of each structured data. Thus, the advantages of various models are fully utilized, the limitations and errors that may be caused by a single model are reduced, the stability and generalization ability of the overall model are improved, and the overall recognition ability of abnormal samples is improved. In addition, the technical problem of high manpower and time costs and low accuracy required for detecting abnormal structured data by manual methods is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a schematic diagram of the structure of an optional computer terminal (or mobile device) according to an embodiment of the present application;

[0021] Figure 2 is a flow chart of an optional structured data anomaly detection method according to an embodiment of the present application;

[0022] Figure 3 It is a schematic diagram of the structure of an optional structured data anomaly detection device according to an embodiment of the present application.

[0023] Figure 4 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] In addition, the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0027] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0028] Structured data: refers to data that is stored and processed in a certain format and organization. It usually exists in the form of tables, databases, XML, JSON, etc., and has clear relationships between fields and values. Therefore, it can also be understood as regular data that can be represented using relational data or logically expressed in the form of a two-dimensional table.

[0029] Gaussian rank normalization (RankGauss) is a novel feature transformation method, similar to normalization (MinMax) and standardization (Standardization), both of which are designed to make the model better fit the data.

[0030] Chebyshev Polynomials: A series of orthogonal polynomials related to de Moivre's Formula, defined recursively. Usually the first kind of Chebyshev polynomials are represented by the symbol T n The second kind of Chebyshev polynomial is represented by the symbol U n Indicates that the Chebyshev polynomial T n or U n Represents a polynomial of order n.

[0031] Cubic Spline interpolation, referred to as Spline interpolation, is a smooth curve through a series of shape value points. Mathematically, it is a process of solving the three-bend matrix equation group to obtain the curve function group. The interpolation process can be understood as dividing the interval [a,b] into n intervals [(x0,x1), (x1,x2),…, (x n-1 ,x n )], where the interval includes n+1 points, of which the two endpoints x0=a,x n = b, and the curve between each small section is a cubic spline equation, and each cubic equation satisfies the following conditions: (1) In each segmented small section (x i ,x i+1), the corresponding cubic spline equation S i (x) are all cubic equations; (2) satisfy the interpolation condition, that is, S(x i )=y i (i=0,1,…,n); (3) The curve is smooth, that is, S(x i ),S′(x i ),S″(x i )continuous.

[0032] The K-Nearest Neighbor (KNN) algorithm, also known as the nearest neighbor algorithm, is a basic classification and regression algorithm. Its basic idea is to find the k training samples closest to the sample to be classified, and determine the category of the sample to be classified according to the classification of these k samples. The steps of the KNN algorithm are as follows: calculate the distance between the sample to be classified and the training sample (generally using Euclidean distance or Manhattan distance, etc.); select k training samples with the closest distance; vote according to the category of these k samples, and classify the sample to be classified into the category with the most votes; output the category of the sample to be classified. In summary, the advantages of the KNN algorithm include being simple to understand, easy to implement, and suitable for multi-classification problems. However, the KNN algorithm also has some disadvantages, including requiring a lot of computing time, may produce misleading results for unbalanced data sets, and may produce feature disasters for high-dimensional data.

[0033] Local Outlier Factor (LOF) algorithm: As a density-based outlier detection algorithm, it can be used to find local outliers in a data set. The algorithm calculates the density of neighboring nodes around each data point based on distance and compares it with the density of the area where the point itself is located. If the density of the area where the point is located is lower than the density of the surrounding neighboring points, the point may be an outlier. The core idea of ​​the LOF algorithm is that if the local density of a data point is relatively low and the density of its neighboring points is relatively high, the point is likely to be an outlier. Therefore, by calculating the local outlier factor, outliers in the data set can be found. In summary, the advantage of the LOF algorithm is that the density threshold can be adaptively adjusted and it is suitable for data sets with various density distributions. However, the computational complexity of the LOF algorithm is high, especially when processing large-scale data sets, a large number of distance calculations and sorting operations are required, so the efficiency is low.

[0034] Connectivity-Based Outlier Factor (COF) algorithm: As a density-based outlier detection algorithm, it considers the relative connectivity between data points and applies the connectivity factor to each data point to calculate its outlier score. The COF algorithm is similar to the above LOF method, but the COF algorithm calculates the connectivity between data points, while the LOF algorithm only considers the density of neighboring points around the data point.

[0035] Angle-Base Outlier Detection (ABOD) algorithm: It determines whether a data point is an outlier by calculating the angle between the data point and other points. Therefore, this algorithm uses angle as the metric for outliers, and artificial outliers form a large number of acute angles or obtuse angles between all data points.

[0036] Learnable Unified Neighborhood-based Anomaly Ranking (LUNAR) algorithm: It is a novel anomaly detection method based on Graph Neural Networks (GNN), which learns to use neighbor information to discover anomalies of each node in a trainable way. Among them, the LUNAR model can divide graph data into labeled nodes and unlabeled nodes, use the label information of labeled nodes to train the model, and perform semi-supervised learning through the feature information of unlabeled nodes. In addition, the LUNAR model maps the features of nodes and edges to a low-dimensional representation space through the encoder; then, the low-dimensional representation is remapped to the original feature space through the decoder, and the node representation is learned by minimizing the difference between the original feature and the reconstructed feature. In this way, the LUNAR model can use the label information of labeled nodes and the feature information of unlabeled nodes for node classification and relationship prediction. Experiments have shown that the LUNAR model has achieved good performance on some graph datasets, indicating its potential and advantages in graph data analysis tasks.

[0037] Adversarial Learn Anomaly Detection (ALAD): It uses Generative Adversarial Network (GAN) to learn encoders from data space to latent space in the training phase to achieve efficient detection in the test phase. The basic idea of ​​the algorithm is to learn the distribution of normal samples by training a generator and a discriminator, and use the discriminator to determine whether the input sample is abnormal. Among them, the generator is an autoencoder that maps the input sample to a low-dimensional hidden representation and tries to reconstruct the input sample from this hidden representation; the discriminator is a binary classifier that tries to distinguish the hidden representation output by the generator from the hidden representation extracted from the normal sample. In addition, the training process of the ALAD algorithm is divided into two stages: pre-training and adversarial training, in which: in the pre-training stage, the generator and the discriminator are trained by the loss function of the autoencoder and the binary classifier respectively; in the adversarial training stage, the generator and the discriminator are trained by alternating updates, in which the generator tries to minimize the reconstruction error of the autoencoder and the discriminator tries to maximize the accuracy of the binary classifier. The ALAD algorithm has certain advantages, such as being able to capture complex anomaly patterns and being robust to different types of anomalies. However, the algorithm may have some challenges when processing high-dimensional data and large-scale data, and further improvement and optimization are needed.

[0038] Example 1

[0039] With the widespread popularity of the Internet and the complexity of telecommunications services, the operations of front-line sales staff in major telecommunications business halls involve multiple business scenarios such as recharging funds, handling sales products, refunds, and applying for telecommunications cards, which makes the operation data extremely large. At the same time, the behavioral information data generated by the sales staff when handling related businesses may be abnormal. If traditional manual labeling methods are used to detect abnormal data, the labor cost will be too high, and the accuracy of the results will be difficult to guarantee continuously, thus failing to timely recover corporate losses and regulate corporate development. Therefore, the quality of the abnormal detection results for structured data will seriously affect the healthy development of related corporate businesses.

[0040] At present, the relevant patent (CN202210222958) "Data anomaly detection method and device, medium and electronic device" proposes a data anomaly detection method based on a fusion model, which adopts the stacking machine learning model fusion method, but the fusion level is only two layers at a time, which can be regarded as the integrated fusion of the traditional single stacking machine learning model, and the integrated fusion is a tree-based homogeneous model. Therefore, this scheme lacks the integrated fusion of other heterogeneous model methods, and it is difficult to ensure the accuracy of the anomaly detection results of structured data.

[0041] The related patent (CN202111231657) "A network anomaly detection method integrating GBDT and neural network" proposes a network structured data anomaly detection method that integrates a fusion tree model and a neural network, but the fusion method is to use a tree model and a neural network model to detect anomalies for category features and numerical features respectively, and finally linearly weight the results. This integrated fusion method uses only two model methods, which cannot make more use of the characteristics of various anomaly detection model methods, and the model integration and fusion process is also relatively simple. It just uses different methods for different sample data and then linearly weights the total result. There is a lack of integrated fusion of hierarchical models, and the results of anomaly detection will be relatively mediocre. The traditional standardization method is also used for data standardization. The number and sorting information of the original data are not fully utilized. There is no improvement on the traditional normalization method, and it is difficult to ensure the accuracy of structured data detection.

[0042] The related patent (CN202210718289) "Data detection method, device, computer equipment and storage medium" proposes a model-based data anomaly detection method. This method uses the traditional isolation forest model, but only uses a single anomaly detection tree model without any improvement. It randomly cuts the hyperplane, has limited anomaly recognition ability, and uses traditional standardization methods to standardize the data. The number and sorting information of the original data are not fully utilized, and there is no improvement on the traditional normalization method, making it difficult to ensure the accuracy of structured data detection.

[0043] The related patent (CN202210741524) "A data anomaly detection method, system, electronic device and readable storage medium" proposes an anomaly detection model method based on multi-dimensional data, but the multi-dimensional data based on this method is obtained through different collection methods, and the original data is directly subjected to multi-angle deep encoder and clustering feature calculations, without model integration and fusion. The data is also standardized by traditional standardization methods, and there is no improvement on the traditional methods, making it difficult to ensure the accuracy of structured data detection.

[0044] The related patent (CN202210632838) "Data Anomaly Detection Method" proposes an anomaly detection method based on the confidence of a statistical probability distribution function. However, this method uses a statistical probability distribution function to calculate the confidence to obtain abnormal samples. This method does not use a certain standardization method for the data, and it is difficult to guarantee the accuracy of structured data detection based on statistical detection methods. At the same time, when establishing a probability distribution function, manually annotated label data is required, which requires enterprises to bear a considerable annotation cost.

[0045] Therefore, none of the above-mentioned prior arts discloses an abnormal data monitoring method based on multi-model fusion, so that the existing solutions cannot guarantee the accuracy of the results of abnormal monitoring of structured data. In order to solve this problem, the present application provides a relevant solution in the embodiments, which is described in detail below.

[0046] According to an embodiment of the present application, an embodiment of a structured data anomaly detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a structured data anomaly detection method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0048] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0049] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the structured data anomaly detection method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the structured data anomaly detection method of the above-mentioned application is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0050] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0051] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0052] In the above operating environment, Figure 2 is a flow chart of an optional structured data anomaly detection method according to an embodiment of the present application, such as Figure 2 As shown, the method at least includes steps S202-S206, wherein:

[0053] Step S202: Acquire multiple structured data in the data set to be detected.

[0054] In the technical solution provided in step S202, the structured data refers to data that can be represented and stored through relational data and is in a two-dimensional form, and its characteristics are that the data is in rows, a row of data represents the information of an entity, and the attributes of each row of data are the same. Therefore, structured data includes but is not limited to: telecommunications business behavior information data, financial fraud user financial behavior data, disease detection user disease feature data, network user traffic for network security detection, etc.

[0055] Step S204: for each structured data, use multiple groups of pre-trained first-layer base models to detect and analyze the structured data to obtain a first anomaly detection score output by each group of first-layer base models; use the pre-trained second-layer base model to analyze and process the first anomaly detection score output by each group of first-layer base models to obtain a corresponding second anomaly detection score.

[0056] In the technical solution provided in step S204, each group of the first-layer base models includes multiple base models, and these base models include three types of models: a distance-based base model, a density-based base model, and a neural network-based base model. The second-layer base models include a support vector machine-based base model and a local outlier factor algorithm-based base model.

[0057] Step S206: determining abnormal structured data in the to-be-detected data set according to the second abnormality detection score of each structured data.

[0058] Based on the scheme defined by the above steps S202 to S206, it can be learned that in the embodiment, by improving the traditional two-level single fusion integrated learning architecture, a multi-level model two-layer fusion architecture is adopted for integrated learning, wherein multiple groups of first-layer base models can mine data information from different dimensions; two base models of different natures in the second-layer base model can fuse the results output by multiple groups of first-layer base models, thereby making full use of the advantages of various models, reducing the limitations and errors that may be caused by a single model, improving the stability and generalization ability of the overall model, and improving the overall recognition ability of abnormal samples. It can be seen that through the technical solution of the embodiment of the present application, deep integrated learning and feature fusion of structured data anomaly detection are realized, so as to more comprehensively and accurately identify abnormal data, especially when processing large-scale and complex structured data sets, and achieve the purpose of timely discovering and processing abnormal data, and providing strong support for risk prevention and control, data analysis and other fields.

[0059] The above method of this embodiment is further introduced below.

[0060] As an optional implementation, in the technical solution provided in the above step S202, the multiple structured data may be obtained by the following method, including:

[0061] Step S2021, obtaining the data set to be detected.

[0062] Step S2022: preprocess the data set to be detected to obtain a plurality of initial structured data.

[0063] Step S2023, performing Gaussian rank normalization processing on the multiple initial structured data to obtain corresponding multiple structured data.

[0064] In this embodiment, since the original data set to be detected may include abnormal characters, missing values, and non-numerical features, etc. Therefore, in order to ensure the accuracy of the model analysis results, the original data set to be detected can be preprocessed to obtain multiple initial structured data, wherein the preprocessing operation includes but is not limited to: removing abnormal characters, missing value processing, feature encoding, etc. In addition, in order to avoid numerical extremism, each initial structured data after preprocessing can also be planned.

[0065] Optionally, the embodiment of the present application may perform Gaussian rank normalization processing on multiple initial structured data according to the following method, including:

[0066] Step 1: Count the values ​​of data in multiple initial structured data according to the preset column dimensions, and determine the element type under each column dimension.

[0067] Specifically, the above element type refers to the state of the element value under the corresponding column dimension, which can be divided into: single-value type, binary type, and multi-value type. Among them:

[0068] A single-value type is one in which all records in a column have the same status value, that is, the column has only one unique value. For example, if a column is named "Whether an adult" when processing a data set, but all people in the data set are adults, all records in this column are marked as "yes", then the element type of this column is a single-value type.

[0069] A binary data type refers to a data column that has only two different state values. For example, in the above example of "Is it an adult?", if the data set contains both adults and minors, then the element type of this data column is two values: "yes" and "no". In addition, "yes / no", "true / false", "1 / 0" and so on are all common binary data representation types.

[0070] Multi-value type means that the value recorded in a column of data has two or more state values. For example, if one of the features in the data set is "number of card applications at night", and the value of this feature is distributed in the data set, that is, there are multiple different values, such as 10, 20, 30, etc., then the feature "number of card applications at night" is multi-value data.

[0071] Step 2: When the element type is a single-value type, all element values ​​under the column dimension are converted to the first fixed value to obtain the normalized element value.

[0072] Considering that all values ​​under a certain column dimension are the same, they may be directly ignored or mapped to a fixed value, such as 0. This is because single-valued data does not contribute to model training and prediction.

[0073] Step 3: When the element type is a binary type, the two types of element values ​​under the column dimension are converted into the second fixed value or the third fixed value respectively to obtain the normalized two types of element values.

[0074] For all values ​​under a certain column dimension, there are two states. The two states can be simply mapped, such as mapping the two states to 0 and 1, which helps the model better understand and process this data type.

[0075] Step 4: When the element type is a multi-value type, determine the value types of each of the multiple element values ​​under the column dimension, and map each element value according to the value type to obtain the normalized multiple element values. The above value types include at least: numerical data, ordered categorical data, non-numerical data, empty data, or unordered categorical data.

[0076] Specifically, for a multi-value column dimension, each type of element value therein can be mapped according to the following rules:

[0077] When the value type is numerical data or ordered categorical data, it means that the element value of this type can be directly transformed by nonlinear transformation of Chebyshev polynomials for more uniform data conversion. The specific conversion process includes:

[0078] First, count the first occurrences and the second occurrences of each element value under the corresponding column dimension, and determine the linear equal division vector of the element value according to the first occurrence and the second occurrence, which can be recorded as rankV;

[0079] Next, the linear division vector rankV is nonlinearly transformed according to the Chebyshev polynomial so that the linear division vector rankV is limited to [0,1] to avoid numerical extremes. The specific conversion formula is as follows:

[0080]

[0081] Where n represents the order of the Chebyshev polynomial, c i represents the fixed coefficients of the Chebyshev polynomial, arccos(2*rankV-1) means mapping the linearly divided vector rankV from [0,1] to the range [0,π / 2].

[0082] Then, combined with the preset scaling factor, the inverse function of the error function is used to reversely solve the Gaussian distribution quantile α of the transformed linearly divided vector. The calculation formula is as follows:

[0083] α=scale_factor*ErfInv(rankV)

[0084] Finally, the quotient of the product of the second occurrence number and the quantile and the first occurrence number is taken as the weighted mean of the element value, and the quantile minus the weighted mean is taken as the normalized element value.

[0085] When the value type is non-numeric data or empty data or unordered categorical data, the element value of this type cannot be transformed by nonlinear transformation of Chebyshev polynomial. In this case, the element value can be mapped by cubic spline interpolation algorithm to obtain the normalized element value.

[0086] Step 5: Convert each data in the multiple initial structured data according to the element values ​​of each category after processing to obtain multiple structured data.

[0087] Taking the salesperson behavior data set obtained from a certain telecommunications big data platform in a certain month as the data set to be tested as an example, the data processing process provided by the above steps S2021-S2023 is briefly described.

[0088] First, 32 feature fields are obtained, such as employee ID, BSS channel ID, agent channel ID, agent name, number of low-cost broadband transfer orders, number of nighttime card applications, amount of resale accounts, number of abnormal age card applications, number of small sales product orders, etc., where all these feature fields are numeric fields; then, the 25,841 original structured data corresponding to these 32 feature fields are determined. In order to ensure the accuracy of subsequent abnormal data detection, the 25,841 original structured data need to be preprocessed such as abnormal character removal, missing value filling, and feature encoding. The feature encoding can be unique hot encoding to obtain the processed initial structured data; then, the features of each dimension are respectively extracted into list numerical elements, and are sorted, frequency counted, and numerically mapped (where the order n of the Chebyshev polynomial is set to 5), and the data of each dimension (i.e., each column) is looped through to obtain multiple structured data after normalization.

[0089] As an optional implementation, in the technical solution provided in the above step S204, the base model based on the distance algorithm includes at least one of the following: a base model based on the K-nearest neighbor algorithm, a base model based on the angle-based outlier detection algorithm; the base model based on the density algorithm includes at least one of the following: a base model based on the local anomaly factor algorithm, a base model based on the local benefit factor algorithm; the base model based on the neural network includes at least one of the following: a base model based on the anomaly ranking algorithm based on a learnable unified neighbor, a base model based on the bad learning anomaly detection algorithm.

[0090] Therefore, the embodiment of the present application can use the base model based on the K-nearest neighbor algorithm, the base model based on the local anomaly factor algorithm, and the base model based on the learnable unified neighbor anomaly ranking algorithm as a group of first-layer base models; and use the base model of the angle-based outlier detection algorithm, the base model based on the local interest group factor algorithm, and the base model based on the bad learning anomaly detection algorithm as another group of first-layer base models. These two groups of first-layer base models can perform anomaly detection on each structured data in parallel and independently to obtain two sets of anomaly score sets corresponding to the first structured data, and each set of anomaly score sets includes three anomaly scores. It should be noted that the above combination of base models is only described as an example. The specific combination of three base models of different qualities can be set according to the actual application scenario, and the present application does not impose specific restrictions on this.

[0091] In addition, each set of first-layer base models also includes a random forest network. The random forest model can analyze the two sets of anomaly score sets output by the two sets of first-layer base models to obtain corresponding anomaly scores.

[0092] Therefore, each set of first-layer base models can detect and analyze each structured data according to the following steps:

[0093] Step 1: Use each base model in the first layer of base models to detect and analyze the structured data to obtain the anomaly detection score output by each base model.

[0094] It should be noted that each base model in the multiple groups of first-layer base models is obtained by iterative training using five-fold cross validation. That is to say, the multiple groups of sample data after Gaussian rank normalization are divided into five parts according to the index, and sent to two groups of three base models for training and inference five times. Each time, four of the five parts of data are used for training, and the other one is used for inference. In this way, the result of each inference (i.e., the anomaly detection score of the three base models) is inferred from the base model trained with four of the data to the other one. After five times, the three base models in each group can obtain independent inference results of the complete data (i.e., the anomaly detection score of each primary base model).

[0095] Step 2: Normalize the anomaly detection scores output by each base model, and use the random forest network to analyze the normalized anomaly detection scores to obtain the corresponding first anomaly detection scores.

[0096] Among them, the training process of the above random forest network includes:

[0097] First, two groups of three anomaly detection values ​​(i.e., feature data) obtained from multiple groups of first-layer base models are input into the improved isolation forest model to establish a binary tree for segmentation until each subspace (isolation tree) contains only one data sample (i.e., only one tree node), and when segmenting the binary tree nodes, the hyperplane corresponding to the selected cutting point adopts Sd gain As the segmentation criterion (i.e., finally selecting multiple feature combinations for segmentation hyperplanes so that the node data space obtains the maximum Sd gain ), and Sd gain The expression is as follows:

[0098]

[0099] Where Y represents the projection of the input data of the current tree node divided by this hyperplane, Y l Indicates the projection of the input data of the current tree node on the left child node after being split according to this hyperplane, Y r It indicates that the input data of the current tree node is projected on the right child node according to this hyperplane. Obviously, Y = Y L ∪Y r , δ represents the standard deviation function, and avg represents the arithmetic mean function. Therefore, compared with the hyperplane of the randomly selected binary tree split point, the embodiment of the present application selects the hyperplane by the standard deviation of the hyperplane of the multiple feature combinations, which can greatly improve the model's ability to accurately identify and detect anomalies.

[0100] Then, for each data sample point x, the following formula can be used to calculate the anomaly score values ​​S (respectively denoted as S1 and S2) of the two sets of first-layer base models:

[0101]

[0102] Where h(x) represents the mean height of the data sample point in each isolated tree (i.e., each base model tree taken by the isolation forest), c(n) represents the average path length of the binary tree constructed by n data in the sample space, and the formula is: c(n) = 2*H(n-1)-2*(n-1) / n, where H(·) represents the harmonic series function.

[0103] In addition, the second-layer base model also includes: a random forest network. Therefore, the second-layer base model can analyze the multiple first anomaly detection scores output by the multiple groups of first-layer base models according to the following steps, including:

[0104] Step 3: Input the first anomaly detection score output by each group of the first-layer base models into the base model based on the support vector machine and the base model based on the local outlier factor algorithm in the second-layer base model, and obtain the anomaly detection scores output by the base model based on the support vector machine and the base model based on the local outlier factor algorithm respectively;

[0105] Step 4: Use the random forest network to analyze the anomaly detection scores output by the base model based on the support vector machine and the base model based on the local outlier factor algorithm to obtain the corresponding second anomaly detection score.

[0106] In this embodiment, the first anomaly detection scores S1 and S2 output by each group of first-layer base models are fused using the support vector machine-based base model and the local outlier factor algorithm-based base model in the second-layer base model to obtain the anomaly detection scores output by the support vector machine-based base model and the local outlier factor algorithm-based base model respectively; then, the random forest network is used again to analyze the anomaly detection scores output by the support vector machine-based base model and the local outlier factor algorithm-based base model respectively to obtain the corresponding second anomaly detection scores.

[0107] It should be noted that the structure of the random forest network in the second-layer base model is the same as that of the random forest network in the first-layer base model. In addition, each base model in the second-layer base model is also obtained by iterative training using five-fold cross-validation. That is to say, the abnormal data features (i.e., anomaly detection scores) output by multiple groups of first-layer base models are divided into five parts according to the index, and sent to the base model based on the support vector machine and the base model based on the local outlier factor algorithm for training and inference five times. Each time, four of the five parts of data are used for training, and the other part is used for inference. In this way, the result of each inference (i.e., the anomaly detection score of the two base models) is inferred from the base model trained by four of the data to the other data. After five times, the two base models can obtain independent inference results of the complete data.

[0108] As an optional implementation, in the technical solution provided in the above step S206, the abnormal structured data in the to-be-detected data set may be determined according to the following method, including:

[0109] Traversing each structured data in the data set to be detected, and determining the relationship between the second anomaly detection score of the structured data and a preset threshold value;

[0110] When the second anomaly detection score of the structured data is not lower than the threshold value, determining that the structured data is abnormal structured data;

[0111] When the second anomaly detection score of the structured data is lower than the threshold value, the structured data is determined to be normal structured data.

[0112] Among them, the above threshold value can be set according to the actual application scenario, and this application does not make any specific restrictions on this.

[0113] As another optional implementation, in the technical solution provided in the above step S206, the abnormal structured data in the to-be-detected data set may be determined according to the following method, including:

[0114] The second anomaly detection score of each structured data in the data set to be detected is sorted in descending order to obtain the corresponding sorting result, and the structured data with a preset number or preset proportion (that is, the ratio of the number of selected abnormal structured data to the total number of all structured data in the data set to be detected) at the head of the sorting result is used as abnormal structured data.

[0115] Therefore, compared with the prior art, the structured data anomaly detection method provided in the embodiment of the present application has the following advantages:

[0116] (1) The collected structured data is normalized using improved Gaussian rank normalization, which fully retains the frequency information and sorting information of the original data while achieving better Gaussian distribution mapping. Compared with the traditional Gaussian rank normalization, the embodiment of the present application makes the following two improvements: First, the nonlinear transformation function of Chebyshev polynomial is used to replace the simple linear transformation function, which can characterize nonlinear changes of various orders, handle different data distributions more flexibly, and better approximate the target distribution, so that the final Gaussian quantile calculation can obtain more accurate results. At the same time, since the statistical frequency of the sorting features used in Gaussian rank normalization is sorted from small to large in order of appearance, the later the ranking, the greater the contribution to the mean. Therefore, while retaining the frequency information and sorting information in the original data sample, the normalization generally makes the mapped data closer to the Gaussian distribution; second, the data that cannot be directly mapped is processed using cubic spline interpolation, wherein the cubic spline interpolation considers the three points closest to the input value to construct a cubic polynomial curve to smoothly interpolate the input value, and obtain a more continuous and natural change result, so this method more accurately fits the curve relationship between the data.

[0117] (2) A multi-level model two-layer fusion architecture is used for ensemble learning to fully improve the ensemble learning ability of the model. Compared with the traditional two-level single-layer fusion ensemble learning architecture, it is improved by randomly combining three types of different models based on distance, density, and neural network to obtain multiple groups of first-layer base models, ensuring that the model can mine data information from the three types of different models of distance, density, and neural network, avoiding the limitations of a single type of single model; then, the output results of each group of first-layer base models are used as the input of two types of different base models based on support vector machines and local outlier factors for ensemble fusion, which is equivalent to stacking the results after stacking fusion. This architecture can fully utilize the advantages of each model, avoid the randomness or limitations of a single fusion, improve the stability and generalization ability of the overall model, and improve the overall ability to identify abnormal samples.

[0118] For example, taking the authoritative public data set thyroid (the thyroid disease detection data set of the Garavan Institute published by the University of California, Irvine) and the personal loan default detection data set (the personal loan default detection data set of the Central Plains Bank published by the China Computer Association) as examples, the base model based on the lonely forest IFOREST algorithm, the base model based on the K-nearest neighbor (KNN) algorithm, the base model based on the angle-based outlier detection ABOD algorithm, the base model based on the local anomaly factor COF algorithm, the base model based on the local interest group factor LOF algorithm, the base model based on the learnable unified neighbor anomaly ranking LANUR algorithm, the bad learning anomaly detection ALAD algorithm and the single fusion model are used to perform anomaly detection on the structured data model, and the results are compared with the above method provided in the embodiments of the present application, and the AUC (Area Under the Curve) value result comparison table shown in Table 1 and the abnormal data accuracy (recall rate) result comparison table shown in Table 2 are obtained:

[0119] Table 1

[0120]

[0121] Table 2

[0122]

[0123] It is not difficult to see from Tables 1 and 2 that, for the above two authoritative public data sets, the AUC value and the abnormal sample accuracy (recall rate) of the method provided by the present application are significantly improved compared with the overall single model methods based on tree models, distance, density, clustering, neural networks, and stacking methods using a single fusion.

[0124] Example 2

[0125] Based on Embodiment 1 of the present application, an embodiment of a structured data anomaly detection device is also provided, and when the device is running, the structured data anomaly detection method of the above embodiment is executed. Figure 3 is a schematic diagram of the structure of an optional structured data anomaly detection device according to an embodiment of the present application, such as Figure 3 As shown, the structured data anomaly detection device at least includes an acquisition module 32, an analysis module 34 and a detection module 36, wherein:

[0126] An acquisition module 32 is used to acquire a plurality of structured data in the data set to be detected;

[0127] The analysis module 34 is used to detect and analyze each structured data using multiple groups of pre-trained first-layer base models to obtain a first anomaly detection score output by each group of first-layer base models, wherein each group of first-layer base models includes a distance-based base model, a density-based base model, and a neural network-based base model; and to analyze and process the first anomaly detection score output by each group of first-layer base models using a pre-trained second-layer base model to obtain a corresponding second anomaly detection score, wherein the second-layer base model includes a support vector machine-based base model and a local outlier factor algorithm-based base model;

[0128] The detection module 36 is used to determine the abnormal structured data in the to-be-detected data set according to the second abnormality detection score of each structured data.

[0129] It should be noted that the various modules in the above-mentioned structured data anomaly detection device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0130] Example 3

[0131] According to an embodiment of the present application, a non-volatile storage medium is further provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the structured data anomaly detection method in Example 1.

[0132] Optionally, the device where the non-volatile storage medium is located implements the following steps by running the program: obtaining multiple structured data in the data set to be detected; for each structured data, using multiple groups of pre-trained first-layer base models to detect and analyze the structured data, and obtaining a first anomaly detection score output by each group of first-layer base models, wherein each group of first-layer base models includes a distance-based base model, a density-based base model, and a neural network-based base model; using a pre-trained second-layer base model to analyze and process the first anomaly detection score output by each group of first-layer base models, and obtain a corresponding second anomaly detection score, wherein the second-layer base model includes a base model based on a support vector machine and a base model based on a local outlier factor algorithm; and determining the abnormal structured data in the data set to be detected based on the second anomaly detection score of each structured data.

[0133] According to an embodiment of the present application, a computer program product is also provided, which includes a stored computer program, wherein when the computer program is executed by a processor, the structured data anomaly detection method in Example 1 is implemented.

[0134] Optionally, the computer program executes the following steps: obtaining multiple structured data in the data set to be detected; for each structured data, using multiple groups of pre-trained first-layer base models to detect and analyze the structured data, and obtaining a first anomaly detection score output by each group of first-layer base models, wherein each group of first-layer base models includes a distance-based base model, a density-based base model, and a neural network-based base model; using a pre-trained second-layer base model to analyze and process the first anomaly detection score output by each group of first-layer base models, and obtain a corresponding second anomaly detection score, wherein the second-layer base model includes a support vector machine-based base model and a local outlier factor algorithm-based base model; determining the abnormal structured data in the data set to be detected based on the second anomaly detection score of each structured data.

[0135] According to an embodiment of the present application, a processor is also provided, which is used to run a program, wherein the structured data anomaly detection method in Example 1 is executed when the program is running.

[0136] Optionally, when the program is running, the following steps are executed: obtaining multiple structured data in the data set to be detected; for each structured data, using multiple groups of pre-trained first-layer base models to detect and analyze the structured data, and obtaining a first anomaly detection score output by each group of first-layer base models, wherein each group of first-layer base models includes a distance-based base model, a density-based base model, and a neural network-based base model; using a pre-trained second-layer base model to analyze and process the first anomaly detection score output by each group of first-layer base models, and obtain a corresponding second anomaly detection score, wherein the second-layer base model includes a support vector machine-based base model and a local outlier factor algorithm-based base model; determining the abnormal structured data in the data set to be detected based on the second anomaly detection score of each structured data.

[0137] According to an embodiment of the present application, an electronic device is also provided, wherein: Figure 4 is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application, such as Figure 4 As shown, the electronic device includes one or more processors; a memory for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are implemented to run the programs, wherein the programs are configured to execute the structured data anomaly detection method in the above-mentioned embodiment 1 when running.

[0138] Optionally, the processor is configured to execute the following steps through a computer program: obtaining multiple structured data in the data set to be detected; for each structured data, using multiple groups of pre-trained first-layer base models to detect and analyze the structured data, and obtain a first anomaly detection score output by each group of first-layer base models, wherein each group of first-layer base models includes a distance-based base model, a density-based base model, and a neural network-based base model; using a pre-trained second-layer base model to analyze and process the first anomaly detection score output by each group of first-layer base models, and obtain a corresponding second anomaly detection score, wherein the second-layer base model includes a base model based on a support vector machine and a base model based on a local outlier factor algorithm; and determining the abnormal structured data in the data set to be detected based on the second anomaly detection score of each structured data.

[0139] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0140] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0142] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0143] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0144] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0145] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A structured data anomaly detection method, characterized in that: include: Obtain multiple structured data in the data set to be detected; For each of the structured data, use a plurality of pre-trained first-layer base models to perform detection and analysis on the structured data to obtain a first anomaly detection score output by each group of the first-layer base models, wherein each group of the first-layer base models includes a distance-based base model, a density-based base model, and a neural network-based base model; use a pre-trained second-layer base model to analyze and process the first anomaly detection score output by each group of the first-layer base models to obtain a corresponding second anomaly detection score, wherein the second-layer base models include a support vector machine-based base model and a local outlier factor algorithm-based base model; The abnormal structured data in the to-be-detected data set is determined according to the second anomaly detection score of each structured data.

2. The method according to claim 1, characterized in that Get multiple structured data in the dataset to be tested, including: Acquire the data set to be detected; Preprocessing the data set to be detected to obtain a plurality of initial structured data, wherein the preprocessing includes at least one of the following: data cleaning and feature encoding; Gaussian rank normalization is performed on the multiple initial structured data to obtain the corresponding multiple structured data.

3. The method according to claim 2, characterized in that Performing Gaussian rank normalization processing on the multiple initial structured data to obtain the corresponding multiple structured data includes: Counting the values ​​of the data in the plurality of initial structured data according to a preset column dimension, and determining the element type under each column dimension; In the case where the element type is a single-value type, all element values ​​under the column dimension are converted into a first fixed value to obtain a normalized element value; In the case where the element type is a binary type, the two types of element values ​​under the column dimension are converted into a second fixed value or a third fixed value respectively to obtain normalized values ​​of the two types of element values; In the case where the element type is a multi-value type, determining the value types of each of the multiple types of element values ​​under the column dimension, and converting each type of the element value according to the value type to obtain the normalized multiple types of element values, wherein the value type includes at least: numerical data, ordered categorical data, non-numerical data, empty data, or unordered categorical data; The plurality of structured data are obtained by converting each data in the plurality of initial structured data according to the element values ​​of each category after processing.

4. The method according to claim 3, characterized in that The element values ​​of each category are converted according to the value type to obtain the standardized element values ​​of the multiple categories, including: For each type of element value under the column dimension, when the value type is the numerical data or the ordered categorical data, the first occurrence number of each type of element value and the second occurrence number of the element value under the corresponding column dimension are counted, and the linear division vector of the element value is determined according to the first occurrence number and the second occurrence number; the linear division vector is transformed according to the Chebyshev polynomial, and in combination with a preset scaling factor, the quantile of the Gaussian distribution of the transformed linear division vector is inversely solved by using the inverse function of the error function; the quotient of the product of the second occurrence number and the quantile and the first occurrence number is used as the weighted mean of the element value, and the quantile minus the weighted mean is used as the normalized element value; When the value type is the non-numeric data or the empty data or the unordered categorical data, the element value is mapped using a cubic spline interpolation algorithm to obtain a normalized element value.

5. The method according to claim 1, characterized in that Each group of the first-layer base models also includes a random forest network, wherein the structured data is detected and analyzed using the pre-trained multiple groups of first-layer base models to obtain the first anomaly detection score output by each group of the first-layer base models, including: For each group of the first-layer base models, each base model in the first-layer base model is used to detect and analyze the structured data to obtain anomaly detection scores output by each base model; the anomaly detection scores output by each base model are normalized, and the normalized anomaly detection scores are analyzed using the random forest network to obtain the corresponding first anomaly detection scores.

6. The method according to claim 5, characterized in that The base model based on the distance algorithm includes at least one of the following: a base model based on the K-nearest neighbor algorithm, a base model based on the angle-based outlier detection algorithm; The base model based on the density algorithm includes at least one of the following: a base model based on a local anomaly factor algorithm, a base model based on a local benefit group factor algorithm; The neural network-based base model includes at least one of the following: a base model based on an anomaly ranking algorithm that can learn unified neighbors, and a base model based on a bad learning anomaly detection algorithm.

7. The method according to claim 1, characterized in that The second-layer base model also includes a random forest network, wherein the first anomaly detection score output by each group of the first-layer base model is analyzed and processed by using the pre-trained second-layer base model to obtain the corresponding second anomaly detection score, including: Inputting the first anomaly detection scores output by the first-layer base models of each group into the base models based on the support vector machine and the base models based on the local outlier factor algorithm in the second-layer base models, and obtaining the anomaly detection scores output by the base models based on the support vector machine and the base models based on the local outlier factor algorithm respectively; The random forest network is used to analyze the anomaly detection scores output by the base model based on the support vector machine and the base model based on the local outlier factor algorithm, respectively, to obtain the corresponding second anomaly detection score.

8. The method according to claim 1, characterized in that Determining abnormal structured data in the to-be-detected data set according to the second abnormality detection score of each structured data includes: Traversing each structured data in the to-be-detected data set, and determining a magnitude relationship between a second anomaly detection score of the structured data and a preset threshold value; When the second anomaly detection score of the structured data is not lower than the threshold value, the structured data is determined to be abnormal structured data.

9. A structured data anomaly detection device, characterized in that: include: An acquisition module, used for acquiring multiple structured data in the data set to be detected; An analysis module is used to detect and analyze each of the structured data using multiple groups of pre-trained first-layer base models to obtain a first anomaly detection score output by each group of the first-layer base models, wherein each group of the first-layer base models includes a distance-based base model, a density-based base model, and a neural network-based base model; and to analyze and process the first anomaly detection score output by each group of the first-layer base models using a pre-trained second-layer base model to obtain a corresponding second anomaly detection score, wherein the second-layer base models include a support vector machine-based base model and a local outlier factor algorithm-based base model; The detection module is used to determine the abnormal structured data in the to-be-detected data set according to the second abnormality detection score of each structured data.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, wherein the device where the non-volatile storage medium is located executes the structured data anomaly detection method according to any one of claims 1 to 8 by running the computer program.

Images