Abnormal data detection method and device, storage medium and electronic equipment

Abstract

The application discloses an abnormal data detection method and device, a storage medium and an electronic device. The method comprises the following steps: obtaining a structured data set to be detected, wherein the structured data set comprises a plurality of feature fields and a plurality of first structured data; determining tree features of the first structured data by using a gradient boosting decision tree, and fusing the tree features with the first structured data to obtain a first fusion data set; constructing a relationship network directed graph based on target feature fields with an association relationship in the first fusion data set, fusing graph features of the relationship network directed graph with the first fusion data set to obtain a second fusion data set; performing abnormal detection on each first fusion data in the second fusion data set by using a target isolation forest model, determining an outlier score of the first fusion data, and regarding the first structured data with an outlier score greater than a preset threshold as abnormal data. The application solves the technical problem that related abnormal data detection methods are difficult to guarantee the accuracy of detection results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data detection, in particular to an abnormal data detection method and device, a storage medium and an electronic device. BACKGROUND

[0002] Abnormal detection refers to finding data problems that do not conform to expected behavior patterns in data. As an important research content in the field of artificial intelligence, it involves machine learning, data mining and other technologies of intelligent data. Therefore, abnormal detection has been widely used in various aspects of production and life, such as risk detection of telecommunications industry, anti-fraud and loan default detection of financial industry, disease detection of medical industry, and network security intrusion detection. The data of telecommunications business information, the data of financial fraud user financial behavior, the data of disease detection user disease characteristics, and the data of network security detection network user traffic and behavior are all structured data in nature (i.e. regular data that can be represented by relational data or logically expressed by two-dimensional table). The advantages and disadvantages of the abnormal detection method for structured data will seriously affect the healthy development of related business.

[0003] At present, the abnormal detection of structured data mostly adopts artificial intelligence machine learning methods based on statistics or based on models. However, the method based on statistics, such as 3σ method and Z-score method, lacks effective learning of data, has insufficient accuracy, and is difficult to meet most production requirements. The model-based method, such as the isolated forest algorithm based on tree, lacks enhancement processing of original feature data, is more dependent on the extraction of data by the model itself, often ignores the information of tree features and relationship network data contained in the data itself, and even if the data normalization method is used, the traditional normalization and standardization method is adopted, which leads to insufficient utilization of the sorting information of the original data, and the detection accuracy of structured data is difficult to guarantee.

[0004] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0005] The embodiments of the present application provide an abnormal data detection method, device, storage medium and electronic device, which at least solve the technical problem that related abnormal data detection methods are difficult to guarantee the accuracy of detection results.

[0006] According to an aspect of the embodiments of the present application, a method for detecting abnormal data is provided, comprising: obtaining a structured data set to be detected, the structured data set comprising: a plurality of feature fields and a plurality of first structured data corresponding to each feature field; determining tree features of each first structured data by a gradient boosting decision tree, and fusing the tree features of each first structured data with the first structured data to obtain a first fused data set; constructing a relational network directed graph based on target feature fields having a correlation relationship in the first fused data set, determining graph features of the relational network directed graph, and fusing the graph features with the first fused data set to obtain a second fused data set; performing abnormal detection on each first fused data in the second fused data set by a target isolation forest model, determining an outlier score of each first fused data, and determining that the first structured data corresponding to the first fused data is abnormal data when the outlier score is greater than a preset threshold, wherein a target hyperplane for dividing a data space of the target feature field in the target isolation forest model is determined by a standard deviation of a random hyperplane created by a plurality of randomly selected feature field combinations.

[0007] Optionally, obtaining the structured data set to be detected comprises: determining a plurality of feature fields, wherein the feature fields are numerical fields; obtaining a plurality of second structured data corresponding to each feature field, and performing a preprocessing operation on the plurality of second structured data to obtain processed first structured data, wherein the preprocessing operation comprises at least one of the following: cleaning, filling, and feature encoding.

[0008] Optionally, determining the tree features of each first structured data by the gradient boosting decision tree comprises: constructing a gradient boosting decision tree, wherein the gradient boosting decision tree comprises: at least one base classifier and at least one leaf node on each base classifier; for each first structured data, determining a target leaf node index of the first structured data on each base classifier, and obtaining the tree features of the first structured data based on the target leaf node index.

[0009] Optionally, constructing the relational network directed graph based on the target feature fields having the correlation relationship in the first fused data set, and determining the graph features of the relational network directed graph comprises: determining a plurality of feature fields having the correlation relationship in the first fused data set; constructing the relational network directed graph based on the plurality of feature fields having the correlation relationship, and analyzing the relational network directed graph by a graph structure algorithm to obtain the graph features of the relational network directed graph, wherein the graph features at least comprise one of the following: a degree of a node, a closeness centrality of a node, an edge betweenness, a multiplicity, a cyclic edge attribute, a pair edge attribute, an information graph member number, and an information graph modularization score.

[0010] Optionally, before inputting the second fused dataset into the target isolated forest model, the method further includes: performing numerical feature normalization on the second fused dataset, wherein the numerical feature normalization includes: data sorting, frequency statistics, and numerical mapping.

[0011] Optionally, the second fused sample dataset is subjected to numerical feature normalization processing, including: sorting each second fused data in the second fused dataset to obtain the sorting features of each first fused sample data; calculating the frequency length of each second fused data in the second fused dataset; and determining the feature value of the second fused data according to the frequency length of each second fused data.

[0012] Optionally, the feature value of the second fused data is determined according to the frequency length of each second fused data, including: when the frequency length is single-valued data, the feature value of the second fused data is determined to be 0; when the frequency length is binary data, the feature value of the second fused data is determined to be 0 or 1 based on the sorting feature; when the frequency length is multi-valued data, the quantile of the first fused data is determined based on the linear equipartition vector of each second fused data and a preset scaling factor, the weighted average of the first fused dataset is calculated based on the quantile of each second fused data, the frequency length, and the total frequency length of all second fused data, and the feature value of the second fused data is obtained by subtracting the weighted average from the quantile of each second fused data, wherein the linear equipartition vector of each second fused data is used to characterize the ratio of the frequency length of the second fused data to the total frequency length of all second fused data in the second fused dataset.

[0013] Optionally, an outlier detection is performed on each of the first fused data points within the second fused dataset using a target isolated forest model to determine the outlier score of each first fused data point. This includes: randomly combining the feature fields of the processed second fused dataset to obtain multiple sets of feature field combinations, and constructing a random hyperplane based on each set of feature field combinations; for each first fused data point in the processed second fused dataset, calculating the standard deviation of the first fused data point on each random hyperplane, and using the random hyperplane with the largest standard deviation as the target hyperplane to partition the data space where the first fused data point is located to obtain an isolated tree; traversing all the isolated trees corresponding to each first fused data point, and calculating the outlier score of each first fused data point.

[0014] According to another aspect of the embodiments of this application, an abnormal data detection device is also provided, comprising: an acquisition module, configured to acquire a structured dataset to be detected, the structured dataset including: multiple feature fields and multiple first structured data corresponding to each feature field; a first feature extraction module, configured to determine the tree features of each first structured data through a gradient boosting decision tree, and fuse the tree features of each first structured data with the first structured data to obtain a first fused dataset; a second feature extraction module, configured to construct a directed graph of a relational network based on the second structured data corresponding to the target feature fields with which there are correlations in the first fused dataset, determine the graph features of the directed graph of the relational network, and fuse the graph features with the first fused dataset to obtain a second fused dataset; and an anomaly detection module, configured to perform anomaly detection on each first fused data in the second fused dataset using a target isolated forest model, determine the outlier score of each first fused data, and determine the first structured data corresponding to the first fused data as abnormal data when the outlier score is greater than a preset threshold, wherein the target hyperplane used to divide the data space of the target feature fields in the target isolated forest model is determined by the standard deviation of a random hyperplane created by a combination of multiple randomly selected feature fields.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described abnormal data detection method through the computer program.

[0016] In this embodiment, a structured dataset to be detected is obtained, which includes multiple feature fields and multiple first structured data corresponding to each feature field. A gradient boosting decision tree is used to determine the tree features of each first structured data, and the tree features of each first structured data are fused with the first structured data to obtain a first fused dataset. A directed graph of a relational network is constructed based on the target feature fields with relationships in the first fused dataset. The graph features of the directed graph of the relational network are determined, and the graph features are fused with the first fused dataset to obtain a second fused dataset. An isolated forest model is used to detect anomalies in each first fused data in the second fused dataset, and an anomaly score is determined for each first fused data. When the anomaly score is greater than a preset threshold, the first structured data corresponding to the first fused data is determined to be an anomaly. In the isolated forest model, the target hyperplane used to partition the data space of the target feature fields is determined by the standard deviation of a random hyperplane created by combining multiple randomly selected feature fields.

[0017] In the above technical solution, structured data is enhanced by gradient boosting trees, which increases the data selection information retained by tree features; at the same time, graph structure features are also used to enhance structured data that fuses tree features, increasing the relational information of graph features; in addition, the improved isolated forest model used in this application calculates outlier scores, wherein the hyperplane selection for binary tree split points is based on the standard deviation of random hyperplanes of multiple feature combinations, which improves the accurate identification and detection capability of anomalies in the isolated forest model, thereby solving the technical problem that related anomaly detection methods are difficult to guarantee the accuracy of detection results. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a hardware structure block diagram of an optional computer terminal for implementing an abnormal data detection method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating an optional abnormal data detection method according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of an optional abnormal data detection device according to an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Furthermore, all information and data (including but not limited to user device information, user personal information, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties. For example, this system has an interface with the relevant user or organization. Before obtaining relevant information, it needs to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving consent from the aforementioned user or organization.

[0025] Example 1

[0026] With the widespread availability of the internet, the operational data generated by frontline sales staff in telecommunications company branches is enormous. Due to the complexity of telecommunications services, sales staff often handle multiple business scenarios, including topping up funds, processing sales, issuing refunds, and issuing SIM cards. The behavioral information data generated by sales staff during these transactions may contain anomalies. Using traditional manual annotation methods to detect anomalies would be too costly in terms of manpower, and the accuracy of the results is difficult to guarantee consistently, thus hindering timely mitigation of losses and the standardization of business operations. Therefore, the quality of anomaly detection results for structured data significantly impacts the healthy development of related businesses.

[0027] Currently, the relevant patent CN202210718289, "Data Detection Method, Apparatus, Computer Equipment, and Storage Medium," proposes a model-based anomaly detection method. This method employs the classic tree-based isolated forest approach. The isolated forest method uses a random hyperplane to divide the data space, randomly selecting features to perform binary tree partitioning of nodes, and then iterating continuously until each subspace (isolated tree) contains only one data point. Finally, it identifies the anomaly sample closest to the root node on the decision tree. However, because this type of method does not enhance the original feature data, it relies heavily on the model's own data extraction, often neglecting the mining and information support of tree features and graph features inherent in the data itself. Furthermore, it uses traditional normalization methods (MinMax, converting data into decimals between 0 and 1) and standardization (converting data into a standard normal distribution with a mean of 0 and a variance of 1), resulting in insufficient utilization of the frequency and ranking information of the original data, making it difficult to guarantee the accuracy of structured data detection.

[0028] The relevant patent CN202210632838, "Method for Detecting Data Anomalies," proposes an anomaly detection method based on the confidence level of a statistical probability distribution function. It uses the statistical probability distribution function to calculate the confidence level to obtain abnormal samples. However, based on the relevant principles of statistics, this type of method considers data that does not meet certain statistical rules as abnormal data. Therefore, it lacks effective learning of the data and is often not accurate enough to meet most production requirements. In addition, this type of method also does not enhance the original feature data, ignores the mining and information support of tree features and graph features of relationship network diagrams in the original data, and does not adopt a certain standardization method for the data.

[0029] In addition, statistical methods, such as the 3sigma method based on normal distribution, the Z-score method for calculating standard scores, the boxplot method based on interquartile range, and the Grubbs method based on hypothesis testing, as well as model-based methods, such as the One-Class SVM method based on classification, the DBSCAN method based on clustering, the KNN method based on distance, and the autoencoder method based on dimensionality reduction, also have the same technical problems as the two related patents mentioned above.

[0030] To address the aforementioned issues, this application provides an anomaly detection method. This method aims to obtain enhanced structured data by fusing tree and graph features of the original structured data, thereby improving the model's ability to identify anomalies. Simultaneously, the enhanced structured data is input into an improved isolated forest model to achieve anomaly detection. In the improved isolated forest model, the hyperplane that divides the data space is determined using the standard deviation of multiple feature combinations.

[0031] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0032] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing an anomaly data detection method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0033] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0034] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the abnormal data detection method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the abnormal data detection method of the application described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0035] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0036] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0037] Figure 2 This is a flowchart illustrating an optional abnormal data detection method according to an embodiment of this application, as shown below. Figure 2 As shown, the method includes at least steps S202-S208, wherein:

[0038] Step S202: Obtain the structured dataset to be detected. The structured dataset includes: multiple feature fields and multiple first structured data corresponding to each feature field.

[0039] In the technical solution provided in step S202, structured data refers to data stored and processed according to a certain format and organization. It typically exists in the form of tables, databases, XML, JSON, etc., and has clearly defined relationships between fields and values. Therefore, it can also be understood as: regularly structured data that can be represented using relational data structures or logically expressed in the form of two-dimensional tables. Generally, structured data includes, but is not limited to: telecommunications business behavior information data, user financial behavior data related to financial fraud, user disease characteristic data for disease detection, and network user traffic for network security detection.

[0040] As an optional implementation, in the technical solution provided in step S202 above, the method may include steps S2021-S2022, wherein:

[0041] Step S2021: Determine multiple feature fields, where the feature fields are numeric fields;

[0042] Step S2022: Obtain multiple second structured data corresponding to each feature field, and perform preprocessing operations on the multiple second structured data to obtain the processed first structured data. The preprocessing operations include at least one of the following: cleaning, padding, and feature encoding.

[0043] Specifically, taking a monthly salesperson behavior dataset obtained from a telecommunications big data platform as the structured dataset to be detected as an example, 32 feature fields are first obtained, such as employee ID, BSS channel ID, agent channel ID, agent name, number of low-price broadband transfer work orders, number of card issuances at night, rebate amount, number of card issuances at abnormal ages, and number of small-value sales items ordered. These feature fields are all numerical fields. Next, 25,841 second structured data entries corresponding to these 32 feature fields are determined. To ensure the accuracy of subsequent abnormal data detection, the 25,841 second structured data entries need to be processed by removing abnormal characters, filling missing values, and feature encoding. Feature encoding can be one-hot encoding to obtain the processed first structured data. The processed first structured data and the corresponding feature fields form the structured dataset.

[0044] Step S204: Determine the tree features of each first structured data by using a gradient boosting decision tree, and fuse the tree features of each first structured data with the first structured data to obtain the first fused dataset.

[0045] In the technical solution provided in step S204, the tree features of each first structured data are determined by a gradient boosting decision tree. The tree features are then concatenated with the first structured data to obtain first structured data with fused tree features. A first fused dataset is then composed of multiple first structured data with fused tree features. In this embodiment, data selection information for retaining tree features can be added to the first structured data. Specifically, the gradient boosting tree in this embodiment can be LightGBM. LightGBM (Gradient Boosting Descision Tree, GBDT) is a machine learning algorithm based on Gradient Boosting Descision Tree (GBDT), which can efficiently process large-scale datasets and provide accurate prediction results.

[0046] As an optional implementation, in the technical solution provided in step S204 above, the method may include steps S2041-S2042, wherein:

[0047] Step S2041: Construct a gradient boosting decision tree, wherein the gradient boosting decision tree includes: at least one base classifier and at least one leaf node on each base classifier.

[0048] Step S2042: For each first structured data, determine the target leaf node index of the first structured data on each base classifier, and obtain the tree features of the first structured data based on the target leaf node index.

[0049] In this embodiment, the number of base classifiers (i.e., decision trees) of LightGBM and the number of leaf nodes on each base classifier are first set. Then, for each first structured data, the optimal leaf node index (i.e., target leaf node index) of the first structured data on each base classifier can be determined. The optimal leaf node index is the optimal selected feature of the first structured data on the base classifier of LightGBM. The feature value of the optimal selected feature determined by the optimal leaf node index for each first structured data is 1 or 0. This feature value can be used as the tree feature of the first structured data.

[0050] For example, if the number of base classifiers is set to 3 and the maximum number of leaf nodes is 4, then 12 tree features can be obtained and incorporated into the original first structured data.

[0051] Step S206: Construct a directed graph of the relational network based on the target feature fields with relationships in the first fused dataset, determine the graph features of the directed graph of the relational network, and fuse the graph features with the first fused dataset to obtain the second fused dataset.

[0052] In the technical solution provided in step S206, a directed graph of a relational network is constructed from the target feature fields with correlations within the first fused dataset composed of the first structured data of the fused tree features. A graph algorithm is then used to mine the directed graph of the relational network, calculating and obtaining potentially meaningful graph features. The graph features are then concatenated with the first structured data of the fused tree features corresponding to the target feature fields to obtain the first structured data of the fused tree features and graph features. Subsequently, a second fused dataset is composed of multiple first structured data of the fused tree features and graph features. This embodiment of the application can utilize graph structural features to add relational information to the first structured data.

[0053] As an optional implementation, in the technical solution provided in step S206 above, the method may include steps S2061-S2062, wherein:

[0054] Step S2061: Determine the target feature fields in the first fused dataset that have a correlation relationship;

[0055] Step S2062: Construct a directed graph of the relational network based on the target feature fields, and use a graph structure algorithm to analyze the directed graph of the relational network to obtain the graph features of the directed graph of the relational network. The graph features include at least one of the following: degree of nodes, proximity centrality of nodes, edge betweenness, multiplicity, cyclic edge attribute, opposite edge attribute, number of information graph members, and information graph modularity score.

[0056] In this embodiment, the target association features of the first fusion dataset are first determined, and the second structured data of the target association features are extracted to form the node numbers of the directed graph of the relationship network. The node codes are connected to the first structured data, and then the directed graph of the relationship network is built using igraph. Edges, nodes, etc. are added to the directed graph of the relationship network to obtain the directed graph of the relationship network.

[0057] Graph structure algorithms are used to mine the graph structure of directed graphs in relational networks and calculate graph features. These features include, but are not limited to, node, edge, and graph clustering attribute characteristics.

[0058] (1) The degree of a node is used to describe the number of edges connecting nodes in a directed graph of a relational network (i.e., the various association features that establish the directed graph mentioned above).

[0059] (2) Closeness, a measure of the propagation efficiency of a node in a directed graph of a relational network to other nodes within its connected components. For each node, in addition to calculating the shortest path between all its node pairs, the sum of its distances to all other nodes is calculated. The reciprocal of this sum is then taken to determine the node's closeness score. The formula for calculating closeness is:

[0060]

[0061] Where u represents the node whose proximity to centrality is to be calculated, n represents the total number of nodes in the graph, and d(u,v) represents the shortest distance between node v and node u;

[0062] (3) The edge betweenness is used to describe the proportion of the number of paths that pass through this edge in the directed graph of the relational network. It can reflect the role and influence of this edge on the critical path (shortest path) in the entire directed graph of the relational network and is an important global geometric quantity.

[0063] (4) The multiple of an edge describes the number of times the edge is associated with the same node in the directed graph network. It represents the correlation of the edge with the entire relational network graph.

[0064] (5) The attribute of loop edge is_loop and the attribute of opposite edge is_mutual are used to describe whether the edge can form a loop edge or an opposite edge in the established directed graph network of relational network. They reflect the closure and connectivity of the edge in the entire relational network.

[0065] (6) Graph clustering attributes: used to build community information graphs for the entire relational network directed graph using the high-quality community graph clustering algorithm Infomap, calculate and extract attributes that characterize the degree of network clustering, such as the number of members in the graph cluster and the modularity score. Among them, the graph clustering attributes include, but are not limited to: the number of information graph members (communities_infomap) and the information graph modularity score (communities_modularity).

[0066] Specifically, taking a structured dataset composed of salesperson behavior data as an example, the associated features in the first fused dataset with fused tree features are determined to be BSS channel ID and agent channel ID. The relevant second structured data of BSS channel ID and agent channel ID are extracted, formatted as node numbers, and the node codes are connected to the first structured data. A directed graph of the relationship network is built using igraph, and edges and nodes are added. The edge names are named employee ID, and the node names are named BSS channel ID. The attributes of the edges are obtained: betweenness, multiple, loop (is_loop), and mutual (is_mutual). The attributes of the nodes are: degree, closeness, information map (communities_infomap), and modularity (communities_modularity). Finally, the edge names and node names are concatenated with the first structured data to fuse the above eight graph features into the salesperson behavior data, resulting in salesperson behavior data with fused tree features and graph features.

[0067] Step S208: Use the target isolated forest model to perform anomaly detection on each of the first fused data in the second fused dataset, determine the outlier score of each of the first fused data, and determine the first structured data corresponding to the first fused data as outlier data when the outlier score is greater than a preset threshold.

[0068] In the technical solution provided in step S208, the traditional isolated forest model iForest uses a random hyperplane to divide the data space. Each division generates two subspaces (isolated trees), and then a random hyperplane is used to divide each subspace, repeating this process until each subspace contains only one data point. Therefore, iForest is only suitable for scenarios with few outliers and global outlier detection. In contrast, this embodiment uses an improved isolated forest model, SciForest, where the target hyperplane used to divide the data space of the target feature fields is determined by the standard deviation of a random hyperplane created by combining multiple sets of randomly selected feature fields.

[0069] As an optional implementation, prior to the technical solution provided in step S208 above, the method may include: performing numerical feature normalization processing on the second fused dataset, wherein the numerical feature normalization processing includes: data sorting, frequency statistics, and numerical mapping.

[0070] Specifically, in this embodiment, Gaussian rank is used to normalize the second fused dataset. Compared with traditional normalization or standardization methods, the Gaussian rank normalization used in this embodiment can ensure that the frequency and sorting information of the first structure data are fully preserved in the third fused dataset.

[0071] Optionally, the second fused dataset can be subjected to numerical feature normalization processing according to the following steps S11-S13, wherein:

[0072] Step S11: Sort each second fusion data in the second fusion dataset to obtain the sorting features of each first fusion sample data;

[0073] Step S12: Calculate the frequency length of each second fused data point within the second fused dataset;

[0074] Step S13: Determine the feature value of the second fused data according to the frequency length of each second fused data.

[0075] Furthermore, the feature values ​​of the second fused data can be determined according to the following rules:

[0076] When the frequency length is single-valued data, the feature value of the second fused data is determined to be 0; when the frequency length is binary data, the feature value of the second fused data is determined to be 0 or 1 based on the sorting feature; when the frequency length is multi-valued data, the quantile of the first fused data is determined based on the linear equipartition vector of each second fused data and the preset scaling factor. The weighted average of the first fused dataset is calculated based on the quantile of each second fused data, the frequency length, and the total frequency length of all second fused data. The feature value of the second fused data is obtained by subtracting the weighted average from the quantile of each second fused data. The linear equipartition vector of each second fused data is used to represent the ratio of the frequency length of the second fused data to the total frequency length of all second fused data in the second fused dataset.

[0077] The frequency length mentioned above can be understood as the specific numerical value of the number of times or the interval between occurrences of the second fused data within the second fused dataset. Therefore, the frequency length of the second fused data can be determined according to the number of times it appears within the second fused dataset. For example, if the second fused data appears 0 times in the second fused dataset, the frequency length is 0 (completely empty data); if the second fused data appears once in the second fused dataset, the frequency length is 1 (single-valued data); if the second fused data appears twice in the second fused dataset, the frequency length is 2 (binary data); if the second fused data appears five times in the second fused dataset, the frequency length is 5 (multi-valued data).

[0078] Specifically, the feature values ​​of the second fused data can be determined based on the feature length of the second fused data and according to the following rules: data with a frequency length of 0 (completely empty data) can be skipped and ignored; data with a frequency length of 1 (single-valued data) can be mapped to 0; data with a frequency length of 2 (i.e., binary data) can be mapped to 0 and 1 respectively according to the sorting; and data with a frequency length greater than 2 (i.e., multi-valued data) can be determined using the following steps:

[0079] First, calculate the proportion of the cumulative occurrences of the second fused data within the second fused dataset to the total cumulative occurrences of all second fused data, and use this proportion as a linearly divided vector rankV with values ​​between 0 and 1.

[0080] Next, the linearly divided vector rankV is linearly transformed according to the following formula, so that the linearly divided vector rankV is limited to the range [0.001, 0.999], avoiding extreme values:

[0081] rankV = rankV * 0.998 + 10 -3

[0082] Then, setting the scaling factor (scale_factor), the Gaussian quantile α is calculated by inversely calculating the Gaussian distribution of the linearly divided vector rankV using the inverse function ErfInv. The formula for calculating α is:

[0083] α = scale factor *ErfInv(rankV)

[0084] The quantile α of each second fusion data is calculated as the weight of the frequency length of the second fusion data. That is, the quantile is multiplied by the frequency length of the second fusion data, and then divided by the total number of occurrences of all second fusion data to obtain the weighted average mean of each second fusion data in the second fusion dataset.

[0085] Finally, the quantile α of each second fused data point is subtracted from the weighted average mean to obtain the feature value of the final mapping of the second fused data.

[0086] As an optional implementation, in the technical solution provided in step S208 above, the method may include steps S2081-S2083, wherein:

[0087] Step S2081: Randomly combine the feature fields of the processed second fused dataset to obtain multiple sets of feature field combinations, and construct a random hyperplane based on each set of feature field combinations;

[0088] Step S2082: For each first fused data in the processed second fused dataset, calculate the standard deviation of the first fused data in each random hyperplane, and take the random hyperplane with the largest standard deviation as the target hyperplane to divide the data space where the first fused data is located to obtain an isolated tree;

[0089] Step S2083: Traverse all isolated trees corresponding to each first fused data point and calculate the outlier score for each first fused data point.

[0090] Specifically, the feature fields in the processed second fused dataset are arbitrarily combined to obtain multiple sets of feature field combinations. A random hyperplane is created based on the feature field combinations, and the standard deviation gain of the random hyperplane constructed by each set of feature field combinations of the first fused data is calculated according to the following formula:

[0091]

[0092] Where Y represents the projection of the input data (second fused data) of the current tree node according to this random hyperplane. l This represents the projection of the input data (second fused data) of the current tree node onto its left child node after being segmented according to this random hyperplane; Y r This represents the projection of the input data (second fused data) of the current tree node onto its right child node after being segmented according to this random hyperplane. Clearly, Y = Y0. l ∪Y r δ() represents the standard deviation function, and avg() represents the arithmetic mean function.

[0093] The standard deviation gain of the first fused data in each random hyperplane can be calculated using the above method. To minimize the confounding between different categories in the second fused data and thus better divide the data space into different categories, in this embodiment, the random hyperplane with the largest standard deviation gain can be used as the target hyperplane. The target hyperplane is then used to determine the data space containing the first fused data (i.e., the data space composed of other fused data with the same feature fields as the first fused data), resulting in an isolation tree. All isolation trees corresponding to each second fused data are traversed, and the outlier score for each second fused data is calculated using the following formula:

[0094]

[0095] Where h(x) represents the average height of the second fused data in each isolated tree (i.e., each base model tree in the isolated forest model), and c(n) represents the average path length of the binary trees constructed from each second fused dataset within the processed second fused dataset. Its expression can be written as: c(n) = 2H(n-1) - 2(n-1) / n, where n is a positive integer greater than 2. In addition, H() in the above expression represents the harmonic series function.

[0096] For example, taking authoritative public datasets such as thyroid (thyroid disease detection dataset from the Garavan Institute, University of California, Irvine), kdd_cup14 (commercial highlight project detection dataset from the Data Mining and Knowledge Discovery Professional Committee of the Association for Computing Machinery), and personal loan default detection dataset (personal loan default detection dataset from Zhongyuan Bank, published by the China Computer Federation) as examples, the improved isolated forest model SciForest and the original isolated forest algorithm IForest used in this application embodiment are compared. The results are shown in Table 1 below: AUC (Area Under the Curve) value comparison table, and Table 2 below: Abnormal data precision (recall) result comparison table.

[0097] Table 1

[0098]

[0099] Table 2

[0100]

[0101] As can be seen from Tables 1 and 2 above, for the three authoritative public datasets, the improved isolated forest model SciForest using the embodiments of this application has significantly improved AUC and outlier precision (recall) compared to the original isolated forest algorithm IForest, showing good results.

[0102] Then, after determining the outlier score S of each piece of first fused data within the second fused dataset, it is determined whether the outlier score S is greater than a preset threshold. If the outlier score is greater than the preset threshold, the first structured data corresponding to the first fused data is determined to be outlier data. Alternatively, a certain number of first structured data with outlier scores greater than the preset threshold (or a proportion of the second fused dataset) are taken as outlier data to obtain the final anomaly detection result.

[0103] Based on the scheme defined in steps S202 to S208 above, it can be understood that in the embodiment, a structured dataset to be detected is obtained, which includes multiple feature fields and multiple first structured data corresponding to each feature field; the tree features of each first structured data are determined by gradient boosting decision tree, and the tree features of each first structured data are fused with the first structured data to obtain a first fused dataset; a directed graph of a relational network is constructed based on the second structured data corresponding to the target feature fields with correlation in the first fused dataset, the graph features of the directed graph of the relational network are determined, and the graph features are fused with the first fused dataset to obtain a second fused dataset; anomaly detection is performed on the second fused dataset using a target isolated forest model, anomaly scores of each first fused data are obtained, and when the anomaly score is greater than a preset threshold, the first structured data corresponding to the first fused data is determined to be anomalous data.

[0104] Therefore, this application's solution enhances structured data using the LightGBM gradient boosting tree, increasing the data selection information retained by the LightGBM tree features. Simultaneously, it enhances structured data with fused tree features using graph structure features, increasing the relational information of the graph features. Furthermore, this application's embodiment uses Gaussian rank to normalize the data, fully preserving the frequency and ranking information of the original structured data. The improved SciForest isolated forest model used in this application's embodiment calculates anomaly scores. Specifically, the hyperplane selection for binary tree split points utilizes the standard deviation of random hyperplanes formed by multiple feature combinations, improving the accurate anomaly detection capability of the isolated forest model and thus solving the technical problem that related anomaly detection methods struggle to guarantee the accuracy of detection results.

[0105] Example 2

[0106] According to an embodiment of this application, an abnormal data detection device for implementing the above-described abnormal data detection method is also provided. Figure 3 This is a schematic diagram of an optional abnormal data detection device according to an embodiment of this application, such as... Figure 3 As shown, the abnormal data detection device includes at least an acquisition module 31, a first feature extraction module 32, a second feature extraction module 33, and an anomaly detection module 34, wherein:

[0107] The acquisition module 31 is used to acquire the structured dataset to be detected. The structured dataset includes: multiple feature fields and multiple first structured data corresponding to each feature field.

[0108] The first feature extraction module 32 is used to determine the tree features of each first structured data through gradient boosting decision tree, and to fuse the tree features of each first structured data with the first structured data to obtain the first fused dataset;

[0109] The second feature extraction module 33 is used to construct a directed graph of the relational network based on the second structured data corresponding to the target feature fields that have a relationship in the first fusion dataset, determine the graph features of the directed graph of the relational network, and fuse the graph features with the first fusion dataset to obtain the second fusion dataset;

[0110] Anomaly detection module 34 is used to perform anomaly detection on each of the first fused data in the second fused dataset using the target isolated forest model, determine the outlier score of each of the first fused data, and determine the first structured data corresponding to the first fused data as outlier data when the outlier score is greater than a preset threshold. In the target isolated forest model, the target hyperplane used to divide the data space of the target feature fields is determined by the standard deviation of the random hyperplane created by combining multiple sets of randomly selected feature fields.

[0111] It should be noted that each module in the abnormal data detection device in this application embodiment corresponds one-to-one with each implementation step of the abnormal data detection method in embodiment 1. Since embodiment 1 has been described in detail, some details not shown in this embodiment can be referred to embodiment 1, and will not be elaborated further here.

[0112] Example 3

[0113] According to an embodiment of this application, a non-volatile storage medium is also provided, which includes a stored program, wherein the device where the non-volatile storage medium is located executes the abnormal data detection method in Embodiment 1 by running the program.

[0114] Specifically, the device containing the non-volatile storage medium executes the following steps by running this program:

[0115] Step S202: Obtain the structured dataset to be detected. The structured dataset includes: multiple feature fields and multiple first structured data corresponding to each feature field.

[0116] Step S204: Determine the tree features of each first structured data by using a gradient boosting decision tree, and fuse the tree features of each first structured data with the first structured data to obtain the first fused dataset;

[0117] Step S206: Construct a directed graph of the relational network based on the target feature fields with relationships in the first fused dataset, determine the graph features of the directed graph of the relational network, and fuse the graph features with the first fused dataset to obtain the second fused dataset;

[0118] Step S208: Anomaly detection is performed on each of the first fused data in the second fused dataset using the target isolated forest model. The outlier score of each first fused data is determined. When the outlier score is greater than a preset threshold, the first structured data corresponding to the first fused data is determined to be outlier data. In the target isolated forest model, the target hyperplane used to divide the data space of the target feature fields is determined by the standard deviation of a random hyperplane created by combining multiple sets of randomly selected feature fields.

[0119] According to an embodiment of this application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the abnormal data detection method of embodiment 1 through the computer program.

[0120] Specifically, the processor is configured to execute the following steps via a computer program:

[0121] Step S202: Obtain the structured dataset to be detected. The structured dataset includes: multiple feature fields and multiple first structured data corresponding to each feature field.

[0122] Step S204: Determine the tree features of each first structured data by using a gradient boosting decision tree, and fuse the tree features of each first structured data with the first structured data to obtain the first fused dataset;

[0123] Step S206: Construct a directed graph of the relational network based on the target feature fields with relationships in the first fused dataset, determine the graph features of the directed graph of the relational network, and fuse the graph features with the first fused dataset to obtain the second fused dataset;

[0124] Step S208: Anomaly detection is performed on each of the first fused data in the second fused dataset using the target isolated forest model. The outlier score of each first fused data is determined. When the outlier score is greater than a preset threshold, the first structured data corresponding to the first fused data is determined to be outlier data. In the target isolated forest model, the target hyperplane used to divide the data space of the target feature fields is determined by the standard deviation of a random hyperplane created by combining multiple sets of randomly selected feature fields.

[0125] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0126] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0128] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0130] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0131] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An abnormal data detection method characterized by comprising: The method comprises the following steps: obtaining a structured data set to be detected, the structured data set comprising a plurality of feature fields and a plurality of first structured data corresponding to each feature field; determining tree features of each first structured data by using a gradient boosting decision tree, and fusing the tree features of each first structured data with the first structured data to obtain a first fused data set; constructing a relational network directed graph based on target feature fields having a correlation relationship in the first fused data set, determining graph features of the relational network directed graph, and fusing the graph features with the first fused data set to obtain a second fused data set; performing anomaly detection on each first fused data in the second fused data set by using a target isolation forest model, determining an outlier score of each first fused data, and determining that the first structured data corresponding to the first fused data is abnormal data when the outlier score is greater than a preset threshold, wherein a target hyperplane in the target isolation forest model for dividing a data space of a target feature field is determined by a standard deviation of a random hyperplane created by a plurality of randomly selected feature field combinations.

2. The method of claim 1, wherein, obtaining a structured data set to be detected, comprising: determining a plurality of feature fields, wherein the feature fields are numerical fields; obtaining a plurality of second structured data corresponding to each feature field, and performing a preprocessing operation on the plurality of second structured data to obtain processed first structured data, wherein the preprocessing operation comprises at least one of the following: cleaning, filling, and feature encoding.

3. The method of claim 1, wherein, determining tree features of each first structured data by using a gradient boosting decision tree, comprising: constructing the gradient boosting decision tree, wherein the gradient boosting decision tree comprises at least one base classifier and at least one leaf node on each base classifier; for each first structured data, determining a target leaf node index of the first structured data on each base classifier, and obtaining the tree features of the first structured data based on the target leaf node index.

4. The method of claim 1, wherein, constructing a relational network directed graph based on target feature fields having a correlation relationship in the first fused data set, determining graph features of the relational network directed graph, comprising: determining a plurality of feature fields having a correlation relationship in the first fused data set; constructing the relational network directed graph based on the plurality of feature fields having a correlation relationship, and analyzing the relational network directed graph by using a graph structure algorithm to obtain the graph features of the relational network directed graph, wherein the graph features at least comprise one of the following: degree of a node, closeness centrality of a node, edge betweenness, multiplicity, cyclic edge attribute, pair edge attribute, information graph member number, and information graph modularization score.

5. The method of claim 1, wherein, Before inputting the second fused data set into the target isolation forest model, the method further comprises: performing numerical feature normalization processing on the second fused data set, wherein the numerical feature normalization processing comprises data sorting, frequency statistics, and numerical mapping.

6. The method of claim 5, wherein, The numerical feature normalization processing is performed on the second fusion sample data set, including: sorting each second fusion data in the second fusion data set to obtain an ordered feature of each first fusion sample data; counting a frequency length of each second fusion data in the second fusion data set; determining a feature value of the second fusion data according to the frequency length of each second fusion data.

7. The method of claim 6, wherein, The feature value of the second fusion data according to the frequency length of each second fusion data includes: when the frequency length is single-value data, determining the feature value of the second fusion data as 0; when the frequency length is binary data, determining the feature value of the second fusion data as 0 or 1 according to the ordered feature; when the frequency length is multi-value data, determining a quantile of the first fusion data based on a linear equal division vector of each second fusion data and a preset scaling factor, calculating a weighted average value of the first fusion data set based on the quantile of each second fusion data, the frequency length and a total frequency length of all second fusion data in the second fusion data set, and obtaining the feature value of the second fusion data by subtracting the weighted average value from the quantile of each second fusion data, wherein the linear equal division vector of each second fusion data is used to represent a ratio of the frequency length of the second fusion data to the total frequency length of all second fusion data in the second fusion data set.

8. The method of claim 6, wherein, The target isolation forest model is used to detect the anomaly of each first fusion data in the second fusion data set to determine an anomaly value score of each first fusion data, including: randomly combining the feature fields of the processed second fusion data set to obtain multiple groups of feature field combinations, and constructing a random hyperplane based on each group of feature field combinations; for each first fusion data of the processed second fusion data set, calculating the standard deviation of the first fusion data in each random hyperplane, and taking the random hyperplane with the maximum standard deviation as the target hyperplane to divide the data space where the first fusion data is located to obtain an isolated tree; traversing all isolated trees corresponding to each first fusion data to calculate the anomaly value score of each first fusion data.

9. An abnormal data detection device characterized by comprising: including: an acquisition module configured to acquire a structured data set to be detected, the structured data set including a plurality of feature fields and a plurality of first structured data corresponding to each feature field; a first feature extraction module configured to determine a tree feature of each first structured data by a gradient boosting decision tree, and fuse the tree features of each first structured data with the first structured data to obtain a first fusion data set; a second feature extraction module configured to construct a relationship network directed graph based on second structured data corresponding to target feature fields having a relationship in the first fusion data set, determine a graph feature of the relationship network directed graph, and fuse the graph feature with the first fusion data set to obtain a second fusion data set; Anomaly detection module, for adopting a target isolation forest model to perform anomaly detection on each first fusion data in the second fusion data set, determine the anomaly score of each first fusion data, and when the anomaly score is greater than a preset threshold, determine that the first structured data corresponding to the first fusion data is abnormal data, wherein the target hyperplane in the target isolation forest model for dividing the data space of the target feature field is determined by the standard deviation of the random hyperplane created by the random selection of multiple groups of feature field combinations.

10. An electronic device, comprising: Comprise: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the anomaly data detection method of any one of claims 1 to 8 through the computer program.

Images