Reducing error rate in abnormal classification

Abstract

The present disclosure describes apparatuses and methods of anomaly classification. In one embodiment, an apparatus performs binary classification of data samples in a dataset to classify the data samples into a normal group or an anomaly group; performs multiclass classification to classify the data samples in the anomaly group into anomaly classes, and identifies a first set of data samples in the anomaly group as false positives resulting from the binary classification when the multiclass classification fails to classify the data samples in the first set into one of the anomaly classes.

Description

Technical Field

[0001] This disclosure relates to the field of data science, and more specifically to training machine learning models for detecting and / or classifying anomalies. Background Technology

[0002] Today, diverse datasets are collected from various sources. For example, business delivery systems providing services (such as mobile communication services), software systems (such as social media platforms, e-commerce websites, and search engines), and cloud systems, and / or other types of systems generate logs or other data describing their operational status (e.g., runtime information). Anomaly detection is used across various domains to detect or label anomalous patterns or events in data. Rapidly detecting and / or classifying anomalies enhances security, confidentiality, and efficiency. Therefore, evaluation metrics such as false positive rate (FRR) and false negative rate (FNR) are crucial for evaluating system performance. A false positive is a result that incorrectly indicates the presence of anomalies or abnormalities in the data. False positives can cause serious problems and operational disruptions in areas such as cybersecurity, autonomous driving, industrial control systems, and public safety. A false negative is a result that incorrectly indicates the absence of anomalies or abnormalities in the data. False negatives lead to missed anomalies, resulting in overlooked threats, system failures, and security vulnerabilities. Therefore, effectively reducing false positives (FP) and false negatives (FN) in anomaly detection / classification remains a challenge. Summary of the Invention

[0003] This paper describes an enhanced data analysis system and related data analysis methods. In summary, the data analysis system described herein uses a multi-layered architecture or method when analyzing anomalies in a dataset. One layer uses binary classification to classify data samples in the dataset into one of two groups or classes: a normal group (or class) for "normal" samples, or an anomalous group (or class) for abnormal or non-normal samples. Another layer uses multi-class classification to classify data samples from the anomalous groups (or classes) output by the binary classification layer into one of multiple (three or more) anomalous categories (or anomalous subclasses). One technical advantage is that the multi-class classification layer can effectively identify and recover false positives identified in the binary classification layer. Therefore, the false positive rate (FRR) of the data analysis system can be reduced.

[0004] The multi-layered architecture of a data analyzer can also include another layer that uses anomaly detection. This layer can perform anomaly detection on data samples from the normal groups (or classes) output by the binary classification layer, and / or on data samples classified as normal or unknown by the multi-class classification layer. One technical advantage is that the anomaly detection layer can effectively identify and recover false negatives identified in the binary and / or multi-class classification layers. Therefore, the false negative rate (FNR) of the data analysis system can be reduced.

[0005] In one embodiment (also referred to as an aspect), an apparatus includes at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least: perform binary classification of data samples in a dataset to classify the data samples into a normal group or an abnormal group; perform multi-class classification to classify data samples in an abnormal group into an abnormal category; and when multi-class classification fails to classify data samples in a first set of data samples in an abnormal group into one of the abnormal categories, identify the first set as a false positive generated by the binary classification.

[0006] In one embodiment, when executed by at least one processor, the instructions cause the apparatus to perform at least: perform anomaly detection on data samples in the normal group; when the anomaly detection detects an anomaly in data samples in a second set of data samples in the normal group, identify the second set as a false negative generated by binary classification; and add data samples from the second set to the anomaly group for multi-class classification.

[0007] In one embodiment, when executed by at least one processor, the instructions cause the apparatus to at least: perform anomaly detection on data samples in a first set identified as false positives; verify one or more data samples in the first set as false negatives generated by binary classification when the anomaly detection does not detect anomalies in one or more data samples in the first set; and / or identify one or more data samples in the first set as false negatives generated by multi-class classification when the anomaly detection detects anomalies in one or more data samples in the first set.

[0008] In one embodiment, a method includes: performing binary classification on data samples in a dataset to classify the data samples into a normal group or an abnormal group; performing multi-class classification to classify data samples in the abnormal group into an abnormal category; and identifying the first set as false positives generated by the binary classification when the multi-class classification fails to classify data samples in the first set of data samples in the abnormal group into one of the abnormal categories.

[0009] In one embodiment, a method includes: performing anomaly detection on data samples in a normal group, identifying the second set as a false negative generated by binary classification when the anomaly detection detects an anomaly in data samples of a second set of data samples in the normal group, and adding data samples of the second set to an anomaly group for multi-class classification.

[0010] In one embodiment, a method includes: performing anomaly detection on data samples in a first set identified as false positives; verifying one or more data samples in the first set as false negatives generated by binary classification when the anomaly detection does not detect anomalies in one or more data samples in the first set; and / or identifying one or more data samples in the first set as false negatives generated by multi-class classification when the anomaly detection detects anomalies in one or more data samples in the first set.

[0011] Other embodiments may include computer-readable media, other systems, or other methods described below.

[0012] The foregoing summary provides a basic understanding of certain aspects of this specification. This summary is not a broad overview of this specification. It is not intended to identify key or essential elements of this specification, nor is it intended to depict any particular embodiment or any scope of the claims. Its sole purpose is to present some concepts of this specification in a simplified form as a prelude to the more detailed description that follows. Attached Figure Description

[0013] Some embodiments of the present invention will now be described by way of example only and with reference to the accompanying drawings. Throughout the drawings, the same reference numerals denote the same elements or elements of the same type.

[0014] Figure 1 This is a block diagram illustrating a data analysis paradigm in an illustrative embodiment.

[0015] Figure 2 This is a block diagram of the data analysis system in an illustrative embodiment.

[0016] Figure 3 This is a flowchart illustrating an anomaly classification method in an illustrative embodiment.

[0017] Figure 4 This is a block diagram illustrating a data analyzer in an illustrative embodiment.

[0018] Figure 5 This is a block diagram illustrating anomaly classification in a data analyzer in an illustrative embodiment.

[0019] Figure 6 This is a flowchart illustrating an anomaly classification method in an illustrative embodiment.

[0020] Figure 7 This is a block diagram illustrating a data analyzer in another illustrative embodiment.

[0021] Figure 8 This is a block diagram illustrating the anomaly classification of a data analyzer in an illustrative embodiment.

[0022] Figure 9 This is a flowchart illustrating an anomaly classification method in an illustrative embodiment.

[0023] Figure 10 This is a block diagram illustrating anomaly classification in a data analyzer in an illustrative embodiment.

[0024] Figure 11 This is a flowchart illustrating an anomaly classification method in an illustrative embodiment.

[0025] Figure 12 This is a block diagram illustrating a data analyzer in an illustrative embodiment.

[0026] Figure 13 This is a schematic diagram illustrating the training and deployment of a machine learning model in an illustrative embodiment.

[0027] Figure 14 This is a schematic diagram illustrating the training and deployment of an autoencoder in an illustrative embodiment.

[0028] Figure 15 This is a schematic diagram illustrating the retraining of a machine learning model in an illustrative embodiment.

[0029] Figure 16 This is a flowchart illustrating a method for training a binary classifier model in an illustrative embodiment.

[0030] Figure 17 This is a flowchart illustrating a method for training a multi-class classifier model in an illustrative embodiment.

[0031] Figure 18 This is a schematic diagram illustrating the retraining of an autoencoder in an illustrative embodiment.

[0032] Figure 19 This is a flowchart illustrating a method for training an autoencoder in an illustrative embodiment. Detailed Implementation

[0033] The accompanying drawings and the following description illustrate specific exemplary embodiments. Therefore, those skilled in the art will understand that, although not explicitly described or shown herein, various schemes embodying the principles of the embodiments and included within the scope of the embodiments can be devised. Furthermore, any examples described herein are intended to aid in understanding the principles of the embodiments and should be construed as not constituting limitation on these specifically enumerated examples and conditions. Therefore, the inventive concepts are not limited to the specific embodiments or examples described below, but are limited by the claims and their equivalents.

[0034] Figure 1This is a block diagram illustrating a data analysis paradigm 100 in an illustrative embodiment. Typically, data analysis paradigm 100 begins with data collection 110 or data ingestion from one or more data sources 102. In data collection 110, one or more datasets 104 are received from data sources(s) 102. The datasets(s) 104 can be structured data, unstructured data, semi-structured data, etc. For example, data sources 102 may include devices, apparatuses, servers, network units, processing units, hardware units, software modules, applications, programs, cloud-based applications, etc., configured to perform actions, operations, activities, services, etc., and generate or output log files, runtime information, data packets (e.g., Internet Protocol (IP)), video, audio, images, Internet of Things (IoT) data, and / or other data.

[0035] Data analytics paradigm 100 may also include data storage and preprocessing 112. The ingested data may be heterogeneous, exhibiting diversity in data type, format, and / or structure. Therefore, the data may be cleaned, transformed, merged, etc., before being loaded into a suitable data repository. Data analytics 114 refers to techniques used to evaluate, process, or otherwise analyze data to extract or derive inferences or insights from it. Reporting 116 refers to communication of any inferences or insights extracted or derived from data, such as physical or digital documents, data visualizations, etc.

[0036] Figure 2 This is a block diagram illustrating a data analysis system 200 in an illustrative embodiment. The data analysis system 200 is a data processing system, apparatus, application, component, etc., configured to perform analysis, reporting, etc., on one or more datasets 104. In one embodiment, the data analysis system 200 may include the following subsystems: a network interface component 202, a data collector 204, a data analyzer 206, and a data storage 208. The network interface component 202 is a hardware component or circuit system that exchanges messages, data packets, data, etc., with other components via a network connection. The network interface component 202 can communicate using various protocols, application programming interfaces (APIs), etc. The data collector 204 includes circuit systems, logic, hardware, components, etc., configured to collect data for analysis. The data analyzer 206 includes circuit systems, logic, hardware, components, etc., configured to analyze, examine, or monitor data. Example operation of the data analyzer 206 will be described in further detail below.

[0037] In one embodiment, data analyzer 206 may implement one or more machine learning (ML) systems 210 for analyzing data. ML system 210 may include circuitry, logic, hardware, software, components, etc., configured to use machine learning techniques to perform the functions described for data analyzer 206. In one embodiment, one or more ML models 216 are trained for ML system 210. Typically, ML model 216 is a program or algorithm that learns from training samples to identify patterns or make decisions. ML system 210 may also include an ML trainer 212 and an ML manager 214. ML trainer 212 may include circuitry, logic, hardware, components, etc., configured to train and / or retrain one or more ML models 216. ML manager 214 may include circuitry, logic, hardware, components, etc., configured to manage one or more trained ML models 216. For example, ML manager 214 may be configured to input data into the trained ML model 216 during testing or after deployment, and receive outputs from the ML model 216, among other functions.

[0038] Data storage 208 includes a repository configured to store data such as ingested dataset(s) 104 collected by data collector 204, training data for ML model 216, and / or other data.

[0039] One or more subsystems of the data analysis system 200 may be implemented on a hardware platform composed of analog and / or digital circuitry. One or more subsystems of the data analysis system 200 may be implemented on a processor 230, which executes instructions 234 stored in memory 232. The processor 230 includes integrated hardware circuitry configured to execute instructions 234 to provide the functionality of the data analysis system 200. Depending on the specific implementation, the processor 230 may include a collection of one or more processors, or may include a multiprocessor core. Memory 232 is a non-transitory computer-readable medium for data, instructions, applications, etc., and is accessible by the processor 230. Memory 232 is a hardware storage device capable of storing information on a temporary and / or permanent basis. Memory 232 may include random access memory, or any other volatile or non-volatile storage device.

[0040] One or more subsystems of the data analytics system 200 may be implemented on a cloud computing platform 240 (e.g., Amazon Web Services (AWS)) or other types of processing platforms. Cloud resources may be provisioned on the cloud computing platform 240, such as processing resources 242 (e.g., physical or hardware processors, servers, virtual servers or virtual machines (VMs), virtual central processing units (vCPUs), etc.), storage resources 244 (e.g., physical or hardware storage, virtual storage, etc.), and / or network resources 246, although other resources are considered herein. The data analytics system 200 may be built on the provisioned resources using instructions, programming, code, etc. For example, a network interface component 202 may be provisioned on network resource 246, a data collector 204 and / or a data analyzer 206 may be provisioned on processing resource 242, and data storage 208 may be provisioned on storage resource 244.

[0041] Data analysis system 200 may include Figure 2 Various other components not specifically shown in the text.

[0042] In one embodiment, the data analysis system 200 is configured to perform anomaly detection / classification on dataset 104 (or more datasets). Figure 3 This is a flowchart illustrating an anomaly classification method 300 in an illustrative embodiment. The steps of method 300 will be described with reference to... Figure 2 The method described in the data analysis system 200 is as follows, but those skilled in the art will understand that method 300 can be performed in other systems or devices. Furthermore, the steps in the flowcharts described herein are not exhaustive and may include other steps not shown, and these steps may be performed in an alternative order.

[0043] Data analysis system 200 (e.g., via network interface 202) receives dataset 104 (step 302). Figure 2 As shown, dataset 104 includes multiple elements referred to as data samples 220. Data samples 220 may include log messages, network traffic, packets, bytes, images, etc. Subsequently, data analysis system 200 (e.g., via data analyzer 206) analyzes dataset 104 to detect and / or classify one or more anomalies (step 304). An anomaly is something that deviates from the normal, standard, or expected. Data analysis system 200 then outputs an indication or classification of any anomalies detected in dataset 104 (step 308). For example, data analysis system 200 may generate a report describing the anomaly, display anomaly information about the user interface, generate warnings or alerts, or tag data containing anomalies or abnormalities, etc.

[0044] By implementing or performing multi-level analysis, the data analysis system 200 is enhanced relative to the previous system in terms of detecting or classifying anomalies (such as in terms of false negative rate (FNR) and / or false positive rate (FRR)) (step 306). Figure 4 This is a block diagram illustrating a data analyzer 206 in an illustrative embodiment. The data analyzer 206 is configured for multi-level analysis via a binary classification layer 410 and a multi-class classification layer 420. The binary classification layer 410 includes one or more binary classifiers 412. Each binary classifier 412 is configured to perform binary classification, the task of which is to classify elements / samples in a set into one of two groups or classes. In one embodiment, the binary classifier 412 is configured to classify elements / samples into either a normal group 414 or an anomaly group 416. The normal group 414 (also referred to as the normal class) is the category of samples classified as normal, standard, or expected based on the binary classifier 412. The anomaly group 416 (also referred to as the anomaly class) is the category of samples classified as deviating from normal, standard, or expected based on the binary classifier 412.

[0045] The multi-classification layer 420 includes one or more multi-class classifiers 422. The multi-class classifiers 422 are configured to perform multi-class classification (or multi-nominal classification), the task of which is to classify elements / samples in the set into three or more categories 428 (also referred to as multi-class categories, predefined categories, subcategories, etc.). In one embodiment, the multi-class classifier 422 may be configured to classify elements / samples into multiple predefined anomalous categories 424 or anomalous subcategories (e.g., anomalous category 1, anomalous category 2, ..., anomalous category n), normal categories 426, etc.

[0046] The data analysis system 200 described in this article provides intelligent and automated solutions to enhance the reliability of data processing and minimize error rates.

[0047] Figure 5 This is a block diagram illustrating anomaly classification in data analyzer 206 in an illustrative embodiment. Figure 6 This is a flowchart illustrating an anomaly classification method 600 in an illustrative embodiment. Figure 5 In this process, data analyzer 206 receives dataset 104 comprising multiple data samples 220. The data samples 220 of dataset 104 are input into binary classifier 412, which performs binary classification on the data samples 220 to classify them into normal group 414 or abnormal group 416 (see [link to data analysis]). Figure 6(Step 602 in the original text). In other words, the binary classifier 412 classifies each data sample 220 into one of the normal group 414 or the abnormal group 416. Therefore, the output of the binary classifier 412 is each data sample 220 that is classified into one of the normal group 414 (or normal category) or the abnormal group 416 (or abnormal category). In one embodiment, the data analyzer 206 may consider or identify the data sample 220 in the normal group 414 as a normal sample 530.

[0048] Data sample 220 from outlier group 416 (i.e., classified by binary classifier 412) is input into multi-class classifier 422, which performs multi-class classification to classify data sample 220 from outlier group 416 into category 428 (see [link to classifier]). Figure 6 (Step 604 in the text). In other words, the multi-class classifier 422 attempts to classify each data sample 220 in the outlier group 416 into one of the outlier categories 424 (or outlier subcategories), normal categories 426, etc. When the multi-class classifier fails to classify the sample data 220 in the first set of one or more data samples 220 in the outlier group 416 into one of the outlier categories 424, the data analyzer 206 may identify that set (referred to as the first set) as a false positive 532 (FP) generated by binary classification (see step 604 in the text). Figure 6 (Step 606 in the original text). For example, the multi-class classifier 422 can classify one or more data samples 220 in the outlier group 416 into the normal category 426. Therefore, the data analyzer 206 identifies the data samples 220 in the normal category 426 (classified by the multi-class classifier 422) as false positives 532 generated by binary classification. In another example, the multi-class classifier 422 may be unable to classify one or more data samples 220 in the outlier group 416 into one of the outlier categories 424 or the normal category 426 when the confidence level or certainty exceeds a predetermined threshold (e.g., 85%, 90%, 95%, etc.). When the multi-class classifier 422 fails or is unable to classify the data samples 220 in the outlier group 416 into one of the outlier categories 424 when the confidence level exceeds the predetermined threshold, the data analyzer 206 identifies these sample data 200 in the outlier group 416 as false positives 532 generated by binary classification. Therefore, the output of the multi-class classifier 422 is a data sample 220 identified as either a false positive 532 or an anomalous category in the anomalous category 424. In one embodiment, the false positive 532 can be considered recovered and can be part of the normal sample 530. A technical advantage is that the data analyzer 206 uses the multi-class classifier 422 to recover the false positive 532 from the binary classifier 412. Therefore, the false positive rate (FRR) of the data analyzer 206 is reduced.

[0049] Figure 7 This is a block diagram illustrating a data analyzer 206 in another illustrative embodiment. In one embodiment, the data analyzer 206 is configured to perform multi-layer analysis through a binary classification layer 410, a multi-class classification layer 420, and an anomaly detection layer 730. The anomaly detection layer 730 includes one or more anomaly detectors 732. The anomaly detectors 732 are configured to detect anomalies or unusual conditions in elements / samples.

[0050] Figure 8 This is a block diagram illustrating anomaly classification in data analyzer 206 in an illustrative embodiment. Figure 9 This is a flowchart illustrating an anomaly classification method 900 in an illustrative embodiment. Figure 8 In this process, data analyzer 206 receives dataset 104 comprising multiple data samples 220, which are input to binary classifier 412. Binary classifier 412 performs binary classification on the data samples 220 to classify the data samples 220 into normal group 414 or abnormal group 416 (see [link to data analysis]). Figure 6 (Step 602 in the original text). (That is, the data sample 220 in the anomaly group 416 classified by the binary classifier 412) is input into the multi-class classifier 422, where the multi-class classifier 422 performs multi-class classification to classify the data sample 220 in the anomaly group 416 into category 428, such as anomaly category 424 (see step 602 in the original text). Figure 6 Step 604 in the process. Data analyzer 206 identifies a set of one or more data samples 220 in outlier group 416 (referred to as the first set) as false positives 532 generated by binary classification (see step 604 in the process). Figure 6 Step 606 in the middle.

[0051] Anomaly detector 732 is configured to validate the classification of binary classifier 412. Therefore, data samples 220 from normal group 414 are input to anomaly detector 732, which performs anomaly detection on the data samples 220 from normal group 414 (see [link to relevant documentation]). Figure 9 (See step 902 in the original text). In other words, the anomaly detector 732 can process each data sample 220 in the normal group 414 to determine whether the data sample 220 contains an anomaly. When the anomaly detector 732 does not detect an anomaly in the data sample 220 in the normal group 414, the data analyzer 206 can consider or identify those data samples 220 as normal samples 530 (see step 902 in the original text). Figure 9(Step 904 in the text). In some cases, the anomaly detector 732 can detect anomalies in one or more data samples 220 in the normal group 414 (also referred to as the second set of data samples 220). When the anomaly detector 732 detects anomalies in the data samples 220 of the second set, the data analyzer 206 identifies the data samples 220 of the second set as false negatives 834 (FN) generated by binary classification (see step 904 in the text). Figure 9 (Step 906 in the previous section). Then, the data analyzer 206 can add the data samples 220 from the second set to the outlier group 416 for multi-class classification (see step 906 in the previous section). Figure 6 Step 908 in the process. The process can be returned to... Figure 6 In step 604, the multi-class classifier 422 performs multi-class classification to classify the false negative 834 as category 428 (see step 604). Figure 6 (Step 604 in the original text). Therefore, the false negatives 834 identified by the anomaly detector 732 from the binary classifier 412 are input into the multi-class classifier 422 to attempt to classify each false negative 834 into one of the anomaly category 424, the normal category 426, etc. A technical advantage is that the data analyzer 206 uses the anomaly detector 732 to recover the false negatives 834 from the binary classifier 412. Therefore, the false negative rate (FNR) of the data analyzer 206 is reduced.

[0052] Figure 10 This is a block diagram illustrating anomaly classification in data analyzer 206 in an illustrative embodiment. Figure 11 This is a flowchart illustrating an anomaly classification method 1100 in an illustrative embodiment. An anomaly detector 732 is configured to verify the classification 422 of a multi-class classifier. Figure 10 As described above, the multi-class classifier outputs a first set of data samples 220 labeled as false positives 532. This first set of data samples 220 is then input to an anomaly detector 732, which performs anomaly detection on the first set of data samples 220 (see reference). Figure 11 (Step 1102 in the previous section). In other words, the anomaly detection 732 can process each data sample 220 identified as a false positive 532 to determine whether the data sample 220 contains an anomaly. When the anomaly detector 732 does not detect an anomaly in the data samples 220 in the first set, the data analyzer 206 verifies that these data samples in the first set are false positives 532 generated by binary classification (see step 1102 in the previous section). Figure 11(Step 1104 in the first set). In some cases, the anomaly detector 732 can detect anomalies in one or more data samples in the first set of data samples 220 (also referred to as the third set of data samples 220). When the anomaly detector 732 detects anomalies in the data samples 220 of the first set, the data analyzer 206 identifies these data samples 220 as false negatives 834 generated by the multi-class classifier (see step 1104 in the first set). Figure 11 (Step 1106 in the original text). The anomaly detector 732 can identify a third set of data samples 220 (which is identified as false negatives 834) as data samples 220 that belong to an unknown anomaly category 824 in the multi-class classification or from the perspective of the multi-class classifier 422 (step 1108). One technical advantage is that the data analyzer 206 uses the anomaly detector 732 to recover the false negatives 834 from the multi-class classifier 422. Therefore, the false negative rate (FNR) of the data analyzer 206 is reduced.

[0053] Figure 12 This is a block diagram illustrating a data analyzer 206 in an illustrative embodiment. In one embodiment, a binary classifier 412 may include an ML binary classifier model 1212. The ML binary classifier model 1212 is trained using supervised learning (e.g., a supervised learning algorithm) to classify a set of elements / samples into one of two groups or classes (e.g., normal group 414 or abnormal group 416). Therefore, a binary classifier based on the ML binary classifier model 1212 can be referred to as a supervised binary classifier. Similarly, a multi-class classifier 422 includes an ML multi-class classifier model 1222. The ML multi-class classifier 1222 is trained using supervised learning to classify elements / samples into one of a plurality of (e.g., three or more) categories 428 (e.g., an abnormal category, either normal category 426 or abnormal category 424). Therefore, a multi-class classifier based on the ML multi-class classifier model 1222 can be referred to as a supervised multi-class classifier.

[0054] Typically, supervised classification techniques using machine learning employ training data with a “complete” set of labels that are “normal” and “abnormal”. Figure 13 This is a schematic diagram illustrating the training and deployment of a machine learning model in an illustrative embodiment. The data analysis system 200 can run in a training phase 1302 and a testing or deployment phase 1304. In the training phase 1302, for example, an ML trainer 212 runs to train an ML binary classifier model 1212 and an ML multi-class classifier model 1222, such as... Figure 2An example of ML model 216 is shown. ML trainer 212 performs training 1324 of ML binary classifier model 1212 using labeled training samples 1312 of training dataset 1310, and performs training 1324 of ML multi-class classifier model 1222 using labeled training samples 1313 of training dataset 1311. Note that although two training datasets 1310 to 1311 are shown, there may still be overlap between training datasets 1310 to 1311. During training 1324, ML trainer 212 can train ML binary classifier model 1212 and ML multi-class classifier model 1222 in multiple iteration cycles 1320, where each iteration cycle 1320 is a single iteration training across the entire training dataset. During an iteration cycle 1320, the ML binary classifier model 1212 sequentially processes labeled training samples 1312 from the training dataset 1310, calculates the loss or otherwise quantizes the predicted output, and updates the model parameters 1318 (e.g., weights) accordingly. Similarly, the ML multi-class classifier model 1222 sequentially processes labeled training samples 1313 from the training dataset 1311, calculates the loss or otherwise quantizes the predicted output, and updates the model parameters 1318 accordingly. The number of iteration cycles 1320 determines how many times the ML binary classifier model 1212 and the ML multi-class classifier model 1222 iterate through the entire training dataset 1310 to 1311, allowing the model to learn and improve the model parameters 1318 through multiple traversals. The ML binary classifier model 1212 and the ML multi-class classifier model 1222 are trained in epochs 1322 from labeled training samples 1312 to 1313 from the training datasets 1310 to 1311 within one iteration cycle. Round 1322 is the number of training samples to go through before updating the model parameters 1318.

[0055] In the testing / deployment phase 1304, for example, the ML manager 214 can use a trained binary classifier model 1212 and a trained ML multi-class classifier model 1222 to classify data samples 220 of the dataset 104. For example, the ML manager 214 can feed the dataset 104 to the ML binary classifier model 1212 (as trained), and the ML binary classifier model 1212 will output a binary classification 1330 for the data sample 220 as belonging to the normal group 414 or the abnormal group 416, and can output an associated confidence score 1332 for the binary classification 1330. Then, the ML manager 214 can feed the data sample 220 of the outlier group 416 to the ML multi-class classifier model 1222 (as trained), and the ML multi-class classifier model 1222 can output the multi-class classification 1340 for the data sample 220 as belonging to an outlier category 424, or belonging to a normal category 426, etc., and can output the associated confidence score 1342 for the multi-class classification 1340.

[0056] exist Figure 12 In this paper, the anomaly detector 732 may include an autoencoder 1232. The autoencoder 1232 is trained using unsupervised learning (i.e., an unsupervised learning algorithm) to perform anomaly detection (also known as outlier detection). Therefore, the anomaly detection described herein may be referred to as unsupervised anomaly detection. Typically, unsupervised learning techniques use unlabeled training data. Unsupervised anomaly detection may be based on some basic assumptions: anomalies in the data occur very rarely, and the features of anomalies are significantly different from those of normal instances. Typically, the autoencoder 1232 is an artificial neural network including an encoder 1234 and a decoder 1236. The encoder 1234 is configured to transform input data (e.g., data sample 220) into an encoded representation in the latent space, and the decoder 1236 is configured to reconstruct the input data from the encoded representation. The autoencoder 1232 outputs a reconstruction loss 1238, which is a measure of how close the output of the decoder 1236 is to the input to the encoder 1234. Therefore, the reconstruction loss 1238 indicates whether the data sample 220 input to the autoencoder 1232 is anomalous. For example, when the reconstruction loss of data sample 220 exceeds a threshold (e.g., the average reconstruction loss of normal unlabeled samples used to train autoencoder 1232), autoencoder 1232 identifies anomalies in data sample 220.

[0057] Figure 14 This is a schematic diagram illustrating the training and deployment of the autoencoder 1232 in an illustrative embodiment. The data analysis system 200 can run in a training phase 1402 and a testing or deployment phase 1404. In the training phase 1402, for example, an ML trainer 212 runs to train the autoencoder 1232, which is as follows: Figure 2An example of ML model 216 is shown. ML trainer 212 performs training 1424 of autoencoder 1232 using unlabeled training samples 1212 from training dataset 1410. Note that although one training dataset 1410 is shown, multiple training datasets 1410 can be used. Training 1424 can be similar to that described above.

[0058] In the testing / deployment phase 1404, for example, the ML manager 214 can use a trained autoencoder 1232 to detect anomalies in the data sample 220. For instance, the ML manager 214 can feed data sample 220 from the normal group 414, as well as data sample 220 identified as a false positive 532, to the autoencoder 1232 (as trained), and the autoencoder 1232 outputs a reconstruction loss 1238 for the data sample 220. The reconstruction loss 1238 can be compared to a threshold to detect anomalies.

[0059] In some instances, the ML binary classifier model 1212 and the ML multi-class classifier model 1222 can be retrained based on the results of anomaly classification. Figure 15 This is a schematic diagram illustrating the retraining of a machine learning model in an illustrative embodiment. The data analysis system 200 can run in a retraining phase 1506 to retrain one or both of the ML binary classifier 1212 and the ML multi-class classifier 1222. In the retraining phase 1506, one or more labeled training samples 1312 can be added to or modified in the training dataset 1310 to generate a modified training dataset 1510. The ML trainer 212 can use the modified training dataset 1510 to perform retraining of the ML binary classifier model 1212. Similarly, one or more labeled training samples 1313 can be added to or modified in the training dataset 1311 to generate a modified training dataset 1511. The ML trainer 212 can use the modified training dataset 1511 to perform retraining of the ML multi-class classifier model 1222. During retraining 1524, the ML trainer 212 may retrain the ML binary classifier model 1212 and / or the ML multi-class classifier model 1222 in multiple iterations 1320, in rounds 1322 of labeled training samples 1312 to 1313 from the modified training datasets 1510 to 1511, as described above for training 1324.

[0060] Figure 16This is a flowchart illustrating a method 1600 for training an ML binary classifier model 1212 in an illustrative embodiment. The ML trainer 212 performs training 1324 (step 1602) on the ML binary classifier model 1212 using labeled training samples 1312 in the training dataset 1310. After training, the ML manager 214 can use the trained ML binary classifier model 1212 to classify data samples 220 as described above. At some point, the ML trainer 212 may modify the training dataset 1310 (step 1604) and perform retraining 1524 (step 1610) on the ML binary classifier model 1212 using the modified training dataset 1310. For example, the ML trainer 212 may modify the training dataset 1310 based on one or more data samples 220 identified as false positives 532 in anomaly classification, along with correct labels (optional step 1606). In another example, the ML trainer 212 can modify the training dataset 1310 (optional step 1608) based on one or more data samples 220 that are identified as false negatives 834 in anomaly classification and the correct labels. One technical advantage is that the accuracy of the ML binary classifier model 1212 can be improved.

[0061] Figure 17 This is a flowchart illustrating a method 1700 for training an ML multi-class classifier model 1222 in an illustrative embodiment. The ML trainer 212 performs training 1324 (step 1702) on the ML multi-class classifier model 1222 using labeled training samples 1313 in the training dataset 1311. After training, the ML manager 214 can use the trained ML multi-class classifier model 1222 to classify data samples 220 as described above. At some point, the ML trainer 212 can modify the training dataset 1311 (step 1704) and perform retraining 1524 (step 1708) on the ML multi-class classifier model 1222 using the modified training dataset 1511. For example, the ML trainer 212 can modify the training dataset 1311 based on one or more data samples 220 identified as false negatives 834 in anomaly classification, along with correct labels (optional step 1706). When the unknown anomaly classification 824 (see...) Figure 10 When identified by the ML multi-class classifier model 1222, one or more data samples 220 in the unknown anomaly category 824 can be provided to the scorer, etc., to assign labels to the data samples 220, which creates training samples 1313 with new labels that can be added to the training dataset 1311.

[0062] In some instances, the autoencoder 1232 can be retrained based on the results of anomaly classification. Figure 18This is a schematic diagram illustrating the retraining of the autoencoder 1232 in an illustrative embodiment. The data analysis system 200 can run in a retraining phase 1806 to retrain the autoencoder 1232. In the retraining phase 1806, one or more unlabeled training samples 1412 can be added to or modified in the training dataset 1410 to generate a modified training dataset 1810. The ML trainer 212 can use the modified training dataset 1810 to perform retraining 1824 on the autoencoder 1232.

[0063] Figure 19 This is a flowchart illustrating a method 1900 for training an autoencoder 1232 in an illustrative embodiment. The ML trainer 212 performs training 1324 (step 1902) on the autoencoder 1232 using unlabeled training samples 1412 in the training dataset 1410. After training, the ML manager 214 can use the trained autoencoder 1232 to detect anomalies in the data samples 220 as described above. At some point, the ML trainer 212 may modify the training dataset 1410 (step 1904) and perform retraining 1824 (step 1908) on the autoencoder 1232 using the modified training dataset 1810. For example, the ML trainer 212 may modify the training dataset 1410 based on one or more data samples 220 identified as unknown anomaly categories 824 in multi-class classification or from the perspective of the multi-class classifier 422 (optional step 1906). Data sample 220 in the unknown anomaly category 824 is significantly different from normal sample 530, even though the multi-class classifier fails to identify data sample 220 as a specific anomaly category 424. Therefore, data sample 220 in the unknown anomaly category 824 can be used as unlabeled training sample 1412. Data sample 220 in the unknown anomaly category 824 can be clustered using a clustering algorithm. To provide more discriminative features and to perform clustering as efficiently as possible with minimal overlap, the ML trainer 212 can concatenate the confidence score of each anomaly category 424 of the multi-class classifier 422 with the current feature vector of the unknown data sample. This represents the probability of similarity between each anomaly category 424 and the unknown data sample. The probability of similarity between each anomaly category 424 and the unknown data sample will help the clustering algorithm to effectively compare data sample 220 in terms of distance-based similarity metrics and will lead to better clustering results for the unknown data sample. One technical advantage is that the accuracy of the autoencoder 1232 can be improved.

[0064] Any element or module shown in the figures or described herein may be implemented as hardware, software, firmware, or a combination thereof. For example, an element may be implemented as dedicated hardware. A dedicated hardware element may be referred to as a “processor,” a “controller,” or similar terms. When provided by a processor, functionality may be provided by a single dedicated processor, by a single shared processor, or by multiple independent processors, some of which may be shared. Furthermore, the explicit use of the terms “processor” or “controller” should not be construed as referring only to hardware capable of executing software, and may implicitly include, but is not limited to: digital signal processor (DSP) hardware, network processors, application-specific integrated circuits (ASICs) or other circuit systems, field-programmable gate arrays (FPGAs), read-only memory (ROM) for storing software, random access memory (RAM), non-volatile memory, logic devices, or some other physical hardware component or module.

[0065] Furthermore, a component can also be implemented as instructions executable by a processor or computer to perform the component's function. Some examples of instructions are software, program code, and firmware. Instructions are operable when executed by a processor to instruct the processor to perform the component's function. Instructions can be stored on a processor-readable storage device. Some examples of storage devices are digital or solid-state memory, magnetic storage media (such as disks and tapes), hard disk drives, or optically readable digital data storage media.

[0066] As used in this application, the term "circuit system" may refer to one or more or all of the following: (a) Hardware circuit implementation only (such as implementation only in analog and / or digital circuit systems); (b) A combination of hardware circuitry and software, such as (if applicable): (i) A combination of (multiple) analog and / or digital hardware circuits having software / firmware, and (ii) Any part of a hardware processor(s) having software (including (multiple) digital signal processors, software, and (multiple) memories, which work together to enable a device (such as a mobile phone or server) to perform various functions); and (c) (Multiple) hardware circuits and / or (multiple) processors, such as (multiple) microprocessors or a portion thereof, which require software (e.g., firmware) to operate, but may be absent when operation is not required.

[0067] This definition of "circuit system" applies to all uses of the term in this application (including in any claim). As another example, as used herein, the term "circuit system" also covers only hardware circuitry or a processor (or multiple processors) or portions of hardware circuitry or a processor and its accompanying software and / or firmware implementation. For example, and if applicable to a particular claim element, the term "circuit system" also covers baseband integrated circuits or processor integrated circuits for mobile devices, or similar integrated circuits in servers, cellular network devices, or other computing or network devices.

[0068] Although specific embodiments have been described herein, the scope of this disclosure is not limited to these specific embodiments. The scope of this disclosure is defined by the following claims and any equivalents thereof.

Claims

1. A device for communication, comprising: At least one processor; as well as At least one memory storing instructions that, when executed by the at least one processor, cause the device to perform at least the following: Perform binary classification on data samples in the dataset to classify the data samples into a normal group or an abnormal group; Perform multi-class classification to categorize the data samples in the anomaly group into anomaly categories; as well as When the multi-class classification fails to classify the data samples in the first set of the data samples in the anomaly group into one of the anomaly categories, the first set is identified as a false positive generated by the binary classification.

2. The apparatus of claim 1, wherein the instructions, when executed by the at least one processor, cause the apparatus to perform at least: Anomaly detection is performed on the data samples in the normal group; When the anomaly detection detects an anomaly in the data samples of the second set of data samples in the normal group, the second set is identified as a false negative generated by the binary classification; The data samples in the second set are added to the anomaly group for use in the multi-class classification. as well as Perform anomaly detection on the data samples in the first set that are identified as false positives; as well as When the anomaly detection fails to detect an anomaly in one or more data samples in the first set, the one or more data samples in the first set are verified as false positives generated by the binary classification.

3. The apparatus according to any one of claims 1 to 2, wherein the instructions, when executed by the at least one processor, cause the apparatus to perform at least: Perform anomaly detection on the data samples in the first set that are identified as false positives; and When the anomaly detection detects an anomaly in one or more data samples in the first set, the one or more data samples in the first set are identified as false negatives generated by the multi-class classification.

4. The apparatus according to claim 3, wherein: The data samples in the first set that are identified as false negatives include: the data samples of unknown anomaly categories in the multi-class classification.

5. The apparatus according to claim 3, wherein: The binary classification is performed using a machine learning binary classifier model trained with supervised learning; and When the instructions are executed by the at least one processor, the device causes the following to be performed: The machine learning binary classifier model is retrained using one or more data samples from the first set that are identified as false positives, along with the correct labels. as well as The machine learning binary classifier model is retrained using one or more data samples from the second set of data samples that are identified as false negatives, along with the correct labels.

6. The apparatus according to claim 3, wherein: The multi-class classification is performed using a machine learning multi-class classifier model trained with supervised learning; and When the instructions are executed by the at least one processor, the device causes the following to be performed: The machine learning multi-class classifier model is retrained using one or more data samples from the first set that are identified as false negatives, along with the correct labels.

7. The apparatus according to claim 3, wherein: The anomaly detection is performed using an autoencoder; and When the instructions are executed by the at least one processor, the device causes the following to be performed: The autoencoder is retrained using one or more data samples from the first set of data samples that are identified as false negatives.

8. A method for communication, comprising: Perform binary classification on data samples in the dataset to classify the data samples into a normal group or an abnormal group; Perform multi-class classification to categorize the data samples in the anomaly group into anomaly categories; When the multi-class classification fails to classify the data samples in the first set of the data samples in the anomaly group into one of the anomaly categories, the first set is identified as a false positive generated by the binary classification; Anomaly detection is performed on the data samples in the normal group; When the anomaly detection detects an anomaly in the data samples of the second set of data samples in the normal group, the second set is identified as a false negative generated by the binary classification; as well as The data samples in the second set are added to the anomaly group for use in the multi-class classification.

9. The method according to claim 8, further comprising: Perform anomaly detection on the data samples in the first set that are identified as false positives; When the anomaly detection fails to detect an anomaly in one or more data samples in the first set, the one or more data samples in the first set are verified as false positives generated by the binary classification. as well as Perform anomaly detection on the data samples in the first set that are identified as false positives; as well as When the anomaly detection detects an anomaly in one or more data samples in the data samples of the first set, the one or more data samples in the data samples of the first set are identified as false negatives generated by the multi-class classification; The anomaly detection is performed using an autoencoder; and The method further includes retraining the autoencoder using one or more data samples from the data samples in the first set that are identified as false negatives.

10. A computer-readable medium comprising program instructions that, when executed by a device, cause the device to perform at least the following: Perform binary classification on data samples in the dataset to classify the data samples into a normal group or an abnormal group; Perform multi-class classification to categorize the data samples in the anomaly group into anomaly categories; and When the multi-class classification fails to classify the data samples in the first set of the data samples in the anomaly group into one of the anomaly categories, the first set is identified as a false positive generated by the binary classification.

Images