A network anomaly detection method and device

Abstract

The embodiment of the application provides a network anomaly detection method, relates to the field of Internet, and can improve the accuracy of network anomaly detection. The method comprises the following steps: firstly, at least one labeled sample set is acquired, the labeled sample set is a sample set obtained by labeling a test sample set based on labeling feedback information of the test sample set; the labeling feedback information is used for indicating that a plurality of network devices corresponding to the test sample set are in a normal state or an abnormal state, and / or indicating that KPI data of the plurality of network devices in the test sample set is in a normal state or an abnormal state; then, an incremental sample set is determined according to the at least one labeled sample set; finally, a network anomaly detection model is updated based on the incremental sample set; wherein the network anomaly detection model is used for detecting KPI data of the plurality of network devices in the test sample set, so as to determine the state of the KPI data of the plurality of network devices.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for detecting network anomalies. Background Technology

[0002] In communication networks, by performing anomaly detection on network devices (such as routers and switches), silent faults in the network can be discovered and dealt with in a timely manner, thereby reducing the impact of the faults.

[0003] Currently, a common method for network anomaly detection is as follows: A classification model M1 is trained on a training dataset D1 using a supervised learning classification algorithm. Then, this model M1 is used to detect anomalies in network traffic data (i.e., the test dataset), resulting in a labeled test dataset D2. Further, a semi-supervised learning method, such as the improved K-Nearest Neighbors (KNN) algorithm, is used to relabel the labeled test dataset D2, resulting in a new labeled test dataset D3. The unlabeled labeled test dataset D2 and the labeled test dataset D3 are then compared, and inconsistent labels are corrected. The corrected labeled test dataset is added as a new sample to the training dataset D1. Finally, the updated training dataset is trained to obtain a classification model M2. The classification accuracy of classification model M1 and classification model M2 is then compared. If the classification accuracy of classification model M2 is higher than that of classification model M1, then classification model M2 is used instead of classification model M1.

[0004] In the aforementioned network anomaly detection methods, the KNN algorithm based on semi-supervised learning relies on manually labeled samples in the training dataset, resulting in low efficiency for network anomaly detection. Furthermore, the method involves adding the corrected labeled test dataset to the training dataset to continue training the classification model. This process repeats itself, and as time increases, the sample size in the training dataset grows larger. If only a portion of the training dataset is selected for training the classification model, the lack of objectivity in some samples prevents the trained classification model from reaching its optimal state, thus reducing the accuracy of anomaly detection. Summary of the Invention

[0005] This application provides a network anomaly detection method and apparatus, which can improve the accuracy of network anomaly detection.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, embodiments of this application provide a network anomaly detection method, the method comprising: acquiring at least one labeled sample set, the labeled sample set being a sample set after labeling the test sample set based on the labeling feedback information of the test sample set; the labeled feedback information being used to indicate whether multiple network devices corresponding to the test sample set are in a normal or abnormal state, and / or to indicate whether the KPI data of multiple network devices in the test sample set are in a normal or abnormal state; determining an incremental sample set based on the at least one labeled sample set; and updating a network anomaly detection model based on the incremental sample set; wherein, the network anomaly detection model is used to detect the KPI data of multiple network devices in the test sample set to determine the state of the KPI data of the multiple network devices.

[0008] In this embodiment, an incremental sample set can be determined based on the acquired multiple labeled sample sets, and then the network anomaly detection model can be updated based on the incremental sample set. Since the network anomaly detection model in this embodiment is updated based on the incremental sample set, the updated network anomaly detection model can continuously adapt to changes in the KPI data of network devices. Therefore, the updated network anomaly detection model has higher applicability and higher accuracy in detecting network anomalies.

[0009] In one possible implementation, the annotation feedback information of the aforementioned test sample set includes coarse-grained annotation feedback information and / or fine-grained annotation feedback information; wherein, the coarse-grained annotation feedback information is used to indicate whether the multiple network devices corresponding to the test sample set are in a normal or abnormal state, and the coarse-grained annotation feedback information is obtained by annotating the network devices based on at least one of the network device's latency, packet loss rate, or effective time; the aforementioned fine-grained annotation feedback information is used to indicate whether the KPI data of the multiple network devices in the aforementioned test sample set are in a normal or abnormal state, and the fine-grained feedback information is obtained by annotating the network device's KPI data based on manual and / or network anomaly detection models.

[0010] In one possible implementation, the annotation feedback information of the aforementioned test sample set includes N types of annotation feedback information, which include X types of coarse-grained annotation feedback information and Y types of fine-grained annotation feedback information; wherein each type of annotation feedback information corresponds to a labeled sample set, N is a positive integer greater than or equal to 2, X + Y = N, X is a positive integer greater than or equal to 0, and Y is an integer greater than or equal to 1; based on the at least one labeled sample set, determining the incremental sample set includes: determining a first sample set, which is the intersection of the N labeled sample sets; determining Y second sample sets, wherein the i-th second sample set among the Y second sample sets is determined according to the sampling frequency wi The i-th third sample set is obtained by sampling; wherein the i-th third sample set is the difference between the i-th labeled sample set and the first sample set among the Y labeled sample sets corresponding to the above Y fine-grained labeled feedback information; the union of the above first sample set and the above Y second sample sets is determined as the incremental sample set.

[0011] It is understandable that coarse-grained annotation feedback information cannot annotate specific KPI data, but the annotation results corresponding to coarse-grained annotation feedback information have higher accuracy and can reflect the apparent network state. Fine-grained annotation feedback information can annotate specific KPI data, but cannot reflect the apparent network state and may contain a certain degree of mislabeling. In the embodiments of this application, coarse-grained annotation feedback information and fine-grained annotation feedback information can be fused (or combined) to obtain an incremental sample set with higher annotation accuracy. Thus, training on this incremental sample set can yield a network anomaly detection model with higher accuracy (i.e., a better model or a high-quality model).

[0012] In one possible implementation, updating the network anomaly detection model based on the aforementioned incremental sample set includes: extracting features from the incremental sample set to obtain a feature set of the incremental sample set, wherein the feature set of the incremental sample set includes at least one of the following features: mean, variance, extreme values, entropy, difference values, and change ratios of KPI data; and training the network anomaly detection model using an incremental training algorithm based on the aforementioned feature set of the incremental sample set to obtain an updated network anomaly detection model; wherein the incremental training algorithm includes any one of the following: incremental tree algorithm, support vector machine algorithm, and convolutional neural network algorithm.

[0013] In one possible implementation, before updating the network anomaly detection model based on the incremental sample set, the method further includes: performing sample balancing processing on the incremental sample set to make the ratio of positive samples to negative samples in the incremental sample set reach a predetermined ratio; negative samples are samples whose KPI data is in an abnormal state, and positive samples are samples whose KPI data is in a normal state.

[0014] In this embodiment, since network devices typically fail infrequently, the number of negative samples is small. Consequently, the ratio of positive to negative samples in the incremental sample set differs significantly, resulting in a low-quality network anomaly detection model trained with such an incremental sample set, exhibiting a high false detection rate. In this situation, a sample balancing method can be used to increase the proportion of negative samples in the incremental sample set.

[0015] In one possible implementation, the method further includes adjusting the sampling frequency of the third sample set based on user evaluation information of the updated network anomaly detection model.

[0016] In one scenario, if the third sample set corresponds to a single-source fine-grained labeled sample set, and this single-source fine-grained labeled sample set is a manually labeled sample set (the labeled sample set obtained through manual labeling is a manually labeled sample set), then if the score of the network anomaly detection model is lower than a predetermined score, the sampling frequency of the third sample set will be increased.

[0017] In another scenario, where the third sample set corresponds to a single-source fine-grained labeled sample set, and this single-source fine-grained labeled sample set is a model-labeled sample set (the labeled sample set obtained through model labeling is the same as the labeled sample set obtained through manual labeling), if the score of the network anomaly detection model is lower than a predetermined score, the sampling frequency of this third sample set is reduced. Optionally, in this case, the user can be visually prompted to introduce manual labeling feedback, i.e., to increase manual labeling, thereby forming a manually labeled sample set.

[0018] In another case, the third sample set corresponds to a multi-source fine-grained labeled sample set (e.g., a manually labeled sample set and a model labeled sample set). If the score of the anomaly detection model of the network is lower than a predetermined score, the sampling frequency of the third sample set corresponding to the model labeled sample set is reduced; or the sampling frequency of the third sample set corresponding to the model labeled sample set is reduced, and the sampling frequency of the third sample set corresponding to the manually labeled sample set is increased.

[0019] In this embodiment, the network anomaly detection model can be evaluated manually to obtain evaluation information, which can be user scores of the anomaly detection model. If the score of the network anomaly detection model is low, it indicates a problem in the training process. One possible reason is the low accuracy of the training sample set (i.e., the incremental sample set). Therefore, when the score of the network anomaly detection model is lower than a predetermined score, the sampling frequency of the third sample set is adjusted to update the second sample set, which in turn updates the incremental sample set, improving its accuracy. Furthermore, the network anomaly detection model is optimized based on the updated incremental sample set.

[0020] In one possible implementation, the method further includes: detecting the KPI data of multiple network devices based on the updated network anomaly detection model and an unsupervised algorithm to determine the status of the KPI data of the multiple network devices. In this embodiment, since the unsupervised algorithm is a statistical, highly generalized, and loosely conditional algorithm, its accuracy in detecting KPI data is not high. Therefore, using the network anomaly detection model to further detect abnormal KPI data in the unsupervised algorithm results can ensure detection accuracy and reduce false positives. Furthermore, since the unsupervised algorithm runs quickly, combining the network anomaly detection model with the unsupervised algorithm to determine the status of the network device's KPI data can improve the efficiency of network anomaly detection.

[0021] In one possible implementation, the aforementioned unsupervised algorithm could be the N sigma algorithm, the box plot algorithm, or an algorithm based on the jump magnitude of historical data.

[0022] Secondly, this application provides a network anomaly detection device, which includes an acquisition module, a determination module, and an update module. The acquisition module acquires at least one labeled sample set, which is a sample set annotated based on the annotation feedback information of a test sample set. The annotation feedback information indicates whether multiple network devices corresponding to the test sample set are in a normal or abnormal state, and / or indicates whether the KPI data of multiple network devices in the test sample set are in a normal or abnormal state. The determination module determines an incremental sample set based on the at least one labeled sample set. The update module updates the network anomaly detection model based on the incremental sample set. The network anomaly detection model detects the KPI data of multiple network devices in the test sample set to determine the state of the KPI data of the multiple network devices.

[0023] In one possible implementation, the annotation feedback information of the aforementioned test sample set includes coarse-grained annotation feedback information and / or fine-grained annotation feedback information; wherein, the coarse-grained annotation feedback information is used to indicate whether the multiple network devices corresponding to the test sample set are in a normal or abnormal state, and the coarse-grained annotation feedback information is obtained by annotating the network devices based on at least one of the network device's latency, packet loss rate, or effective time; the fine-grained annotation feedback information is used to indicate whether the KPI data of the multiple network devices in the test sample set are in a normal or abnormal state, and the fine-grained feedback information is obtained by annotating the network device's KPI data based on manual and / or network anomaly detection models.

[0024] In one possible implementation, the annotation feedback information of the above test sample set includes N types of annotation feedback information, which include X types of coarse-grained annotation feedback information and Y types of fine-grained annotation feedback information; wherein, each type of annotation feedback information in the N types of annotation feedback information corresponds to an annotation sample set, N is a positive integer greater than or equal to 2, X+Y=N, X is a positive integer greater than or equal to 0, and Y is an integer greater than or equal to 1;

[0025] The aforementioned determining module is specifically used to determine the first sample set, which is the intersection of N labeled sample sets;

[0026] Specifically, the determination module is also used to determine Y second sample sets, wherein the i-th second sample set among the Y second sample sets is determined according to the sampling frequency w. i The i-th third sample set is obtained by sampling; wherein the i-th third sample set is the difference between the i-th labeled sample set and the first sample set among the Y labeled sample sets corresponding to the above Y fine-grained labeled feedback information; the determination module is further used to determine the union of the above first sample set and the above Y second sample sets as the incremental sample set.

[0027] In one possible implementation, the network anomaly detection device provided in this application embodiment further includes a feature extraction module; the feature extraction module is used to extract features from the incremental sample set to obtain a feature set of the incremental sample set, the feature set of the incremental sample set including at least one of the following features: mean, variance, extreme values, entropy, difference value, and change ratio of KPI data; the update module is specifically used to train the network anomaly detection model based on the feature set of the incremental sample set using an incremental training algorithm to obtain an updated network anomaly detection model; wherein, the incremental training algorithm includes any one of the following: incremental tree algorithm, support vector machine algorithm, and convolutional neural network algorithm.

[0028] In one possible implementation, the network anomaly detection device provided in this application embodiment further includes an equalization processing module; the equalization processing module is used to perform sample equalization processing on the incremental sample set so that the ratio of positive samples to negative samples in the incremental sample set reaches a predetermined ratio; the negative samples are samples whose KPI data is in an abnormal state, and the positive samples are samples whose KPI data is in a normal state.

[0029] In one possible implementation, the network anomaly detection device provided in this application embodiment further includes an adjustment module: the adjustment module is used to adjust the sampling frequency of the third sample set according to the user's evaluation information on the updated network anomaly detection model.

[0030] In one possible implementation, the network anomaly detection device provided in this application embodiment further includes a detection module: the detection module is used to detect the KPI data of multiple network devices based on the updated network anomaly detection model and unsupervised algorithm, and determine the status of the KPI data of multiple network devices.

[0031] Thirdly, an embodiment of this application provides a network anomaly detection device, which includes a memory and a processor, the memory being coupled to the processor; the memory is used to store computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the network anomaly detection device performs any of the methods of the first aspect.

[0032] Fourthly, an embodiment of this application provides a computer storage medium including computer instructions that, when executed on a network anomaly detection device, cause the network anomaly detection device to perform any of the methods of the first aspect.

[0033] Fifthly, an embodiment of this application provides a computer program product that, when run on a computer, causes the computer to perform any of the methods in the first aspect.

[0034] It should be understood that the beneficial effects achieved by the second to fifth aspects of the technical solutions and the corresponding possible implementations of the embodiments of this application can be referred to the above-described technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0035] Figure 1 A schematic flowchart of a network anomaly detection method provided in this application embodiment. Figure 1 ;

[0036] Figure 2 A schematic diagram of an anomaly recognition framework based on user feedback provided in this application embodiment. Figure 1 ;

[0037] Figure 3 A schematic flowchart of a network anomaly detection method provided in this application embodiment. Figure 2 ;

[0038] Figure 4 A schematic flowchart of a network anomaly detection method provided in this application embodiment. Figure 3 ;

[0039] Figure 5 Hardware schematic of a network anomaly detection device provided in this application embodiment Figure 1 ;

[0040] Figure 6aA schematic diagram of the structure of a network anomaly detection device provided in this application embodiment. Figure 1 ;

[0041] Figure 6b A schematic diagram of the structure of a network anomaly detection device provided in this application embodiment. Figure 2 ;

[0042] Figure 7 A schematic diagram of a network anomaly detection method framework provided in this application embodiment. Figure 1 ;

[0043] Figure 8 A schematic flowchart of a network anomaly detection method provided in this application embodiment. Figure 4 ;

[0044] Figure 9 A flowchart illustrating a method for determining an incremental sample set provided in this application embodiment. Figure 1 ;

[0045] Figure 10 A method flow for determining the type of a labeled sample set provided in this application embodiment Figure 1 ;

[0046] Figure 11 A flowchart illustrating a method for updating a network anomaly detection model provided in this application embodiment. Figure 1 ;

[0047] Figure 12 A schematic flowchart of a network anomaly detection method provided in this application embodiment. Figure 5 ;

[0048] Figure 13 A schematic diagram of a network anomaly detection method provided in this application embodiment is shown in Figure 6.

[0049] Figure 14 A schematic flowchart of a network anomaly detection method provided in this application embodiment. Figure 7 ;

[0050] Figure 15 A schematic diagram of the structure of a network anomaly detection device provided in this application embodiment. Figure 3 ;

[0051] Figure 16 A schematic diagram of the structure of a network anomaly detection device provided in this application embodiment. Figure 4 . Detailed Implementation

[0052] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0053] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, rather than to describe a specific order of objects. For example, "first sample set" and "second sample set," etc., are used to distinguish different sample sets, rather than to describe a specific order of sample sets.

[0054] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0055] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple network devices refer to two or more processing units; multiple systems refer to two or more systems.

[0056] First, some concepts involved in the network anomaly detection method and apparatus provided in the embodiments of this application will be explained.

[0057] Key performance indicators (KPIs): In this application embodiment, for network devices (such as routers or switches) in the network, the KPIs of the network devices may include, but are not limited to, at least one of the following: number of lost packets or packet loss rate, traffic volume, percentage of erroneous packets, processor (central processing unit, CPU) utilization, or number of protocol received packets.

[0058] For example, when the network device is a router, the KPIs of the aforementioned network device may also include indicators such as multicast internal table entry aging count, the number of neighbor devices going offline (neighbor device down count), and static or dynamic neighbor device technology, etc.

[0059] With the development of technology, people are paying more and more attention to network health, and it is becoming increasingly important to adopt appropriate methods to detect network anomalies and determine whether devices in the network are in an abnormal state.

[0060] Combination Figure 1In one existing implementation of network anomaly detection, the data module of the device used for network anomaly detection acquires at least one of the following metrics of the network device to be detected: CPU load rate, memory read / write speed, or network I / O size, forming a dataset. Then, a low-pass filter is used to filter the dataset, removing data with small variations. A preset detection method is then used to detect the filtered dataset, obtaining the detection result. Finally, an anomaly probability of the dataset is obtained through Z-testing and a Gaussian algorithm, and this probability is compared with a predetermined probability to determine whether an anomaly exists in the dataset. The preset detection method may include at least one of the following: a smoothed data point detection method based on a Gaussian distribution model; a detection method that compares the absolute value of the difference between a single data point in the dataset and the average value of the dataset with a threshold of 1; a detection method that compares the second derivative of the average value of the dataset with a threshold of 2; or an autoregressive algorithm based on a Gaussian model.

[0061] Since the above method is an unsupervised learning method that detects anomalies by finding patterns in the dataset, if the distribution of the dataset fluctuates, the detection method cannot quickly adapt and adjust, which leads to a decrease in the accuracy of the detection results.

[0062] Combination Figure 2 The anomaly detection framework based on user feedback shown involves the model training module of the device used for network anomaly detection sending the initial or previously trained recognition model (model 1) to the anomaly detection module. The anomaly detection module performs anomaly detection based on this recognition model (model 1) and sends the detection results to the visualization module through the model training module. The visualization module provides feedback based on the detection results of the recognition model (model 1) and sends the feedback information back to the model training module, thereby enabling the model training module to adjust the recognition model (model 1) based on the feedback information.

[0063] The above-mentioned technical solutions also require manual feedback on the model's detection results, which is inefficient. In addition, since the feedback results from manual methods are not comprehensive and may produce erroneous results, the accuracy of anomaly detection is reduced when the model trained based on these feedback results performs anomaly detection.

[0064] Combination Figure 3As shown, in another network anomaly detection method, a classification model M1 is trained on the training dataset D1 based on a supervised learning classification algorithm. Then, the classification model M1 is used to detect anomalies in network traffic data (i.e., the test dataset), and the test results of the test dataset are obtained, which is called the labeled test dataset D2. Further, a semi-supervised learning method, such as the improved K-Nearest Neighbors (KNN) algorithm, is used to cluster the test dataset to obtain the labeled test dataset D3. Then, the labeled test dataset D2 and the labeled test dataset D3 are compared. For inconsistent labels, the label of the test dataset D2 is corrected based on the label of the test dataset D3. The corrected labeled test dataset is added as a new sample to the training dataset D1. Finally, the updated training dataset is trained to obtain the classification model M2. Then, the classification accuracy of the classification model M1 and the classification model M2 are compared. If the classification accuracy of the classification model M2 is higher than that of the classification model M1, then the classification model M2 is used to replace the classification model M1.

[0065] The clustering of the test dataset in the aforementioned network anomaly detection method is based on the KNN semi-supervised learning method. In the semi-supervised KNN algorithm, the labeled samples in the training sample set are manually labeled, which results in low efficiency for network anomaly detection. In addition, the above method adds the corrected labeled test dataset to the training dataset D1 to continue training the classification model. As time increases, the number of samples in the training dataset increases, which will increase the computational load of the classification model training process and thus increase the demand for computing resources.

[0066] The three existing anomaly detection methods mentioned above all suffer from low accuracy. Therefore, this application provides a network anomaly detection method and apparatus. The apparatus acquires at least one labeled sample set, which is a sample set annotated based on the annotation feedback information of a test sample set. Based on the at least one labeled sample set, an incremental sample set is determined; and based on the incremental sample set, the network anomaly detection model is updated. The technical solution provided by this application provides a more accurate network anomaly detection model, thereby improving the accuracy of anomaly detection.

[0067] The network anomaly detection method provided in this application can provide a detection framework for a detection method that can be Figure 4The detection framework shown includes a model selector (MS), a multiple-reaction tagger (MT), and a device-processing (DP). The MT determines an incremental sample set based on at least one set of labeled samples; the MS trains the incremental sample set to obtain a new anomaly detection model and sends this new model or an initial anomaly detection model to the DP; the DP performs anomaly detection on the network device's KPI data based on the anomaly detection model sent by the MS.

[0068] Combination Figure 4 The aforementioned MS can send the newly trained anomaly detection model or the initial anomaly detection model to the DP. Within the MS, KPI data sent by the DP can be manually labeled, and the labeling results can be fed back to the MT. The MS can also score the newly trained anomaly detection model and feed the score back to the MT.

[0069] Combination Figure 4 DP can feed back the detection results of KPI data to MT, and DP can collect KPI data and send the KPI data to MS.

[0070] Combination Figure 4 The MT determines the incremental sample set based on the annotation results sent by the MS, the detection results of the KPI data sent by the DP, and other annotation results. This incremental sample set is then sent to the MS, or alternatively to the DP for use. Optionally, the MT can update the incremental sample set based on the scores provided by the MS.

[0071] In this embodiment, the apparatus executing the network anomaly detection method is referred to as a network anomaly detection device. The network anomaly detection device can be a computer, a switching device, or a routing device. This embodiment does not impose any special limitations on the specific form of the network anomaly detection device. The following embodiments use a computer as an example to describe the hardware structure of the network anomaly detection device.

[0072] Figure 5 This is a hardware schematic diagram of the network anomaly detection device provided in the embodiments of this application, such as... Figure 5 As shown, the network anomaly detection device may include a processor 501, a memory 502, and a network interface 503.

[0073] The processor 501 includes one or more central processing units (CPUs). The CPU can be a single-core CPU or a multi-core CPU.

[0074] The memory 502 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical memory, or disk memory.

[0075] Optionally, the processor 501 implements the network anomaly detection method provided in this application embodiment by internally stored instructions, or the processor 501 implements the network anomaly detection method provided in this application embodiment by reading instructions stored in the memory 502. When the processor 501 implements the method in the above embodiment by reading instructions stored in the memory 502, the memory 502 stores instructions for implementing the network anomaly detection method provided in this application embodiment.

[0076] Network interface 503 is a wired interface (port), such as a fiber distributed data interface (FDDI) or a gigabit Ethernet (GE) interface. Alternatively, network interface 503 can be a wireless interface. It should be understood that network interface 503 includes multiple physical ports and is used to output network anomaly detection results or for communication with other devices.

[0077] The methods in the following embodiments can all be used in situations where... Figure 5 The network anomaly detection device with the hardware structure shown is implemented in the embodiment of this application. The network anomaly detection device has the above-mentioned features. Figure 4 The functions of MS, MT, and DP in the detection framework shown.

[0078] Optionally, in this embodiment, the functions of MS, MT, and DP described above can be integrated together, for example, integrated on a single computer (e.g., a personal computer or server). The functions of MS, MT, and DP can also be implemented by different physical devices. For example, MS can be a computer with processing and storage functions; MT can be a computer, or a device such as a switch or router, and MT can be in a distributed cluster mode or a centralized mode; DP can be a computer, or a device such as a switch or router, and DP can be in a distributed cluster mode or a centralized mode.

[0079] Accordingly, Figure 5 The network anomaly detection device shown can be as follows: Figure 6a The fully centralized structure shown can also be as follows: Figure 6b The device shown is a fully distributed structure.

[0080] exist Figure 6a In the fully centralized structure shown, on one hand, the MS distributes the anomaly detection model to the main DP, which in turn distributes the received anomaly detection model to all sub-DPs. Each sub-DP then performs anomaly detection on its local KPI data (i.e., test samples) based on the anomaly detection model distributed by the MS, obtaining its own KPI data detection results. In this embodiment, the set of KPI data detection results obtained based on the anomaly detection model can be called the model-annotated sample set. Each sub-DP then reports its own model-annotated sample set to the main DP. After aggregation on the main DP, the main DP sends the aggregated model-annotated sample set to the main MT. On the other hand, all sub-DPs collect KPI data from their respective sub-devices and report the KPI data to the main DP. After aggregation on the main DP, the main DP sends the aggregated KPI data to the MS. Then, on the visualization interaction module deployed on the MS, these KPI data are annotated based on human (or expert) experience, indicating whether the KPI data is in a normal or abnormal state, thus forming a manually annotated sample set.

[0081] exist Figure 6a In this context, it can be understood that the main MT can receive the set of model-annotated samples (including the processing results of different sub-DPs) sent by the main DP, and the main MT can also receive the set of manually annotated samples (including the manually annotated results of KPI data collected by different sub-DPs) sent by the MS. Then, the main MT divides the set of model-annotated samples into multiple subsets of model-annotated samples (for example, the processing results corresponding to each sub-DP form a subset of model-annotated samples), and sends multiple subsets of model-annotated samples to the corresponding sub-MTs. Similarly, the main MT divides the set of manually annotated samples into multiple subsets of manually annotated samples (for example, the manually annotated results of KPI data collected by each sub-DP form a subset of manually annotated samples).

[0082] exist Figure 6a In this process, each sub-MT receives a subset of model-annotated samples and a subset of manually annotated samples. Then, each sub-MT obtains the tool-annotated sample subset of the sub-device corresponding to the aforementioned subset of model-annotated samples or manually annotated samples. It should be understood that the tool-annotated sample subset is a set composed of the results of annotating the device-level indicator data by an automatic annotation tool (also known as a coarse-grained annotation tool). Further, the sub-MT determines the incremental sample subset based on the subset of model-annotated samples, the subset of manually annotated samples, and the subset of tool-annotated samples. Then, each sub-MT sends its determined incremental sample subset to the master MT, which merges the multiple incremental sample subsets to obtain the incremental sample set.

[0083] Furthermore, the main MT feeds back the incremental sample set to the MS, which trains the incremental sample set to obtain a new anomaly detection model and distributes the new anomaly detection model to the main DP. Thus, the main DP can distribute the new anomaly detection model to each sub-DP.

[0084] Optionally, in this embodiment, the main MT may also feed back the incremental sample set to the main DP, which will then distribute the incremental sample set to different sub-DPs for use by the sub-DPs.

[0085] Optionally, the MS can also score the newly trained anomaly detection model and feed the score back to each sub-MT through the main MT. In this way, each sub-MT updates its own incremental sample subset based on the score.

[0086] Optionally, in the embodiments of this application, the aforementioned sub-DP and corresponding sub-MT can be integrated into a single device, or they can be implemented through different devices. The embodiments of this application do not impose specific limitations.

[0087] exist Figure 6b In the fully distributed structure shown, on one hand, sub-DP1, sub-DP2, and sub-DP3 each collect KPI data from their respective sub-devices. Then, based on the anomaly detection model issued by the MS, they train their respective collected KPI data to obtain their respective model-annotated sample sets. They then send their respective model-annotated sample sets to the corresponding sub-MT on the MT side. On the other hand, all sub-DPs send the collected KPI data to the MS, where a visualization and interactive module deployed on the MS annotates these KPI data based on human (or expert) experience, marking them as normal or abnormal, thus forming a manually annotated sample set.

[0088] exist Figure 6b In this context, it is understood that each of the aforementioned sub-MTs can receive the model annotation sample set sent by the corresponding sub-DP, and each sub-MT can also receive the manual annotation sample set sent by the MS for the corresponding device (for example, sub-MT1 receives the manual annotation sample set sent by the MS for manually annotating the KPI data on sub-DP1 device).

[0089] exist Figure 6bIn this process, each sub-MT receives a subset of model-annotated samples and a subset of manually annotated samples. Then, each sub-MT obtains the tool-annotated sample subset corresponding to the aforementioned subset of model-annotated samples or manually annotated samples. It should be understood that the tool-annotated sample subset is a set of results from automatic annotation tools (also known as coarse-grained annotation tools) annotating the device-level indicator data. Furthermore, the sub-MT determines the incremental sample subset based on the subset of model-annotated samples, the subset of manually annotated samples, and the subset of tool-annotated samples.

[0090] Furthermore, each sub-MT feeds back its determined subset of incremental samples to the MS. The MS merges the subsets of incremental samples determined by each sub-MT to obtain an incremental sample set. The MS trains the incremental sample set to obtain a new anomaly detection model and distributes the new anomaly detection model to each DP.

[0091] Optionally, in this embodiment of the application, each sub-MT may also feed back its determined incremental sample set to its corresponding sub-DP for use by the sub-DP.

[0092] Optionally, the MS can also score the newly trained anomaly detection model and feed the score back to each sub-MT, so that each sub-MT can update its own incremental sample subset based on the score.

[0093] Below, in conjunction with Figure 7 The complete detection framework shown provides a detailed description of the network anomaly detection method provided in the embodiments of this application. For example... Figure 8 As shown, the network anomaly detection method provided in this application embodiment may include S801-S803.

[0094] S801. Obtain at least one set of labeled samples.

[0095] The labeled sample set is a set of samples annotated based on the annotation feedback information of the test sample set. In essence, each sample in the labeled sample set is labeled as either in an abnormal or normal state; that is, all samples in the labeled sample set are labeled. The annotation feedback information is used to indicate whether the multiple network devices corresponding to the test sample set are in a normal or abnormal state, and / or to indicate whether the KPI data of the multiple network devices in the test sample set are in a normal or abnormal state.

[0096] Optionally, the annotation feedback information of the above test sample set includes coarse-grained annotation feedback information and / or fine-grained annotation feedback information.

[0097] In this embodiment of the application, coarse-grained annotation feedback information is used to indicate whether the multiple network devices corresponding to the test sample set are in a normal or abnormal state.

[0098] Optionally, coarse-grained annotation feedback information is obtained by annotating the network device based on at least one of its latency, packet loss rate, or effective time. Effective time refers to the duration during which the network device operates normally within a certain period (e.g., a specific observation period). It should be understood that at least one of the aforementioned latency, packet loss rate, or effective time is a metric used to evaluate the network device as a whole, reflecting its current state. For example, high latency indicates a potential malfunction; a high packet loss rate indicates a potential malfunction; and a short effective time indicates a potential malfunction.

[0099] It should be understood that a coarse-grained annotation tool (hereinafter referred to as an automatic annotation tool) can be constructed based on at least one of the network device's latency, packet loss rate, or effective time. Based on the automatic annotation tool, coarse-grained annotation feedback information of the network device is obtained, and the network device is annotated based on the coarse-grained annotation feedback information to obtain the coarse-grained annotation result of the network device.

[0100] In this embodiment of the application, when coarse-grained annotation feedback information is used to annotate the test sample set, the test sample set consists of information from multiple network devices (i.e., at least one of the aforementioned latency, packet loss, or effective time). An automatic annotation tool annotates this test sample set to obtain coarse-grained annotation results for each network device, thus forming a coarse-grained annotated sample set. For ease of description, this coarse-grained annotated sample set is referred to as the tool-annotated sample set. For example... Figure 7 The tool-annotated sample set 201.

[0101] In this embodiment of the application, fine-grained annotation feedback information is used to indicate whether the KPI data of multiple network devices in the test sample set are in a normal or abnormal state.

[0102] The KPIs of a network device may include, but are not limited to, at least one of the following: packet loss count or packet loss rate, traffic volume, percentage of erroneous packets, CPU utilization, or number of protocol-received packets. Optionally, the KPIs of a network device may be KPI data corresponding to one or more modules in the network device, which is not limited in this embodiment.

[0103] Optionally, a network device's KPI data may include multiple KPI data, where one KPI data may correspond to the KPI data of a module (e.g., an interface) in the network device.

[0104] Optionally, in this embodiment of the application, a KPI data of a network device can be a KPI data at a single moment, or it can be a sequence of multiple KPI data over a period of time (referred to as KPI sequence data).

[0105] Optionally, the fine-grained feedback information is obtained by labeling the KPI data of the network device based on at least one of the following: manual annotation, a network anomaly detection model, or a combination of a network anomaly detection model and an unsupervised algorithm. In this embodiment, when labeling the test sample set according to the fine-grained annotation feedback information, the test sample set consists of KPI data from multiple network devices. Each KPI data in the test sample set is labeled using the fine-grained annotation feedback information to obtain the fine-grained annotation result, thereby forming a fine-grained annotation set. For ease of description, the fine-grained annotation sample set obtained by using the fine-grained feedback information obtained manually is called the manually labeled sample set, for example... Figure 7 The manually labeled sample set 202; the fine-grained labeled sample set obtained by using the fine-grained feedback information corresponding to the network anomaly detection model is called the model labeled sample set, for example. Figure 7 The model annotation sample set is 203.

[0106] For example, suppose the test sample set consists of KPI data from 4 devices. The contents of the test sample set are shown in Table 1. The test sample set is labeled by the network anomaly detection model to obtain the model labeled sample set. The contents of the model labeled sample set are shown in Table 2. The value of each KPI data in the model labeled sample set corresponds to a state, which includes a normal state or an abnormal state.

[0107] Table 1

[0108]

[0109] For ease of description, Table 1 uses KPI data at a single moment as an example.

[0110] Table 2

[0111]

[0112] It should be noted that the above-mentioned network devices belong to the same network (i.e., the same site). Correspondingly, the anomaly detection is to detect the status of the KPI data of the network devices in the network.

[0113] Combination Figure 7The aforementioned manually labeled sample set 202 comes from the output of the manual visualization labeling module 107 on the MS side. Specifically, the data subscription module 301 on the DP side can subscribe to the KPI data of the device on the DP side, so that the DP side collects KPI data and stores it in the data cache module 302. The DP side can upload the collected KPI data to the MT side. On the MS side, the KPI data can be stored in the data monitoring storage module 106. The manual visualization labeling module 107 reads the KPI data from the monitoring storage module 106, and then labels the KPI data manually to obtain the manually labeled sample set, and feeds it back to the MT side.

[0114] Combination Figure 7 The aforementioned model-annotated sample set 203 comes from the detection results of the KPI data by the anomaly detection module 303 on the DP side. Specifically, the anomaly detection module 303 on the DP side detects the KPI data according to the anomaly detection model issued by the MS side, and then obtains the model-annotated sample set.

[0115] Optionally, the anomaly detection model issued by the MS side may be the initial anomaly detection model, denoted as Model A, or it may be the anomaly detection model obtained from the previous training, denoted as Model C.

[0116] When the solution of the embodiment of this application is first implemented, the initial anomaly detection model on the MS side, i.e., model A above, can be Figure 7 The platform side is based on the anomaly detection model trained on the KPI data (also known as global KPI data) of network devices in multiple networks. The training process of model A includes data preprocessing, feature extraction and model training using a predetermined algorithm.

[0117] It is understandable that the anomaly detection model trained based on global KPI data may not perform well when detecting a specific location.

[0118] The anomaly detection model obtained through the above training is an anomaly detection model updated by the MS based on the feedback information from the MT, which will be described in detail in the following embodiments.

[0119] S802. Determine the incremental sample set based on at least one labeled sample set.

[0120] In this embodiment of the application, the annotation feedback information of the test sample set includes N types of annotation feedback information, which include X types of coarse-grained annotation feedback information and Y types of fine-grained annotation feedback information. Each of the N types of annotation feedback information corresponds to a set of annotated samples, where N is a positive integer greater than or equal to 2, X + Y = N, X is a positive integer greater than or equal to 0, and Y is an integer greater than or equal to 1.

[0121] Based on the above description of S801, for example, the set of labeled samples corresponding to the above X types of coarse-grained annotation feedback information (i.e., the annotation results of the test sample set annotated with coarse-grained annotation feedback information) may include the tool-labeled sample set obtained by the above automated annotation tool. The set of labeled samples corresponding to the above Y types of coarse-grained annotation feedback information (i.e., the annotation results of the test sample set annotated with fine-grained annotation feedback information) includes the model-labeled sample set and the manually labeled sample set.

[0122] Optional, such as Figure 9 As shown, the method for determining the incremental sample set based on at least one labeled sample set obtained in S801 may specifically include S901-S903.

[0123] S901. Determine the first sample set, which is the intersection of the N labeled sample sets obtained.

[0124] It should be understood that each of the N types of annotation feedback information corresponds to a set of annotation samples, and the N types of annotation feedback information correspond to N sets of annotation samples, which include X sets of coarse-grained annotation samples and Y sets of fine-grained annotation samples.

[0125] In the first implementation, when X = 0, all N labeled sample sets can be fine-grained labeled sample sets, and the first sample set is the intersection of the N fine-grained labeled sample sets.

[0126] For example, assuming N=2 and X=0, the obtained labeled sample set includes two fine-grained labeled sample sets, such as the manually labeled sample set and the model labeled sample set. The contents of the manually labeled sample set and the model labeled sample set are shown in Table 3 below. The intersection of the KPI data and the corresponding states of the manually labeled sample set and the model labeled sample set in Table 3 is used to obtain the first sample set as shown in Table 4.

[0127] Table 3

[0128]

[0129]

[0130] Table 4

[0131]

[0132] In Table 4 above, " / " indicates an empty state, meaning that there is no KPI data and therefore no such KPI data exists. The same applies to subsequent tables, so we will not repeat the details.

[0133] Optionally, the method for finding the intersection of the manually labeled sample set and the model labeled sample set in Table 3 above is: based on the state of the KPI data, take the intersection of the KPI data of a certain device in the manually labeled sample set and the corresponding KPI data of the corresponding device in the model labeled sample set.

[0134] For example, as shown in Table 3, taking the number of packet losses as KPI data, for device C, the state corresponding to 20 packet losses for device C in the manually labeled sample set is normal, while the state corresponding to 20 packet losses for device C in the model labeled sample set is abnormal. Therefore, the intersection of the manually labeled sample set and the model labeled sample set does not include the number of packet losses for device C and its corresponding state (represented by " / " in Table 3). Similarly, for another example, taking the error packet percentage as KPI data, for device C, the state corresponding to 30% error packet percentage for device C in the manually labeled sample set is abnormal, and the state corresponding to 30% error packet percentage for device C in the model labeled sample set is also abnormal. Therefore, the intersection of the manually labeled sample set and the model labeled sample set includes the error packet percentage for device C, that is, the 30% error packet percentage for device C and its corresponding abnormal state are elements of the intersection of the manually labeled sample set and the model labeled sample set.

[0135] In the second implementation, X≠0 and Y≠0, then the first sample set is the intersection of X sets of coarse-grained labeled samples and Y sets of fine-grained labeled samples.

[0136] For example, suppose N=2, X=1, Y=1, where a fine-grained set of labeled samples can be either a model-labeled sample set or a manually labeled sample set. In the example in Table 5 below, the fine-grained set is the model-labeled sample set, and the coarse-grained set of tool-labeled sample sets contains time delay as the metric data, as shown in Table 6.

[0137] Table 5

[0138]

[0139] Table 6

[0140]

[0141] Optionally, the method for finding the intersection of the model-labeled sample set and the tool-labeled sample set in Table 5 above is as follows: In the model-labeled sample set, take the KPI data with the same device status as in the tool-labeled sample set as the element of the intersection of the model-labeled sample set and the tool-labeled sample set. For example, if the device status of a certain device in the tool-labeled sample set is abnormal, then take the KPI data of the corresponding abnormal status in the model-labeled sample set as the element of the intersection of the model-labeled sample set and the tool-labeled sample set.

[0142] For example, in Table 5 above, taking the KPI data of device A as an example, in the tool annotation sample set, the device status of device A is normal; in the model annotation sample set, the number of packet losses for device A is 10, and the status of this KPI data is normal; furthermore, in the model annotation sample set, the packet error rate for device A is 10%, and the status of this KPI data is normal; in the model annotation sample set, the CPU utilization rate for device A is 60%, and the status of this KPI data is abnormal. Thus, it can be seen that the status of the number of packet losses and the packet error rate for device A in the model annotation sample set is consistent with the device status of device A in the tool annotation sample set. Therefore, the number of packet losses of 10 and the corresponding normal status, and the packet error rate of 10% and its corresponding normal status are all elements in the intersection of the model annotation sample set and the tool annotation sample set.

[0143] S902. Determine Y sets of second samples.

[0144] The i-th second sample set among the aforementioned Y second sample sets is determined according to the sampling frequency w. i The i-th third sample set is obtained by sampling. This i-th third sample set is the difference between the i-th labeled sample set and the first sample set among the Y labeled sample sets corresponding to Y types of fine-grained labeled feedback information.

[0145] Based on the first implementation method described above, for the implementation method where all N labeled sample sets are fine-grained labeled sample sets, firstly, combining the examples corresponding to Tables 3 and 4 above, two third sample sets can be determined. The first third sample set is the difference between the manually labeled sample set in Table 3 and the first sample set shown in Table 4, and the second third sample set is the difference between the model labeled sample set in Table 3 and the first sample set shown in Table 4. For example, the contents of the first third sample set are shown in Table 7 below, and the contents of the second third sample set are shown in Table 8 below. Then, the first third sample set is sampled according to sampling frequency w1 to obtain the first second sample set, and the second third sample set is sampled according to sampling frequency w2 to obtain the second second sample set.

[0146] Table 7

[0147]

[0148] Table 8

[0149]

[0150]

[0151] In the model annotation sample set shown in Table 3 above, the KPI data of device C and the state corresponding to the KPI data are removed from the first sample set in Table 4 above to obtain the first third sample set, which is the third sample set shown in Table 6.

[0152] Based on the second implementation method described above, for the aforementioned N labeled sample sets, including X coarse-grained labeled sample sets and Y fine-grained implementation methods, firstly, by referring to the examples corresponding to Tables 5 and 6, a third sample set can be determined. This third sample set is the difference between the model labeled sample set in Table 5 and the first sample set shown in Table 6. For example, the contents of this third sample set are shown in Table 9 below. Then, the third sample set is sampled according to the sampling frequency w to obtain the second sample set.

[0153] Table 9

[0154]

[0155] Optionally, in the above implementation of N labeled sample sets including X coarse-grained labeled sample sets and Y fine-grained labeled sample sets, the number of coarse-grained and fine-grained labeled sample sets can be set according to actual needs. For example, the labeled sample set includes 1 coarse-grained labeled sample set and 2 fine-grained labeled sample sets, where the 2 fine-grained labeled sample sets can be a manual labeled sample set and a model labeled sample set, respectively. In the process of determining the incremental sample set, firstly, the intersection of the three labeled sample sets is obtained to get the first sample set. Then, the difference between the aforementioned manual labeled sample set and the model labeled sample set and the first sample set obtained above is calculated to obtain two third sample sets. Finally, two second sample sets are determined based on the two third sample sets. The specific method can be referred to the description of the above embodiment, and will not be detailed here.

[0156] S903. The union of the first sample set and Y second sample sets is determined as the incremental sample set.

[0157] It should be understood that the incremental sample set includes KPI data from multiple network devices, which are KPI data marked as abnormal or normal states.

[0158] Optionally, after obtaining at least one set of labeled samples (i.e., S801), the network anomaly detection method provided in this application embodiment includes determining which type of labeled sample set the at least one set of labeled samples is, and executing the processes S901 to S903 according to the determination result.

[0159] Specifically, such as Figure 10 As shown, the process of determining the type of at least one set of labeled samples obtained includes:

[0160] S1001. Determine whether a coarse-grained set of labeled samples (i.e., a tool-labeled set) exists in at least one set of labeled samples obtained.

[0161] In this embodiment of the application, if at least one set of labeled samples contains a coarse-grained set of labeled samples, then S1002 is executed; if at least one set of labeled samples obtained above does not contain a coarse-grained set of labeled samples, but only a fine-grained set of labeled samples exists, then S1004 is executed.

[0162] S1002. Determine whether the at least one set of labeled samples obtained contains only a set of coarse-grained labeled samples.

[0163] In this embodiment, if the acquired at least one set of labeled samples contains only a coarse-grained set of labeled samples, since the coarse-grained set of labeled samples can only indicate the state of the network device and not the state of the network device's KPI data, the incremental set of labeled samples cannot be determined based on this coarse-grained set of labeled samples. In this case, the incremental set of labeled samples is not fed back to the MS side, that is, S901-S903 is not executed, and the model training ends without generating a new network anomaly detection model. If both a coarse-grained set of labeled samples and a fine-grained set of labeled samples exist, S1003 is executed.

[0164] S1003. Determine the incremental sample set based on the coarse-grained labeled sample set and the fine-grained labeled sample set.

[0165] It should be understood that the method for determining the incremental sample set based on the coarse-grained labeled sample set and the fine-grained labeled sample set is implemented according to the second implementation method in S901 to S903. For details, please refer to the description of the above embodiments, which will not be repeated here.

[0166] S1004. Determine whether the fine-grained labeled sample set is a single-source fine-grained labeled sample set.

[0167] It should be understood that the aforementioned single-source fine-grained annotation sample set refers to a single, one-type fine-grained annotation sample set. For example, the fine-grained annotation sample set may only include manually annotated sample sets or only model-annotated sample sets.

[0168] If the fine-grained labeled sample set is a single-source fine-grained labeled sample set, then execute S1005; if the fine-grained labeled sample set includes multi-source fine-grained labeled sample sets, then execute S1008.

[0169] S1005. Determine whether the set of fine-grained labeled samples from a single source is the set of labeled samples for the model.

[0170] If the above-mentioned single-source fine-grained labeled sample set is the model labeled sample set, then execute S1006; otherwise, execute S1007.

[0171] S1006. Determine whether the labeled sample set of the model is obtained by combining the anomaly detection model and other algorithms.

[0172] If the above model-labeled sample set is obtained by combining the anomaly detection model with other algorithms, then execute S1007; otherwise, end the model training and do not generate a new network anomaly detection model.

[0173] S1007. Determine the incremental sample set based on the single-source fine-grained labeled sample set.

[0174] In this embodiment of the application, determining the incremental sample set based on the single-source fine-grained labeled sample set specifically includes using the single-source fine-grained labeled sample set as the incremental sample set.

[0175] If the aforementioned single-source fine-grained labeled sample set is not the model labeled sample set, for example, if the single-source fine-grained labeled sample set is a manually labeled sample set, then the manually labeled sample set is used as the incremental sample set.

[0176] If the above model-labeled sample set is obtained by combining an anomaly detection model and other algorithms, then the model-labeled sample set is used as the incremental sample set.

[0177] S1008. Determine the incremental sample set based on the multi-source fine-grained labeled sample set.

[0178] The method for determining the incremental sample set based on the multi-source fine-grained labeled sample set is implemented according to the first implementation method in S901-S903 above. The description of the above embodiment can be referred to here, and will not be repeated here.

[0179] S803. Update the network anomaly detection model based on the incremental sample set.

[0180] The aforementioned network anomaly detection model is used to detect the KPI data of multiple network devices in the test sample set in order to determine the status of the KPI data of these multiple network devices.

[0181] It is understandable that coarse-grained annotation feedback information cannot annotate specific KPI data, but the annotation results corresponding to coarse-grained annotation feedback information have higher accuracy and can reflect the apparent network state. Fine-grained annotation feedback information can annotate specific KPI data, but cannot reflect the apparent network state and may contain a certain degree of mislabeling. In the embodiments of this application, coarse-grained annotation feedback information and fine-grained annotation feedback information can be fused (or combined) to obtain an incremental sample set with higher annotation accuracy. Thus, training on this incremental sample set can yield a network anomaly detection model with higher accuracy (i.e., a better model or a high-quality model).

[0182] Optional, such as Figure 11 As shown, the method for updating the network anomaly detection model based on the incremental sample set specifically includes S8031-S8032.

[0183] S8031. Extract features from the incremental sample set to obtain the feature set of the incremental sample set.

[0184] In this embodiment of the application, the feature set of the above-mentioned incremental sample set may include at least one of the following features: mean, variance, extreme values, entropy, difference value, and change ratio of KPI data.

[0185] One way to extract features from an incremental sample set is to extract features from individual data points (i.e., individual KPIs) in the incremental sample set, such as the mean, variance, extreme values, and entropy of the KPI data mentioned above.

[0186] Optionally, the extraction of single-point features from the incremental sample set may also include: the original value of the single data point in the incremental sample set, the quantile of the single data point, etc.

[0187] One way to extract features from an incremental sample set is to extract features from the KPI sequence data in the incremental sample set, such as the difference values ​​(e.g., first-order differences) and the percentage change of the KPI data mentioned above. Features of the KPI sequence data can also include the degree of anomaly in the KPI data.

[0188] Furthermore, optionally, based on the above-mentioned extraction of features from single data points in the incremental sample set / or extraction of features from KPI sequence data in the incremental sample set, features can be added to the feature set in conjunction with business logic. For example, multiple features of KPI data within a window (i.e., a time period, such as holidays, workdays, etc.) can be combined. For example, after performing calculations on multiple features, new features are obtained and added to the feature set. For instance, the difference between the mean and extreme values ​​of KPI data in a window can be calculated, and the difference between the mean and extreme values ​​can be used as a new feature. Alternatively, the features of KPI data from multiple windows can be compared, and the ratios, differences, etc., of the features of KPI data from multiple windows can be calculated to obtain new features, which can be added to the feature set. For instance, the ratio between the mean of KPI data in window 1 and the mean of KPI data in window 2 can be calculated, and the ratio of the means can be used as a new feature. Of course, the feature set can also be obtained in other ways, and this application embodiment does not limit this.

[0189] S8032. Based on the feature set of the incremental sample set, the incremental training algorithm is used to train the network anomaly detection model to obtain the updated network anomaly detection model.

[0190] The incremental training algorithm includes any of the following: incremental tree algorithm, support vector machine algorithm, and convolutional neural network algorithm.

[0191] In this embodiment of the application, the process of training a network anomaly detection model is briefly introduced using the incremental tree algorithm as an example.

[0192] In this embodiment, the process of training a network anomaly detection model is briefly introduced using the incremental tree (hoeffding) algorithm as an example. The incremental tree algorithm is a decision tree algorithm that incrementally generates a decision tree from the data stream. The incremental tree algorithm can meet the processing requirements of potentially unlimited, fast, and single-scan processing of high-speed data streams while ensuring high efficiency.

[0193] Understandably, the basic principle of the decision tree algorithm is: based on the features (or attribute values) of the samples in the sample set, the samples in the sample set are continuously divided, similar to the growth process of a tree. The tree splits from the root node into different child nodes. Then, at the child node, the child node is further split according to the features of the samples belonging to that child node, so that the child node splits into multiple child nodes, and so on, until the child node can no longer be split, thus forming a decision tree.

[0194] In this embodiment, when training the network anomaly detection model using the incremental tree algorithm, the samples in the incremental sample set are first randomly arranged. Then, the splitting features of each node are calculated, and a decision tree is generated based on the splitting features. Specifically, for the root node, when the number of accumulated samples (from the incremental sample set) at the root node reaches the minimum lower limit number of the root node (this minimum lower limit data is the condition for the root node to start splitting), the splitting features of the root node are calculated. It should be understood that the splitting features of the root node are calculated based on the features of the KPI data belonging to the root node. Specifically, the Gini coefficient of each feature in the above feature set is calculated, and the feature corresponding to the smallest Gini coefficient is taken as the splitting feature of the root node. Then, based on the splitting features of the root node, the root node is classified into multiple child nodes, for example, split into two child nodes, i.e., the root node is split into two child nodes. Based on the partitioning features of the root node, the accumulated samples are divided into two classes. Furthermore, for each child node, when the accumulated samples on the child node reach the minimum lower limit number of the child node, the partitioning features of each child node are calculated based on the samples belonging to each child node. The calculation method of the partitioning features of the child node is similar to that of the partitioning features of the root node. Based on the partitioning features of the child node, the child node is split into multiple child nodes. This splitting process is repeated until the number of samples in each node is less than the minimum lower limit number of the node or the Gini coefficient of the node is greater than a preset value, thus generating an incremental decision tree.

[0195] In the above method of training a network anomaly detection model using the incremental tree algorithm, after eliminating the problem caused by sample imbalance, the relevant parameters in the incremental tree algorithm can be adjusted to optimize the network anomaly detection model. For example, one or more of the following can be changed: the feature splitting point, the height of the incremental tree (i.e., the maximum depth of the incremental tree), the minimum number of samples, or the threshold of the Gini coefficient.

[0196] In one implementation, the number of training iterations can be set, and multiple incremental training algorithms can be run to obtain multiple network anomaly detection models. Optionally, the feature set of the incremental sample set can be randomly sampled to form multiple training sample sets. The incremental training algorithm can then be used to train these multiple training sample sets to obtain multiple network anomaly detection models corresponding to the multiple training sample sets. Evaluation metrics (e.g., precision and / or recall) for each of the multiple network anomaly detection models can be calculated. The network anomaly detection model with the highest precision and / or recall is selected as the network anomaly detection model trained by the algorithm (i.e., model C mentioned above). Optionally, the evaluation metrics for the network anomaly detection model can also be accuracy, a comprehensive metric (e.g., F-score, which can comprehensively reflect the aforementioned precision and recall), or other evaluation metrics. This application embodiment does not limit the specific evaluation metrics.

[0197] The method described above, which adjusts the number of training iterations, can significantly shorten the model training time. Furthermore, in scenarios involving online training of network anomaly detection models, this can improve the detection speed of network anomaly detection methods.

[0198] In one implementation, a target precision and / or recall rate can be set. After each run of the incremental training algorithm to obtain the network anomaly detection model, the precision and / or recall rate of the network anomaly detection model are calculated. When the precision and / or recall rate of the model reaches the target precision and / or recall rate, the network anomaly detection model obtained by the current training is used as the network anomaly detection model trained by the algorithm (i.e., model C mentioned above).

[0199] It should be understood that the above S8031 is... Figure 7 The feature extraction module 102 in S8032 is implemented by... Figure 7 The model training module 103 in the middle implements the above feature extraction process by Figure 7 The preprocessing module 101 in the above-mentioned network anomaly detection model trained by the algorithm selected from multiple network anomaly detection models is implemented by... Figure 7 The model selection module 105 is implemented in the model selection module.

[0200] Optionally, the model selection module 105 compares the network anomaly detection model (i.e., model C) trained by the algorithm with the network anomaly detection model used last time. If model C is better than the anomaly detection model used last time, the model selection module 105 sends model C to the anomaly detection module 303 on the device side. In the subsequent anomaly detection process, anomaly detection is performed based on model C.

[0201] Combination Figure 8 ,like Figure 12 As shown, after obtaining the updated network anomaly detection model, the network anomaly detection method provided in this application embodiment further includes S804.

[0202] S804. Detect the KPI data of multiple network devices based on the updated network anomaly detection model, and determine the status of the KPI data of multiple network devices.

[0203] By inputting the KPI data of the aforementioned network devices into the updated network anomaly detection model, the status of the network device's KPI data can be obtained.

[0204] Optionally, in this embodiment of the application, the KPI data of multiple network devices can also be detected based on the updated network anomaly detection model and unsupervised algorithm to determine the status of the KPI data of multiple network devices.

[0205] For example, the KPI data of multiple network devices can be detected first using an unsupervised algorithm to obtain the results of the unsupervised algorithm. Then, the abnormal KPI data in the results of the unsupervised algorithm can be detected using a network anomaly detection model to obtain the status of the KPI data of multiple network devices.

[0206] In this embodiment, since unsupervised algorithms are statistical, highly generalizable, and have relaxed conditions, their accuracy in detecting KPI data is not high. Therefore, using a network anomaly detection model to further detect abnormal KPI data in the unsupervised algorithm results can ensure detection accuracy and reduce false positives. Furthermore, because unsupervised algorithms run relatively quickly, combining the network anomaly detection model with unsupervised algorithms to determine the status of network device KPI data can improve the efficiency of network anomaly detection.

[0207] Optionally, the above unsupervised algorithm can be an N sigma algorithm, a box plot algorithm, or an algorithm based on the jump magnitude of historical data, etc., and the specific embodiments of this application are not limited.

[0208] For example, for equally spaced KPI sequence data, the N sigma method can be used to determine whether the KPI data is in an abnormal or normal state.

[0209] Alternatively, in the box plot algorithm, the upper and lower limits of the boxes are calculated based on the KPI sequence data, and the length of the boxes is calculated based on the upper and lower limits. Then, based on the upper limit and length of the boxes, or based on the lower limit and length of the boxes, it is determined whether the KPI data is in an abnormal or normal state. In one approach, it is determined whether the KPI data value is greater than a first threshold, which is equal to the lower limit of the box plus a predetermined coefficient multiplied by the length of the box. If the KPI data value is greater than the first threshold, it is determined to be in an abnormal state; otherwise, the KPI data is in a normal state. In another approach, it is determined whether the KPI data value is less than a second threshold, which is equal to the upper limit of the box minus a predetermined coefficient multiplied by the length of the box. If the KPI data is less than the second threshold, it is determined to be in an abnormal state; otherwise, the KPI data is in a normal state.

[0210] Alternatively, based on a given coefficient α and the historical baseline information N of the data, when the value of the KPI data exceeds α*N, the KPI data is considered to be in an abnormal state; conversely, when the value is less than α*N, the KPI data is considered to be in a normal state.

[0211] In summary, this application provides a network anomaly detection method that can determine an incremental sample set based on multiple acquired labeled sample sets, and then update the network anomaly detection model based on the incremental sample set. Since this application's embodiment updates the network anomaly detection model based on the incremental sample set, the updated network anomaly detection model can continuously adapt to changes in the network device's KPI data. Therefore, the updated network anomaly detection model has higher applicability, and the accuracy of network anomaly detection using this network anomaly detection model is higher.

[0212] Combination Figure 8 or Figure 12 ,like Figure 13 Before updating the network anomaly detection model based on the incremental sample set, the network anomaly detection method provided in this application embodiment also includes S805.

[0213] S805. Perform sample balancing on the incremental sample set to make the ratio of positive samples to negative samples in the incremental sample set reach a predetermined ratio.

[0214] The negative samples mentioned above are samples where the KPI data is in an abnormal state, and the positive samples mentioned above are samples where the KPI data is in a normal state.

[0215] The method described above for achieving a predetermined ratio of positive to negative samples in the incremental sample set can be: changing the number of positive samples and / or the number of negative samples in the incremental sample set.

[0216] In this embodiment, since network devices typically fail infrequently, the number of negative samples is small. Consequently, the ratio of positive to negative samples in the incremental sample set differs significantly, resulting in a low-quality network anomaly detection model trained with such an incremental sample set, exhibiting a high false detection rate. In this situation, a sample balancing method can be used to increase the proportion of negative samples in the incremental sample set.

[0217] Optionally, methods to increase the proportion of negative samples in the incremental sample set may include increasing the number of negative samples and / or decreasing the number of positive samples, depending on the actual situation, and are not limited in this application embodiment. For example, in this application embodiment, a variant of the SMOTE algorithm may be used to increase the number of negative samples in the incremental sample set.

[0218] By performing sample balancing on the incremental sample set, the incremental sample set can be made more balanced in scenarios where the number of negative samples in the incremental sample set is small. As a result, the accuracy of network anomaly detection based on the trained network anomaly detection model is higher.

[0219] Optionally, in the embodiments of this application, during the process of training the network anomaly detection model multiple times, the proportion of negative samples in the incremental sample set used to train the network anomaly detection model can be adjusted, and the proportion of negative samples can be continuously increased until the proportion of negative samples reaches a preset value. That is, one or more network anomaly detection models are trained on the incremental sample set with the increased proportion of negative samples.

[0220] Combination Figure 8 or Figure 10 ,like Figure 14 As shown, after updating the network anomaly detection model based on the incremental sample set, the network anomaly detection method provided in this application embodiment further includes S806.

[0221] S806. Adjust the sampling frequency of the third sample set based on the user's evaluation information of the updated network anomaly detection model.

[0222] In this embodiment, the network anomaly detection model can be evaluated manually to obtain evaluation information, which can be user scores of the anomaly detection model. The aforementioned evaluation of the network anomaly detection model can be performed... Figure 7 The manual visualization annotation module 107 shown is implemented.

[0223] Optionally, the user can score the network anomaly detection model based on at least one of the precision or recall. For example, the score can be in the range of [0, 10], and this application does not limit the specific implementation.

[0224] In this embodiment, if the score of the network anomaly detection model is low, it indicates that there is a problem in the training process of the network anomaly detection model. One possible reason is that the accuracy of the training sample set (i.e., the incremental sample set) is low. Therefore, when the score of the network anomaly detection model is lower than the predetermined score, the sampling frequency of the third sample set is adjusted to update the second sample set, which in turn updates the incremental sample set, thereby improving the accuracy of the incremental sample set. Furthermore, the network anomaly detection model is optimized based on the updated incremental sample set.

[0225] In one scenario, if the third sample set corresponds to a single-source fine-grained labeled sample set, and this single-source fine-grained labeled sample set is a manually labeled sample set, then if the score of the network anomaly detection model is lower than a predetermined score, the sampling frequency of the third sample set will be increased.

[0226] In another scenario, where the third sample set corresponds to a single-source fine-grained labeled sample set, and this single-source fine-grained labeled sample set is also a model labeled sample set, if the score of the network anomaly detection model is lower than a predetermined score, the sampling frequency of the third sample set is reduced. Optionally, in this case, the user can be visually prompted to introduce manual annotation feedback, i.e., to increase manual annotation, thereby forming a manually labeled sample set.

[0227] In another case, the third sample set corresponds to a multi-source fine-grained labeled sample set (e.g., a manually labeled sample set and a model labeled sample set). If the score of the anomaly detection model of the network is lower than a predetermined score, the sampling frequency of the third sample set corresponding to the model labeled sample set is reduced; or the sampling frequency of the third sample set corresponding to the model labeled sample set is reduced, and the sampling frequency of the third sample set corresponding to the manually labeled sample set is increased.

[0228] In such Figure 7 In the detection framework shown, in the manual visualization annotation module 107, the human evaluates and scores the network anomaly detection model (model C) based on its precision and / or recall. When the evaluation score of model C is less than a predetermined score, the score is sent to the MT side. On the MT side, the incremental sample set is adjusted according to the score. Specifically, when there are multiple sources of fine-grained annotation feedback information corresponding to the annotation sample set (e.g., the human-annotated sample set and the model-annotated sample set) on the MT side, the sampling frequency W1 of the third sample set corresponding to the model-annotated sample set is reduced, thereby increasing the proportion of the human-annotated sample set in the incremental sample set. When there is a single source of fine-grained annotation feedback information corresponding to the annotation sample set on the MT side, such as the human-annotated sample set or the model-annotated sample set, the sampling frequency W1 of the third sample set corresponding to the model-annotated sample set or the sampling frequency W2 of the human-annotated sample set is reduced according to the above-mentioned score feedback.

[0229] Accordingly, this application provides a network anomaly detection device, which is used to execute the steps of the above-described network anomaly detection method. This application can divide the network anomaly detection device into functional modules based on the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. The module division in this application is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0230] When dividing each function into modules according to its corresponding function. Figure 15This diagram illustrates a possible structure of the network anomaly detection device involved in the above embodiments. For example... Figure 15 As shown, the network anomaly detection device includes an acquisition module 1501, a determination module 1502, and an update module 1503.

[0231] The acquisition module 1501 is used to acquire at least one set of labeled samples, for example, by executing S801 in the above method embodiment.

[0232] The determination module 1502 is used to determine an incremental sample set based on at least one labeled sample set, for example, by performing S802 in the above method embodiment.

[0233] The update module 1503 is used to update the network anomaly detection model based on the incremental sample set, for example, by executing S803 in the above method embodiment.

[0234] Optionally, the determination module 1502 is specifically used to determine the first sample set, for example, by executing S901 in the above method embodiment.

[0235] The determining module 1502 is also specifically used to determine Y second sample sets, for example, by performing S902 in the above method embodiment.

[0236] The determining module 1502 is also specifically used to determine the union of the first sample set and the Y second sample sets as the incremental sample set, for example, by executing S903 in the above method embodiment.

[0237] Optionally, the network anomaly detection device provided in this application embodiment further includes a feature extraction module 1504; the feature extraction module 1504 is used to extract features from the incremental sample set to obtain a feature set of the incremental sample set, for example, by executing S8031 in the above method embodiment.

[0238] The aforementioned update module 1503 is specifically used to train the network anomaly detection model based on the feature set of the incremental sample set and using an incremental training algorithm to obtain the updated network anomaly detection model, for example, by executing S8032 in the above method embodiment.

[0239] Optionally, the network anomaly detection device provided in this application embodiment further includes an equalization processing module 1505; the equalization processing module 1505 is used to perform sample equalization processing on the incremental sample set; for example, to execute S805 in the above method embodiment.

[0240] Optionally, the network anomaly detection device provided in this application embodiment further includes an adjustment module 1506, which is used to adjust the sampling frequency of the third sample set according to the user's evaluation information on the updated network anomaly detection model; for example, by executing S806 in the above method embodiment.

[0241] Optionally, the network anomaly detection device provided in this application embodiment further includes a detection module 1507, which is used to detect the KPI data of multiple network devices based on the updated network anomaly detection model and unsupervised algorithm, and determine the status of the KPI data of multiple network devices; for example, executing S804 in the above method embodiment.

[0242] Each module of the above-mentioned network anomaly detection device can also be used to perform other actions in the above-mentioned method embodiments. All relevant content of each step involved in the above-mentioned method embodiments can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0243] When using integrated units, the structural schematic diagram of the network anomaly detection device provided in this application embodiment is as follows: Figure 16 As shown. In Figure 16 The network anomaly detection device includes a processing module 1601 and a communication module 1602. The processing module 1601 controls and manages the operations of the network anomaly detection device, for example, executing the steps performed by the acquisition module 1501, determination module 1502, update module 1503, feature extraction module 1504, equalization processing module 1505, adjustment module 1506, and detection module 1507, and / or performing other processes described herein. The communication module 1602 supports interaction between the network anomaly detection device and other devices. Figure 16 As shown, the network anomaly detection device may also include a storage module 1603, which is used to store the program code, labeled sample set, and incremental sample set of the network anomaly detection device.

[0244] The processing module 1601 can be a processor or a controller, for example... Figure 5 The processor 501 is located in the [processor name]. The communication module 1602 can be a transceiver, RF circuit, or communication interface, for example... Figure 5 The network interface 503 is included. The storage module 1603 can be a memory, such as... Figure 5 The internal memory 502 in it.

[0245] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center integrating one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0246] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0247] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0248] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0249] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0250] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0251] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for detecting network anomalies, characterized in that, include: Obtain at least one set of labeled samples, wherein the set of labeled samples is a set of samples after labeling the test sample set based on the labeling feedback information of the test sample set; The annotation feedback information includes: N types of annotation feedback information, which include X types of coarse-grained annotation feedback information and Y types of fine-grained annotation feedback information; N is a positive integer greater than or equal to 2, X + Y = N, X is a positive integer greater than or equal to 1, and Y is an integer greater than or equal to 1; each type of annotation feedback information corresponds to a set of annotation samples. The coarse-grained annotation feedback information is used to indicate whether the multiple network devices corresponding to the test sample set are in a normal or abnormal state, and the fine-grained annotation feedback information is used to indicate whether the KPI data of the multiple network devices in the test sample set are in a normal or abnormal state. Determine a first sample set, which is the intersection of N labeled sample sets; Y second sample sets are determined, wherein the i-th second sample set among the Y second sample sets is determined according to the sampling frequency w. i The i-th third sample set is obtained by sampling the i-th third sample set; wherein, the i-th third sample set is the difference between the i-th labeled sample set in the Y labeled sample sets corresponding to the Y types of fine-grained labeled feedback information and the first sample set; The union of the first sample set and the Y second sample sets is determined as the incremental sample set; based on the incremental sample set, the network anomaly detection model is updated; wherein, the network anomaly detection model is used to detect the KPI data of multiple network devices in the test sample set to determine the status of the KPI data of the multiple network devices.

2. The method according to claim 1, characterized in that, The coarse-grained annotation feedback information is obtained by annotating the network device based on at least one of the network device's latency, packet loss rate, or effective time. The fine-grained feedback information is obtained by annotating the KPI data of the network devices based on manual and / or network anomaly detection models.

3. The method according to claim 1 or 2, characterized in that, Based on the incremental sample set, the network anomaly detection model is updated, including: Feature extraction is performed on the incremental sample set to obtain the feature set of the incremental sample set. The feature set of the incremental sample set includes at least one of the following features: mean, variance, extreme value, entropy, difference value, and change ratio of KPI data. Based on the feature set of the incremental sample set, an incremental training algorithm is used to train the network anomaly detection model to obtain an updated network anomaly detection model; wherein, the incremental training algorithm includes any of the following: incremental tree algorithm, support vector machine algorithm, and convolutional neural network algorithm.

4. The method according to claim 3, characterized in that, Before updating the network anomaly detection model based on the incremental sample set, the method further includes: The incremental sample set is subjected to sample balancing processing to make the ratio of positive samples to negative samples in the incremental sample set reach a predetermined ratio; the negative samples are samples whose KPI data is in an abnormal state, and the positive samples are samples whose KPI data is in a normal state.

5. The method according to claim 3 or 4, characterized in that, The method further includes: The sampling frequency of the third sample set is adjusted based on user evaluations of the updated network anomaly detection model.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Based on the updated network anomaly detection model and unsupervised algorithm, the KPI data of multiple network devices are detected to determine the status of the KPI data of the multiple network devices.

7. A network anomaly detection device, characterized in that, The network anomaly detection device includes an acquisition module, a determination module, and an update module; The acquisition module is used to acquire at least one set of labeled samples, which is a set of samples after labeling the test sample set based on the labeling feedback information of the test sample set; The annotation feedback information includes: N types of annotation feedback information, which include X types of coarse-grained annotation feedback information and Y types of fine-grained annotation feedback information; N is a positive integer greater than or equal to 2, X + Y = N, where X is a positive integer greater than or equal to 1, and Y is an integer greater than or equal to 1; each type of annotation feedback information corresponds to a set of annotated samples. The coarse-grained annotation feedback information is used to indicate whether the multiple network devices corresponding to the test sample set are in a normal or abnormal state, and the fine-grained annotation feedback information is used to indicate whether the KPI data of the multiple network devices in the test sample set are in a normal or abnormal state. The determining module is specifically used for: Determine a first sample set, which is the intersection of N labeled sample sets; Y second sample sets are determined, wherein the i-th second sample set among the Y second sample sets is determined according to the sampling frequency w. i The i-th third sample set is obtained by sampling the i-th third sample set; wherein, the i-th third sample set is the difference between the i-th labeled sample set in the Y labeled sample sets corresponding to the Y types of fine-grained labeled feedback information and the first sample set; The union of the first sample set and the Y second sample sets is determined as the incremental sample set; The update module is used to update the network anomaly detection model based on the incremental sample set; wherein, the network anomaly detection model is used to detect the KPI data of multiple network devices in the test sample set to determine the status of the KPI data of the multiple network devices.

8. The network anomaly detection device according to claim 7, characterized in that, The coarse-grained annotation feedback information is obtained by annotating the network device based on at least one of the network device's latency, packet loss rate, or effective time. The fine-grained feedback information is obtained by annotating the KPI data of the network devices based on manual and / or network anomaly detection models.

9. The network anomaly detection device according to claim 7 or 8, characterized in that, The network anomaly detection device also includes a feature extraction module; The feature extraction module is used to extract features from the incremental sample set to obtain a feature set of the incremental sample set. The feature set of the incremental sample set includes at least one of the following features: mean, variance, extreme values, entropy, difference value, and change ratio of KPI data. The update module is specifically used to train the network anomaly detection model based on the feature set of the incremental sample set using an incremental training algorithm to obtain an updated network anomaly detection model; wherein, the incremental training algorithm includes any of the following: incremental tree algorithm, support vector machine algorithm, and convolutional neural network algorithm.

10. The network anomaly detection device according to claim 9, characterized in that, The network anomaly detection device also includes a load balancing module; The equalization processing module is used to perform sample equalization processing on the incremental sample set so that the ratio of positive samples to negative samples in the incremental sample set reaches a predetermined ratio. The negative samples are those where the KPI data is in an abnormal state, and the positive samples are those where the KPI data is in a normal state.

11. The network anomaly detection device according to claim 9 or 10, characterized in that, The network anomaly detection device also includes an adjustment module: The adjustment module is used to adjust the sampling frequency of the third sample set based on the user's evaluation information of the updated network anomaly detection model.

12. The network anomaly detection device according to any one of claims 7 to 11, characterized in that, The network anomaly detection device also includes a detection module; The detection module is used to detect the KPI data of multiple network devices based on the updated network anomaly detection model and unsupervised algorithm, and to determine the status of the KPI data of the multiple network devices.

13. A network anomaly detection device, characterized in that, The device includes a memory and a processor, the memory being coupled to the processor; the memory is used to store computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the network anomaly detection device causes the device to perform the method as described in any one of claims 1 to 6.

14. A computer storage medium, characterized in that, Includes computer instructions that, when executed on a network anomaly detection device, cause the network anomaly detection device to perform the method as described in any one of claims 1 to 6.

15. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 6.

Images