Anomaly detection method, apparatus and electronic device

Abstract

The application discloses an abnormality detection method, device and electronic equipment. The method comprises the following steps: obtaining a normal flow set and an abnormal flow set based on a pre-training model; processing the normal flow set and the abnormal flow set by selecting a noise reduction mode, obtaining the most normal flow and the most abnormal flow; obtaining a target model based on the most normal flow and the most abnormal flow, and performing abnormality detection on the flow by using the target model. The technical scheme provided by the embodiment of the application solves the problem that the boundary between the normal flow and the abnormal flow is fuzzy, and reduces the false positive rate of abnormality detection.

Description

Technical Field

[0001] This application relates to the field of network and information security technology, and in particular to an anomaly detection method, device and electronic equipment. Background Technology

[0002] Anomaly detection refers to the discovery of patterns in data that do not conform to expected behavior. Therefore, a simple anomaly detection method is to define a region representing normal behavior and declare any observations in the data that do not belong to this normal region as anomalies.

[0003] However, in traffic anomaly detection, the methods for disguising abnormal traffic attacks as normal traffic access (that is, disguising abnormal traffic as normal traffic, and abnormal behavior as normal behavior) have been constantly improving, making it difficult to define truly normal traffic. This has led to a blurring of the line between normal and abnormal traffic, resulting in a high false alarm rate in traffic anomaly detection. Summary of the Invention

[0004] This application provides an anomaly detection method, apparatus, and electronic device to address the problem of high false alarm rates in anomaly detection caused by the blurred boundary between normal and abnormal traffic. The specific implementation scheme is as follows:

[0005] Firstly, this application provides an anomaly detection method, the method comprising:

[0006] Based on the pre-trained model, obtain the normal traffic set and the abnormal traffic set;

[0007] By using an election-based noise reduction method, the normal traffic set and the abnormal traffic set are processed separately to obtain the most normal traffic and the most abnormal traffic; wherein, the most normal traffic is the most normal normal traffic in the normal traffic set; and the most abnormal traffic is the most abnormal abnormal traffic in the abnormal traffic set.

[0008] Based on the most normal traffic and the most abnormal traffic, a target model is obtained, and anomaly detection is performed on the traffic using the target model.

[0009] Through the above-described embodiments, a normal traffic set and an abnormal traffic set are obtained based on a pre-trained model, thus achieving an initial division between normal and abnormal traffic. Then, an election-based noise reduction method is used to process the normal and abnormal traffic sets, obtaining the most normal traffic in the normal traffic set and the most abnormal traffic in the abnormal traffic set, thereby achieving a further division between normal and abnormal traffic and clarifying the boundary between them, thus solving the problem of blurred boundaries between normal and abnormal traffic. Finally, a target model is obtained based on the clearly defined most normal and most abnormal traffic, and anomaly detection is performed using the target model, reducing the false alarm rate of anomaly detection.

[0010] In one possible implementation, before obtaining the normal traffic set and the abnormal traffic set based on the pre-trained model, the method further includes:

[0011] The training traffic is obtained, and the normal traffic is preprocessed to obtain input data; wherein, the training traffic is normal traffic.

[0012] The input data is fed into the encoder in the deep multi-fusion autoencoder model to obtain high-dimensional features;

[0013] The high-dimensional features are processed by the classification model in the deep multi-fusion autoencoder model to obtain a classification result; and a loss is calculated based on the classification result and the input data to obtain a classification loss value; and

[0014] The high-dimensional features are processed by the reconstruction model in the deep multi-fusion autoencoder model to obtain the reconstruction result; and the reconstruction loss value is obtained by calculating the loss based on the reconstruction result and the input data.

[0015] The classification loss value and the reconstruction loss value are combined to obtain the combined loss value;

[0016] Based on the merged loss value, the deep multi-fusion autoencoder model is updated to obtain the pre-trained model.

[0017] Through the above-described embodiments, the deep multi-fusion autoencoder model is jointly updated based on the classification model and the reconstruction model in the deep multi-fusion autoencoder model. This ensures that the pre-trained model takes into account both the classification model and the reconstruction model, which is beneficial to improving the performance of the pre-trained model. This, in turn, helps to improve the accuracy of the normal traffic set and abnormal traffic set obtained based on the pre-trained model. Furthermore, the pre-trained model can identify the distribution characteristics of the given normal traffic and preliminarily determine the normal traffic based on these characteristics.

[0018] In one possible implementation, the step of processing the high-dimensional features through the reconstruction model in the deep multi-fusion autoencoder model to obtain the reconstruction result includes:

[0019] The high-dimensional features are sparsely processed by the intercept function in the reconstruction model of the deep multi-fusion autoencoder model to obtain the first feature;

[0020] The first feature is processed by the transformation matrix in the reconstruction model to obtain the second feature;

[0021] The second feature is decoded and reconstructed using the decoder in the reconstruction model to obtain the reconstruction result.

[0022] In the above-described embodiments, the high-dimensional features are first sparsified using the intercept function in the reconstruction model to filter out redundant information features and avoid their harmful effects. Then, the first feature, after sparsification, is processed using the transformation matrix in the reconstruction model to transform its dimension into a second feature conforming to a preset dimension, incorporating key information to ensure the second feature meets requirements. Finally, the second feature is decoded and reconstructed using the decoder in the reconstruction model, resulting in a more accurate reconstruction result and improving the accuracy of the reconstruction loss value obtained based on the reconstruction result.

[0023] In one possible implementation, obtaining the normal traffic set and the abnormal traffic set based on the pre-trained model includes:

[0024] The traffic to be used is input into the pre-trained model to obtain a first result and a second result; wherein, the first result is the abnormal result output by the classification model in the pre-trained model, and the second result is the abnormal result output by the reconstruction model in the pre-trained model;

[0025] A third result is obtained by performing a binary OR operation on the first result and the second result using a fusion judgment method.

[0026] If the third result indicates that the traffic to be used is normal traffic, then the traffic to be used is added to the set of normal traffic.

[0027] If the third result indicates that the traffic to be used is abnormal traffic, then the traffic to be used is added to the abnormal traffic set.

[0028] Through the above application embodiments, the traffic to be used is input into the pre-trained model, and the abnormal results output by the classification model in the pre-trained model (i.e., the first result) and the abnormal results output by the reconstruction model in the pre-trained model (i.e., the second result) are obtained. Then, the first result and the second result are XORed by the fusion judgment method, thereby correcting the abnormal results output by the classification model (i.e., the first result) and the abnormal results output by the reconstruction model (i.e., the second result), obtaining the preliminary boundary between normal traffic and abnormal traffic, and making the credibility of the obtained third result higher, thereby further improving the credibility of the normal traffic set and abnormal traffic set obtained based on the third result.

[0029] In one possible implementation, the step of performing an XOR judgment on the first result and the second result using a fusion judgment method to obtain a third result includes:

[0030] If the first result indicates that the traffic to be used is normal traffic, and the second result indicates that the traffic to be used is normal traffic, then the third result indicating that the traffic to be used is normal traffic is obtained;

[0031] If the first result indicates that the traffic to be used is abnormal traffic, and the second result indicates that the traffic to be used is abnormal traffic, then the third result indicating that the traffic to be used is abnormal traffic is obtained.

[0032] Through the above application embodiments, the third result is determined to be normal traffic only when both the first result and the second result indicate that the traffic to be used is normal traffic, and the third result is determined to be abnormal traffic only when both the first result and the second result indicate that the traffic to be used is abnormal traffic, thereby further improving the credibility of the third result.

[0033] In one possible implementation, the step of processing the normal traffic set and the abnormal traffic set separately through election noise reduction to obtain the most normal traffic and the most abnormal traffic includes:

[0034] At each first interval, the normal traffic set is processed using the election noise reduction method to obtain the most normal traffic, and the normal traffic set is then cleared; and

[0035] Every second time interval, the abnormal traffic set is processed using the election noise reduction method to obtain the most abnormal traffic, and the abnormal traffic set is then cleared.

[0036] Through the above-described embodiments, the most normal traffic is selected from the normal traffic set at a first time interval, and the most abnormal traffic is selected from the abnormal traffic set at a second time interval, ensuring that the boundary between normal and abnormal traffic is clear. Furthermore, by selecting the most normal and most abnormal traffic periodically (i.e., every first time interval and every second time interval), a certain amount of traffic is included in both the normal and abnormal traffic sets when selecting the most normal and most abnormal traffic. Additionally, by periodically selecting the most normal and most abnormal traffic (i.e., every first time interval and every second time interval), and clearing the normal traffic set after selecting the most normal traffic and the abnormal traffic set after selecting the most abnormal traffic, a new round of normal and abnormal traffic sets can be continuously obtained. This allows for a continuous and automatic increase in positive and negative traffic samples (i.e., normal and abnormal traffic), providing abnormal traffic data capabilities and solving the problem of manual sample acquisition.

[0037] In one possible implementation, the step of processing the normal traffic set and the abnormal traffic set separately through election noise reduction to obtain the most normal traffic and the most abnormal traffic includes:

[0038] Obtain the first result and the second result corresponding to each normal traffic in the normal traffic set, and obtain the first result and the second result corresponding to each abnormal traffic in the abnormal traffic set.

[0039] By jointly calculating the first result and the corresponding second result, the joint value corresponding to the flow rate is obtained;

[0040] The joint values ​​corresponding to the normal traffic in the normal traffic set are sorted in ascending order, and the normal traffic corresponding to the first joint value is selected as the most normal traffic. The joint values ​​corresponding to the abnormal traffic in the abnormal traffic set are sorted in descending order, and the abnormal traffic corresponding to the first joint value is selected as the most abnormal traffic.

[0041] Through the above-described embodiments, a joint calculation was performed on the first and second results corresponding to each normal traffic flow in the normal traffic set. The resulting joint values ​​were then sorted in ascending order, and the normal traffic flow corresponding to the first joint value was selected as the most normal traffic flow, maximizing the confidence that the selected most normal traffic flow is normal. Similarly, a joint calculation was performed on the first and second results corresponding to each abnormal traffic flow in the abnormal traffic set. The resulting joint values ​​were then sorted in descending order, and the abnormal traffic flow corresponding to the first joint value was selected as the most abnormal traffic flow, maximizing the confidence that the selected most abnormal traffic flow is abnormal. Thus, by selecting the most normal and most abnormal traffic flows with the highest confidence, the problem of the blurred boundary between normal and abnormal traffic is resolved.

[0042] In one possible implementation, the joint calculation of the first result and the corresponding second result to obtain the joint value corresponding to the traffic flow includes:

[0043] Normal traffic and abnormal traffic that meet the ignore conditions in the normal traffic set and the abnormal traffic set are all ignored traffic; wherein, the ignore conditions are either a first result between a first threshold and a second threshold or a second result between a third threshold and a fourth threshold;

[0044] For the traffic in the normal traffic set and the abnormal traffic set excluding the ignored traffic, perform a joint calculation of the corresponding first result and second result to obtain the joint value corresponding to the traffic.

[0045] Through the above application embodiments, the ignored traffic that meets the ignoring conditions is first determined in the normal traffic set and the abnormal traffic set, so that the ignored traffic can be ignored in the subsequent joint calculation, thereby performing a preliminary screening of the normal traffic and abnormal traffic in the normal traffic set and the abnormal traffic set. Then, the first result and the second result are jointly calculated for the traffic in the normal traffic set and the abnormal traffic set other than the ignored traffic, so that the reliability of the selected most normal traffic and most abnormal traffic is further improved.

[0046] In one possible implementation, obtaining the target model based on the most normal traffic and the most abnormal traffic includes:

[0047] Every third time interval, the most normal traffic and the most abnormal traffic are fed back to the pre-trained model for enhanced training to obtain the target model.

[0048] Through the above application embodiments, the most normal traffic and the most abnormal traffic are fed back to the pre-trained model for enhanced training every third time period, thereby regularly updating the model and continuously improving the model, making the model results more reliable, maintaining the model's timeliness, and improving the model's accuracy in detecting traffic anomalies.

[0049] Secondly, this application also provides an anomaly detection device, the device comprising:

[0050] The first partitioning module is used to obtain the normal traffic set and the abnormal traffic set based on the pre-trained model;

[0051] The second partitioning module is used to process the normal traffic set and the abnormal traffic set respectively through an election noise reduction method to obtain the most normal traffic and the most abnormal traffic; wherein, the most normal traffic is the most normal normal traffic in the normal traffic set; and the most abnormal traffic is the most abnormal abnormal traffic in the abnormal traffic set.

[0052] The processing module is used to obtain a target model based on the most normal traffic and the most abnormal traffic, and to perform anomaly detection on the traffic using the target model.

[0053] In one possible implementation, the apparatus further includes a pre-training module, configured to acquire the traffic to be trained and preprocess the normal traffic to obtain input data; wherein the traffic to be trained is normal traffic; input the input data into the encoder of a deep multi-fusion autoencoder model to obtain high-dimensional features; process the high-dimensional features through a classification model in the deep multi-fusion autoencoder model to obtain a classification result; calculate a loss based on the classification result and the input data to obtain a classification loss value; process the high-dimensional features through a reconstruction model in the deep multi-fusion autoencoder model to obtain a reconstruction result; calculate a loss based on the reconstruction result and the input data to obtain a reconstruction loss value; merge the classification loss value and the reconstruction loss value to obtain a merged loss value; and update the deep multi-fusion autoencoder model based on the merged loss value to obtain the pre-trained model.

[0054] In one possible implementation, the pre-training module is specifically used to perform sparse processing on the high-dimensional features through the intercept function in the reconstruction model of the deep multi-fusion autoencoder model to obtain a first feature; process the first feature through the transformation matrix in the reconstruction model to obtain a second feature; and decode and reconstruct the second feature through the decoder in the reconstruction model to obtain the reconstruction result.

[0055] In one possible implementation, the first partitioning module is specifically used to input the traffic to be used into the pre-trained model to obtain a first result and a second result; wherein, the first result is an abnormal result output by the classification model in the pre-trained model, and the second result is an abnormal result output by the reconstruction model in the pre-trained model; the first result and the second result are XORed by a fusion judgment method to obtain a third result; if it is determined that the third result indicates that the traffic to be used is normal traffic, then the traffic to be used is added to the normal traffic set; if it is determined that the third result indicates that the traffic to be used is abnormal traffic, then the traffic to be used is added to the abnormal traffic set.

[0056] In one possible implementation, the first segmentation module is specifically configured to obtain the third result indicating that the traffic to be used is normal traffic if the first result indicates that the traffic to be used is normal traffic and the second result indicates that the traffic to be used is normal traffic; and to obtain the third result indicating that the traffic to be used is abnormal traffic if the first result indicates that the traffic to be used is abnormal traffic and the second result indicates that the traffic to be used is abnormal traffic.

[0057] In one possible implementation, the second partitioning module is specifically used to process the normal traffic set at each first time interval using the election noise reduction method to obtain the most normal traffic and clear the normal traffic set; and at each second time interval, process the abnormal traffic set using the election noise reduction method to obtain the most abnormal traffic and clear the abnormal traffic set.

[0058] In one possible implementation, the second partitioning module is specifically configured to obtain a first result and a second result corresponding to each normal flow in the normal flow set, and to obtain a first result and a second result corresponding to each abnormal flow in the abnormal flow set; jointly calculate the first result and the corresponding second result to obtain a joint value corresponding to the corresponding flow; sort the joint values ​​corresponding to the normal flows in the normal flow set in ascending order, select the normal flow corresponding to the first joint value as the most normal flow, and sort the joint values ​​corresponding to the abnormal flows in the abnormal flow set in descending order, select the abnormal flow corresponding to the first joint value as the most abnormal flow.

[0059] In one possible implementation, the second partitioning module is specifically used to treat all normal traffic and abnormal traffic in the normal traffic set and the abnormal traffic set that meet the ignoring conditions as ignored traffic; wherein, the ignoring conditions are that the first result is between the first threshold and the second threshold or the second result is between the third threshold and the fourth threshold; for the traffic in the normal traffic set and the abnormal traffic set other than the ignored traffic, the corresponding first result and the second result are jointly calculated to obtain the joint value corresponding to the corresponding traffic.

[0060] In one possible implementation, the processing module is specifically used to feed back the most normal traffic and the most abnormal traffic to the pre-trained model for enhanced training every third time interval to obtain the target model.

[0061] Thirdly, this application provides an electronic device, comprising:

[0062] Memory, used to store computer programs;

[0063] When the processor executes the computer program stored in the memory, it implements the above-described steps of an anomaly detection method.

[0064] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described steps of an anomaly detection method.

[0065] For the various aspects of the second to fourth aspects mentioned above, and the technical effects that each aspect may achieve, please refer to the above description of the technical effects that can be achieved for the first aspect or the various possible solutions in the first aspect, which will not be repeated here. Attached Figure Description

[0066] Figure 1 A schematic diagram of a deep multi-fusion autoencoder model provided in an embodiment of this application;

[0067] Figure 2 A flowchart illustrating an anomaly detection method provided in an embodiment of this application;

[0068] Figure 3 A schematic diagram of the pre-training process of a deep multi-fusion autoencoder model provided in an embodiment of this application;

[0069] Figure 4 This is a schematic diagram of the normal traffic set and the abnormal traffic set provided in the embodiments of this application;

[0070] Figure 5 A schematic diagram illustrating the processing steps of an anomaly detection method provided in an embodiment of this application;

[0071] Figure 6 This is a schematic diagram of an anomaly detection device provided in an embodiment of this application;

[0072] Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The specific operational methods in the method embodiments can also be applied to the device embodiments or system embodiments. It should be noted that in the description of this application, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A connected to B can represent: A and B directly connected, and A and B connected through C. Furthermore, in the description of this application, terms such as "first" and "second" are used only for distinguishing the purpose of description and should not be construed as indicating or implying relative importance or order.

[0074] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0075] Most existing methods for detecting traffic anomalies employ automatic learning techniques such as machine learning or deep learning to identify abnormal traffic. However, these methods do not address the blurred line between normal and abnormal traffic, and generate a large number of false abnormal and normal traffic during the detection process, resulting in a high false alarm rate for traffic anomaly detection.

[0076] Therefore, this application proposes an anomaly detection method. First, a pre-trained model is used to obtain a set of normal traffic and a set of abnormal traffic, thus achieving an initial division between normal and abnormal traffic. Then, an election-based denoising method is used to process the normal and abnormal traffic sets separately, obtaining the most normal normal traffic (i.e., the most normal traffic) in the normal traffic set and the most abnormal traffic (i.e., the abnormal traffic set) in the abnormal traffic set. This further refines the division between normal and abnormal traffic, resolving the problem of blurred boundaries between the two. Finally, a target model obtained based on the most normal and most abnormal traffic is used to detect anomalies, significantly reducing the false alarm rate of anomaly detection.

[0077] In this embodiment, the target model is obtained by enhancing the training of the pre-trained model; the pre-trained model is obtained by pre-training the deep multi-fusion autoencoder model.

[0078] The above-mentioned deep multi-fusion autoencoder model can be as follows: Figure 1 As shown, this deep multi-fusion autoencoder model includes an encoder, a classification model, a sparse transform, a decoder, a fusion decision module, and an election and denoising module. The sparse transform and decoder can form a reconstruction model.

[0079] The encoder in the aforementioned deep multi-fusion autoencoder model is used to process data (i.e. Figure 1 The input is processed to obtain high-dimensional features (i.e., ...). Figure 1 (z) represents the data after preprocessing by the preprocessing module. The preprocessing methods used in this module include common methods such as data cleaning and normalization, which will not be elaborated upon here.

[0080] The classification model in the aforementioned deep multi-fusion autoencoder model is used to process high-dimensional features to obtain classification results (i.e., Figure 1 (y' in the text). This classification model can be a conventional deep learning classification. The classification result can be represented in the form of probabilities. Based on the classification result, it can be determined whether the corresponding input is abnormal, that is, whether the corresponding input is normal traffic or abnormal traffic.

[0081] For example, traffic A, preprocessed by the preprocessing module, is input into the encoder of a deep multi-fusion autoencoder. The encoder's output (i.e., high-dimensional features) is then input into the classification model within the deep multi-fusion autoencoder. This classification model outputs a classification result (y'), which is the output probability of traffic A. This output probability is then compared to a probability threshold. If the output probability is less than the probability threshold, traffic A is determined to be abnormal traffic. If the output probability is not less than the probability threshold, traffic A is determined to be normal traffic.

[0082] The output probabilities mentioned above are typically between 0 and 1, and the probability threshold can be 0.5, but it is not limited to this. In other words, if the output probability is less than 0.5, then traffic A is abnormal traffic; if the output probability is not less than 0.5, then traffic A is normal traffic.

[0083] The reconstruction model in the aforementioned deep multi-fusion autoencoder model is used to process high-dimensional features to obtain the reconstruction result (i.e. Figure 1 (x' in the text).

[0084] The fusion judgment module in the aforementioned deep multi-fusion autoencoder model is used to perform an XOR judgment on the abnormal results output by the classification model and the abnormal results output by the reconstruction model in order to obtain more accurate results.

[0085] For example, if the abnormal result output by the classification model indicates that the corresponding traffic is normal traffic, and the abnormal result output by the reconstruction model also indicates that the corresponding traffic is normal traffic, then the corresponding traffic is determined to be normal traffic.

[0086] If the abnormal results output by the classification model indicate that the corresponding traffic is abnormal, and the abnormal results output by the reconstruction model also indicate that the corresponding traffic is abnormal, then the corresponding traffic is determined to be abnormal traffic.

[0087] If the abnormal result output by the classification model indicates that the corresponding traffic is abnormal, while the abnormal result output by the reconstruction model indicates that the corresponding traffic is normal; or if the abnormal result output by the classification model indicates that the corresponding traffic is normal, while the abnormal result output by the reconstruction model also indicates that the corresponding traffic is abnormal, then the corresponding traffic is discarded.

[0088] Therefore, the traffic is determined to be normal only when both the abnormal results output by the classification model and the reconstruction model indicate that the traffic to be used is normal traffic, and the traffic is determined to be abnormal only when both the abnormal results output by the classification model and the reconstruction model indicate that the traffic to be used is abnormal traffic, thus further improving the credibility of whether the traffic is normal or abnormal.

[0089] The abnormal results output by the above classification model are the classification results of the aforementioned classification model.

[0090] However, it's important to note that the abnormal results output by the aforementioned reconstruction model are not the actual reconstruction results. Instead, they are reconstruction loss values ​​calculated based on the reconstruction results and the input variable 'x'. Specifically, if the reconstruction loss value is greater than a score threshold, the corresponding traffic is considered abnormal. If the reconstruction loss value is not greater than the score threshold, the corresponding traffic is considered normal. This score threshold can be 200, but it is not limited to this and can be adjusted according to the specific application scenario.

[0091] The reconstruction loss value mentioned above can be calculated using the L2 norm error method. Specifically, the formula for calculating the reconstruction loss value using the L2 norm error method is as follows:

[0092] L(θ2)=‖x ′ -x‖2

[0093] Where L(θ2) represents the loss value of the reconstruction model, i.e., the reconstruction loss value.

[0094] The election and noise reduction module in the aforementioned deep multi-fusion autoencoder model is used to periodically select the most normal traffic from the normal traffic set and the most abnormal traffic from the abnormal traffic set.

[0095] The normal traffic set includes normal traffic, which is traffic determined as normal by the fusion judgment module; the abnormal traffic set includes abnormal traffic, which is traffic determined as abnormal by the fusion judgment module. The most normal traffic is the most normal traffic in the normal traffic set; the most abnormal traffic is the most abnormal traffic in the abnormal traffic set.

[0096] Reference Figure 2 The diagram shown is a flowchart of an anomaly detection method provided in an embodiment of this application. The method includes:

[0097] S201, based on a pre-trained model, obtains the normal traffic set and the abnormal traffic set.

[0098] Before obtaining the normal traffic set and the abnormal traffic set based on the pre-trained model, it is necessary to first obtain the pre-trained model.

[0099] Optionally, the process of obtaining the pre-trained model (i.e., the pre-training process of the deep multi-fusion autoencoder model) can be as follows: Figure 3 The steps S301-S308 are shown below:

[0100] S301: Obtain the traffic to be trained and preprocess it to obtain the input data.

[0101] The training traffic is normal traffic. This training traffic can also be readily available normal traffic to facilitate better pre-training. The preprocessing method for this training traffic can be as described above. Figure 1 The preprocessing methods of the preprocessing module in the document will not be elaborated here.

[0102] The aforementioned training traffic can be obtained using data packets.

[0103] Optionally, when acquiring the training traffic in the form of data packets, the preprocessing method for the training traffic may further include the following:

[0104] Specifically, the acquired data packet (pcap) is first divided into sessions according to 5-tuples. Then, a fixed-size length is truncated from each session to obtain binary segments. These binary segments are then normalized to obtain preprocessed input data, thereby improving the data quality of the input data and thus helping to improve the performance of models pre-trained based on the input data.

[0105] S302 inputs the input data into the encoder in the deep multi-fusion autoencoder model to obtain high-dimensional features.

[0106] In this embodiment, during the process of inputting input data into the encoder of the deep multi-fusion autoencoder model, batch training can be used to input the input data into the encoder to obtain high-dimensional features. These high-dimensional features can be 1×n features; where n is a positive integer.

[0107] S303 processes high-dimensional features using a classification model within a deep multi-fusion autoencoder model to obtain classification results.

[0108] S304. Based on the classification results and input data, the loss is calculated to obtain the classification loss value.

[0109] In this embodiment of the application, the classification loss value can be calculated using the cross-entropy loss function.

[0110] However, since the input data during pre-training only includes normal traffic, the cross-entropy loss function used by the classification model to calculate the classification loss value is as follows:

[0111]

[0112] Where L(θ1) represents the loss value of the classification model, i.e., the classification loss value; y represents the label corresponding to the i-th data in the input data; y ′ This represents the predicted value corresponding to the i-th data in the input data, i.e., the classification result corresponding to the i-th data; m represents the number of samples (i.e., the number of traffic).

[0113] Since the input data only includes normal traffic, y represents normal traffic. For example, if normal traffic is represented by 1 and abnormal traffic is represented by 0, then y is 1.

[0114] S305 processes high-dimensional features through a reconstruction model in a deep multi-fusion autoencoder model to obtain the reconstruction result.

[0115] To improve the accuracy of the reconstruction results, instead of directly reconstructing the high-dimensional features, we first perform a sparse transformation to convert the high-dimensional features into second features, and then decode and reconstruct the second features.

[0116] Therefore, the specific process of processing high-dimensional features and obtaining reconstruction results through the reconstruction model in the deep multi-fusion autoencoder model can be described as follows:

[0117] First, the high-dimensional features are sparsified using the intercept function in the reconstruction model to obtain the first feature, which helps filter out redundant information. Then, the first feature is transformed into the second feature using the transformation matrix in the reconstruction model, thus converting the dimension of the first feature to a preset dimension. This preset dimension can be 1×n, meaning the dimension of the second feature is 1×n. Finally, the high-dimensional features are reconstructed using the reconstruction model to obtain the reconstruction result.

[0118] In this embodiment, the intercept function can be a rectified linear unit (reLU) function. The transformation matrix can be an n×n matrix with the same dimension as the high-dimensional feature. The specific methods of the sparsity processing and dimensionality transformation are shown in the following formula:

[0119]

[0120] z ′ =temp*N

[0121] Among them, z i denoted as the i-th high-dimensional feature; λ represents the intercept of the ReLU function, a preset filtering threshold to filter out redundant information features; δ represents the bias value, a very small bias value used to prevent computational problems caused by a denominator of 0; temp i Let represent the intermediate value corresponding to the i-th high-dimensional feature; temp has a dimension of 1×n; N represents the transformation matrix, which has a dimension of n×n; z ′ This indicates the second characteristic.

[0122] S306. Based on the reconstruction results and input data, the loss is calculated to obtain the reconstruction loss value.

[0123] S307, combine the classification loss value and the reconstruction loss value to obtain the combined loss value.

[0124] Specifically, the following formula can be used to combine the classification loss value and the reconstruction loss value.

[0125] L(θ1, θ2)=αL(θ1)+(1-α)L(θ2)

[0126] Where L(θ1, θ2) represents the merged loss value; α represents the merged weight.

[0127] Optionally, to make the reconstruction loss value more important than the classification loss value, the merging weight (i.e., α) can be set to 0.3 so that the reconstruction loss value has a greater weight than the classification loss value. However, this is not the only option; the merging weight (i.e., α) can be adjusted according to the specific application scenario.

[0128] S308, based on the merged loss value, pre-trains a deep multi-fusion autoencoder model to obtain a pre-trained model.

[0129] The process of pre-training a deep multi-fusion autoencoder model based on the merged loss value described above can be pre-trained using existing techniques that pre-train models based on loss values, and will not be elaborated further here.

[0130] However, it should be noted that in the process of pre-training the deep multi-fusion autoencoder model based on the merged loss value, the pre-training is performed on the encoder, classification model and reconstruction model in the deep multi-fusion autoencoder model, but not on the fusion judgment module and the election denoising module in the deep multi-fusion autoencoder model.

[0131] Thus, the optimal pre-trained model for the current stage is obtained through the above steps S301-S308, so as to obtain the combination of normal traffic set and abnormal traffic based on the pre-trained model.

[0132] The specific process of obtaining the normal traffic set and the abnormal traffic set based on the pre-trained model can be described as follows:

[0133] First, the raw traffic to be used is obtained, and then the raw traffic to be used is preprocessed to obtain the traffic to be used. The preprocessing process of the raw traffic to be used is the same as the preprocessing method of the training traffic in step S301 above, and will not be repeated here.

[0134] Then, the traffic to be used is input into the pre-trained model to obtain a first result and a second result. The first result is the abnormal result output by the classification model in the pre-trained model; the second result is the abnormal result output by the reconstruction model in the pre-trained model. A fusion judgment method is then used to perform a XOR judgment on the first and second results to obtain a third result.

[0135] If the third result is determined to be a normal result, the traffic to be used will be added to the normal traffic set.

[0136] If the third result is determined to be an abnormal result, the traffic to be used will be added to the abnormal traffic set.

[0137] By using the above method, the abnormal results output by the classification model (i.e., the first result) and the abnormal results output by the reconstruction model (i.e., the second result) are corrected, making the third result more reliable. This further improves the reliability of the normal traffic set and the abnormal traffic set obtained based on the third result.

[0138] The aforementioned fusion judgment method can be the fusion judgment module in the aforementioned deep multi-fusion autoencoder model. Therefore, the specific process of performing an XOR judgment on the first result and the second result using the fusion judgment method is consistent with the specific process of the fusion judgment module in the aforementioned deep multi-fusion autoencoder model performing an XOR judgment on the abnormal results output by the classification model and the abnormal results output by the reconstruction model, and will not be repeated here.

[0139] S202 uses an election-based noise reduction method to process the normal traffic set and the abnormal traffic set separately, obtaining the most normal traffic and the most abnormal traffic.

[0140] Due to the methods used to forge abnormal traffic, it is inevitable that a small amount of abnormal traffic will appear within a normal traffic area (i.e., the set of normal traffic), and a small amount of normal traffic will appear within an abnormal traffic area (i.e., the set of abnormal traffic). In other words, normal traffic within a normal traffic set may be pseudo-normal traffic, and abnormal traffic within an abnormal traffic set may be pseudo-abnormal traffic. For example... Figure 4 As shown, there are pseudo-normal flows in the normal flow set and pseudo-abnormal flows in the abnormal flow set; where the dashed circles in the normal flow set represent pseudo-normal flows (i.e., abnormal flows), and the solid circles represent normal flows; the dashed circles in the abnormal flow set represent abnormal flows, and the solid circles represent pseudo-abnormal flows (i.e., normal flows).

[0141] To solve this problem, an election-based noise reduction method can be used to process the normal traffic set and the abnormal traffic set separately. This yields the most normal normal traffic (i.e., the most normal traffic) in the normal traffic set and the most abnormal abnormal traffic (i.e., the most abnormal traffic) in the abnormal traffic set, thus resolving the issue of the blurred boundary between normal and abnormal traffic. For example, ... Figure 4 As shown, the election-based noise reduction method can select the most normal traffic (i.e., circle 1 in the normal traffic set) from the normal traffic set, and the most abnormal traffic (i.e., circle 1' in the abnormal traffic set) from the abnormal traffic set.

[0142] Specifically, after obtaining the normal traffic set and the abnormal traffic set in step S101, at the first time interval, the normal traffic set is processed through election denoising to obtain the most normal traffic, and at the second time interval, the abnormal traffic set is processed through election denoising to obtain the most abnormal traffic. This periodically selects the most normal traffic from the normal traffic set and the most abnormal traffic from the abnormal traffic set through election denoising, ensuring the boundary between normal and abnormal traffic is clear. Furthermore, after selecting the most normal traffic from the normal traffic set, the normal traffic set is cleared; after selecting the most abnormal traffic from the abnormal traffic set, the abnormal traffic set is cleared, so that a new round of normal and abnormal traffic sets can be obtained continuously, thus solving the problem of manually obtaining samples.

[0143] The first and second times mentioned above may or may not be the same.

[0144] The aforementioned election denoising method can be the election denoising module in the aforementioned deep multi-fusion autoencoder model.

[0145] The specific process of processing the normal traffic set using the election denoising method to obtain the most normal traffic can be described as follows:

[0146] First, obtain the first result and the second result for each normal traffic in the normal traffic set.

[0147] Since each normal traffic in the normal traffic set originates from traffic to be used, the abnormal result output by the classification model in the pre-trained model for each normal traffic in the normal traffic set is the aforementioned first result, and the abnormal result output by the reconstruction model in the pre-trained model is the aforementioned second result.

[0148] Therefore, obtaining the abnormal results output by the classification model and the abnormal results output by the reconstruction model for each normal traffic in the normal traffic set is equivalent to obtaining the first result and the second result for each normal traffic in the normal traffic set.

[0149] Then, the first result and the corresponding second result of each normal traffic in the normal traffic set are jointly calculated to obtain the joint value corresponding to each normal traffic in the normal traffic set. The obtained joint values ​​are then sorted in ascending order, and the normal traffic corresponding to the first joint value in the sorted joint values ​​is selected as the most normal traffic. This ensures that the selected most normal traffic has the highest confidence in being normal, thus solving the problem of the blurred boundary between normal and abnormal traffic.

[0150] To further refine the selection of the most normal traffic flow, the first result and corresponding second result of each normal traffic flow in the normal traffic flow set are jointly calculated to obtain the joint value for each normal traffic flow in the normal traffic flow set. This can be done in the following way:

[0151] First, within the normal traffic set, normal traffic meeting the ignore criteria is designated as ignore traffic, thus identifying the traffic that needs to be ignored. This ignore traffic is the traffic that needs to be ignored. The ignore criteria can be either a first result between a first threshold and a second threshold, or a second result between a third threshold and a fourth threshold. Then, for the other normal traffic in the normal traffic set excluding the ignore traffic, the first and second results are jointly calculated to obtain the corresponding joint value for each traffic. This ignores normal traffic where the first result is between the first and second thresholds, and normal traffic where the second result is between the third and fourth thresholds. This initial screening of normal traffic in the normal traffic set, followed by the joint calculation of the other normal traffic excluding the ignore traffic, further improves the reliability of the selected most normal traffic.

[0152] The first and second thresholds mentioned above can be probability thresholds, and the third and fourth thresholds mentioned above can be score thresholds.

[0153] For example, the normal traffic set includes normal traffic of type C1, type C2, type C3, type C4, and type C5. Specifically, the first result for C1 traffic is 0.1, and the second result is 20; the first result for C2 traffic is 0.05, and the second result is 7; the first result for C3 traffic is 0.8, and the second result is 80; the first result for C4 traffic is 0.9, and the second result is 180; and the first result for C5 traffic is 0.6, and the second result is 101. The first probability threshold is 0.3, the second probability threshold is 0.7, the first score threshold is 12, and the second score threshold is 100.

[0154] Therefore, traffic is ignored and classified as C1, C3, and C5. Then, the first and second results corresponding to traffic C2 are jointly calculated to obtain the joint value for traffic C2, and the first and second results corresponding to traffic C4 are jointly calculated to obtain the joint value for traffic C4.

[0155] The first result and the corresponding second result obtained by the joint calculation can be calculated by logarithmic transformation and summation, but are not limited to this method.

[0156] Specifically, the first result is logarithmically transformed to obtain the first logarithmic result. The second result is then logarithmically transformed to obtain the second logarithmic result. The first and second logarithmic results are then summed; the sum is the joint value. This makes the calculated joint value more accurate, which helps to increase the reliability of the most normal traffic selected based on the joint value.

[0157] The process of processing the abnormal traffic set to obtain the most abnormal traffic using the election denoising method described above is the same as the process of processing the normal traffic set to obtain the most normal traffic using the election denoising method described above, and will not be repeated here.

[0158] However, it should be noted that in the process of processing the abnormal traffic set through the election denoising method to obtain the most abnormal traffic, after obtaining the joint value, the joint value is sorted in descending order, and then the abnormal traffic corresponding to the first joint value after sorting is selected as the most abnormal traffic, so as to select the most abnormal traffic.

[0159] For example, such as Figure 4 As shown, the joint values ​​corresponding to the normal flows in the normal flow set are sorted in ascending order, and the order represented by this sorting is as follows: Figure 4 The numbers circled in the normal flow set are shown in the diagram; the normal flow corresponding to the first joint value is circle 1. The joint values ​​corresponding to the abnormal flows in the abnormal flow set are sorted in descending order, and the order represented by this sorting is as follows: Figure 4 The numbers circled in the abnormal traffic set are shown in the diagram. The abnormal traffic corresponding to the first joint value is circle 1'.

[0160] S203, based on the most normal traffic and the most abnormal traffic, obtains the target model, and uses the target model to detect traffic anomalies.

[0161] After obtaining the most normal and most abnormal traffic in step S102, relatively reliable normal and abnormal traffic data can be obtained, gradually enhancing data completeness. Then, every third time interval, the most normal and most abnormal traffic data are fed back into the pre-trained model for enhanced training to obtain the target model. This target model is then used to detect traffic anomalies. This method enables the model to continuously learn, improving the timeliness and accuracy of detecting increasingly updated abnormal traffic patterns.

[0162] Furthermore, by feeding back the most normal traffic and the most abnormal traffic to the pre-trained model every third time interval, the number of most normal traffic and most abnormal traffic is not limited to one, which helps to strengthen the training of the pre-trained model.

[0163] The training methods described above for enhanced training are the same as those for pre-training in step S101, and will not be repeated here.

[0164] However, it should be noted that after feeding the most normal traffic and the most abnormal traffic back to the pre-trained model, the new training dataset includes abnormal traffic in addition to normal traffic, so the labels are not limited to normal traffic. Therefore, in the process of calculating the classification loss value using the cross-entropy loss function in the aforementioned step S304, the cross-entropy loss function can be the cross-entropy loss function of the entire dataset, as follows:

[0165]

[0166] The specific process of using the target model to detect anomalies in traffic can be described as follows:

[0167] After obtaining the raw traffic to be detected, the raw traffic to be detected is preprocessed to obtain the traffic to be detected. The preprocessing process of the raw traffic to be detected is the same as the preprocessing process of the training traffic in step S301 above, and will not be repeated here.

[0168] Then, the traffic to be detected is input into the target model. The target model processes the traffic to be detected to obtain the detection result. The detection result includes whether the traffic to be detected is abnormal or normal, thus achieving accurate anomaly detection.

[0169] In addition, since the most normal traffic and the most abnormal traffic are fed back to the pre-trained model for enhanced training every third time period, the target model will also be updated after a period of time. At this time, the updated target model is used to detect traffic anomalies.

[0170] In summary, the anomaly detection method proposed in this application learns the features of normal traffic through a deep autoencoder (i.e., encoder and reconstruction model) in a deep multi-fusion autoencoder model, enabling it to reconstruct normal traffic features. Furthermore, a classification judgment (i.e., classification model) is added as a side branch to the encoder of the deep multi-fusion autoencoder model, allowing for pre-training on high-dimensional features using only normal traffic (e.g., pre-supervised training) and post-correction training using a mixture of normal and anomaly traffic, thus gradually enhancing data completeness. In addition, the anomaly detection method provided in this application's embodiments starts from incomplete data, does not rely on the detection results of pre-trained models, and the data completeness gradually increases as the model is used longer.

[0171] Furthermore, by performing sparse transformation on the high-dimensional features between the encoder and decoder in the deep multi-fusion autoencoder model (i.e., performing sparse transformation on the high-dimensional features in the reconstruction model), the harmful effects of redundant information features are filtered out.

[0172] Then, the fusion judgment method (i.e. the fusion judgment module in the deep multi-fusion autoencoder model) is used to combine the side branch classification results (i.e. the abnormal results output by the classification model) and the reconstruction results (i.e. the abnormal results output by the reconstruction model) to determine the preliminary boundary between abnormal traffic and normal traffic, so as to obtain the set of normal traffic initially classified as normal traffic and the set of abnormal traffic initially classified as abnormal traffic.

[0173] Then, by using an election denoising method (i.e., the election denoising module in the deep multi-fusion autoencoder model), the most normal normal traffic in the normal traffic set (i.e., the most normal traffic) and the most abnormal traffic in the abnormal traffic set (i.e., the most abnormal traffic) are further processed to obtain the most normal normal traffic in the normal traffic set (i.e., the most abnormal traffic), thereby solving the problem of blurred boundaries between normal and abnormal traffic and helping to improve the accuracy of abnormal traffic detection.

[0174] Next, the obtained maximum and minimum values ​​(i.e., the most normal traffic and the most abnormal traffic) are periodically fed back into the pre-trained model for enhanced training. That is, multiple most normal traffic and most abnormal traffic values ​​are fed back into the pre-trained model for enhanced training, realizing the model's continuous self-learning ability and improving the timeliness and accuracy of detecting increasingly updated abnormal traffic patterns.

[0175] Furthermore, the anomaly detection method provided in this application embodiment continuously and automatically adds positive and negative traffic samples (i.e., normal traffic and abnormal traffic), providing anomaly traffic data capability.

[0176] The technical solution of this application will be further explained below with reference to a specific application process.

[0177] like Figure 5 The diagram illustrates the processing steps of the anomaly detection method. First, in the preprocessing unit, normal traffic is acquired and preprocessed to obtain input data. This input data is then transmitted to the pre-training unit.

[0178] In the pre-training unit, the input data is processed by the encoder in the deep multi-fusion autoencoder model to obtain high-dimensional features. Then, the high-dimensional features are processed by the classification model in the deep multi-fusion autoencoder model to obtain a classification result. A loss calculation is then performed based on the classification result and the input data to obtain a classification loss value. Simultaneously, the high-dimensional features are processed by the reconstruction model in the deep multi-fusion autoencoder model to obtain a reconstruction result. A loss calculation is then performed based on the reconstruction result and the input data to obtain a reconstruction loss value. Next, the classification loss value and the reconstruction loss value are merged to obtain a combined loss value. The deep multi-fusion autoencoder model is then updated using the combined loss value to obtain the pre-trained model. This pre-trained model is then transmitted to the traffic partitioning unit.

[0179] In the traffic segmentation unit, the received traffic to be used is processed by the model processing module using the received pre-trained model to obtain a first result and a second result. Then, the fusion judgment module in the deep multi-fusion autoencoder model performs a XOR judgment on the first and second results to obtain a third result. If the third result indicates that the traffic to be used is abnormal, then the traffic to be used is added to the abnormal traffic set. If the third result indicates that the traffic to be used is normal, then the traffic to be used is added to the normal traffic set. After obtaining the normal traffic set and the abnormal traffic set, the election and denoising module in the deep multi-fusion autoencoder model periodically selects the most normal traffic from the normal traffic set and the most abnormal traffic from the abnormal traffic set. The most abnormal traffic and the most normal traffic are then transmitted to the feedback unit.

[0180] In the feedback unit, the most normal traffic and the most abnormal traffic are periodically fed back to the pre-trained model for enhanced training to obtain the target model. Based on the target model, anomaly detection is performed on the traffic.

[0181] Based on the same inventive concept, this application also provides an anomaly detection device, such as... Figure 6 The diagram shown is a structural schematic of an anomaly detection device provided in this application. The device includes:

[0182] The first partitioning module 601 is used to obtain the normal traffic set and the abnormal traffic set based on the pre-trained model;

[0183] The second partitioning module 602 is used to process the normal traffic set and the abnormal traffic set respectively through an election noise reduction method to obtain the most normal traffic and the most abnormal traffic; wherein, the most normal traffic is the most normal normal traffic in the normal traffic set; and the most abnormal traffic is the most abnormal abnormal traffic in the abnormal traffic set.

[0184] The processing module 603 is used to obtain a target model based on the most normal traffic and the most abnormal traffic, and to perform anomaly detection on the traffic using the target model.

[0185] In one possible implementation, the apparatus further includes a pre-training module, configured to acquire the traffic to be trained and preprocess the normal traffic to obtain input data; wherein the traffic to be trained is normal traffic; input the input data into the encoder of a deep multi-fusion autoencoder model to obtain high-dimensional features; process the high-dimensional features through a classification model in the deep multi-fusion autoencoder model to obtain a classification result; calculate a loss based on the classification result and the input data to obtain a classification loss value; process the high-dimensional features through a reconstruction model in the deep multi-fusion autoencoder model to obtain a reconstruction result; calculate a loss based on the reconstruction result and the input data to obtain a reconstruction loss value; merge the classification loss value and the reconstruction loss value to obtain a merged loss value; and update the deep multi-fusion autoencoder model based on the merged loss value to obtain the pre-trained model.

[0186] In one possible implementation, the pre-training module is specifically used to perform sparse processing on the high-dimensional features through the intercept function in the reconstruction model of the deep multi-fusion autoencoder model to obtain a first feature; process the first feature through the transformation matrix in the reconstruction model to obtain a second feature; and decode and reconstruct the second feature through the decoder in the reconstruction model to obtain the reconstruction result.

[0187] In one possible implementation, the first partitioning module 601 is specifically used to input the traffic to be used into the pre-trained model to obtain a first result and a second result; wherein, the first result is an abnormal result output by the classification model in the pre-trained model, and the second result is an abnormal result output by the reconstruction model in the pre-trained model; the first result and the second result are XORed by a fusion judgment method to obtain a third result; if it is determined that the third result indicates that the traffic to be used is normal traffic, then the traffic to be used is added to the normal traffic set; if it is determined that the third result indicates that the traffic to be used is abnormal traffic, then the traffic to be used is added to the abnormal traffic set.

[0188] In one possible implementation, the first segmentation module 601 is specifically configured to obtain the third result indicating that the traffic to be used is normal traffic if the first result indicates that the traffic to be used is normal traffic and the second result indicates that the traffic to be used is normal traffic; and to obtain the third result indicating that the traffic to be used is abnormal traffic if the first result indicates that the traffic to be used is abnormal traffic and the second result indicates that the traffic to be used is abnormal traffic.

[0189] In one possible implementation, the second partitioning module 602 is specifically configured to process the normal traffic set at each first time interval using the election noise reduction method to obtain the most normal traffic and clear the normal traffic set; and to process the abnormal traffic set at each second time interval using the election noise reduction method to obtain the most abnormal traffic and clear the abnormal traffic set.

[0190] In one possible implementation, the second partitioning module 602 is specifically configured to obtain a first result and a second result corresponding to each normal flow in the normal flow set, and to obtain a first result and a second result corresponding to each abnormal flow in the abnormal flow set; jointly calculate the first result and the corresponding second result to obtain a joint value corresponding to the corresponding flow; sort the joint values ​​corresponding to the normal flows in the normal flow set in ascending order, select the normal flow corresponding to the first joint value as the most normal flow, and sort the joint values ​​corresponding to the abnormal flows in the abnormal flow set in descending order, select the abnormal flow corresponding to the first joint value as the most abnormal flow.

[0191] In one possible implementation, the second partitioning module 602 is specifically used to treat all normal traffic and abnormal traffic in the normal traffic set and the abnormal traffic set that meet the ignoring conditions as ignored traffic; wherein, the ignoring conditions are that the first result is between the first threshold and the second threshold or the second result is between the third threshold and the fourth threshold; for the traffic in the normal traffic set and the abnormal traffic set other than the ignored traffic, the corresponding first result and the second result are jointly calculated to obtain the joint value corresponding to the corresponding traffic.

[0192] In one possible implementation, the processing module 603 is specifically used to feed back the most normal traffic and the most abnormal traffic to the pre-trained model for enhanced training every third time interval to obtain the target model.

[0193] Based on the same inventive concept, this application also provides an electronic device that can realize the function of the aforementioned anomaly detection device. (Refer to...) Figure 7 The aforementioned electronic devices include:

[0194] At least one processor 701 and a memory 702 connected to at least one processor 701. In this embodiment, the specific connection medium between the processor 701 and the memory 702 is not limited. Figure 7 The example shown is the connection between processor 701 and memory 702 via bus 700. Bus 700 is... Figure 7The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. The 700 bus can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 7 The term is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, the processor 701 can also be called a controller; there is no restriction on the name.

[0195] In this embodiment, memory 702 stores instructions executable by at least one processor 701. By executing the instructions stored in memory 702, at least one processor 701 can perform the anomaly detection method described above. Processor 701 can implement... Figure 6 The functions of each module in the device shown.

[0196] The processor 701 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 702 and calling data stored in memory 702, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0197] In one possible design, processor 701 may include one or more processing units. Processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 701. In some embodiments, processor 701 and memory 702 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0198] The processor 701 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the anomaly detection method disclosed in the embodiments of this application can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0199] Memory 702, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 702 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory 702 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 702 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0200] By designing and programming the processor 701, the code corresponding to the anomaly detection method described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute it during operation. Figure 2 The steps of the anomaly detection method in the illustrated embodiment are described below. How to design and program the processor 701 is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0201] Based on the same inventive concept, embodiments of this application also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the anomaly detection method described above.

[0202] In some possible implementations, various aspects of the anomaly detection method provided in this application may also be implemented in the form of a program product, which includes program code that, when the program product is run on a device, causes the control device to perform the steps in the anomaly detection method according to the various exemplary embodiments of this application described above.

[0203] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0204] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0205] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0206] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0207] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An anomaly detection method, characterized in that, include: Based on the pre-trained model, obtain the normal traffic set and the abnormal traffic set; Obtain the first result and the second result corresponding to each normal traffic in the normal traffic set, and obtain the first result and the second result corresponding to each abnormal traffic in the abnormal traffic set; wherein, the first result corresponding to the normal traffic and the first result corresponding to the abnormal traffic are both abnormal results output by the classification model in the pre-trained model; the second result corresponding to the normal traffic and the second result corresponding to the abnormal traffic are both abnormal results output by the reconstruction model in the pre-trained model. By jointly calculating the first result and the corresponding second result, the joint value corresponding to the flow rate is obtained; The joint values ​​corresponding to the normal traffic in the normal traffic set are sorted in ascending order, and the normal traffic corresponding to the first joint value is selected as the most normal traffic. The joint values ​​corresponding to the abnormal traffic in the abnormal traffic set are sorted in descending order, and the abnormal traffic corresponding to the first joint value is selected as the most abnormal traffic. The most normal traffic is the most normal traffic in the normal traffic set, and the most abnormal traffic is the most abnormal traffic in the abnormal traffic set. Every third time interval, the most normal traffic and the most abnormal traffic are fed back to the pre-trained model for enhanced training to obtain the target model, and the target model is used to detect traffic anomalies.

2. The method as described in claim 1, characterized in that, Before obtaining the normal traffic set and the abnormal traffic set based on the pre-trained model, the following is also included: The training traffic is acquired and preprocessed to obtain input data; wherein the training traffic is normal traffic. The input data is fed into the encoder in the deep multi-fusion autoencoder model to obtain high-dimensional features; The high-dimensional features are processed by the classification model in the deep multi-fusion autoencoder model to obtain a classification result; and a loss is calculated based on the classification result and the input data to obtain a classification loss value; and The high-dimensional features are processed by the reconstruction model in the deep multi-fusion autoencoder model to obtain the reconstruction result; and the reconstruction loss value is obtained by calculating the loss based on the reconstruction result and the input data. The classification loss value and the reconstruction loss value are combined to obtain the combined loss value; Based on the merged loss value, the deep multi-fusion autoencoder model is updated to obtain the pre-trained model.

3. The method as described in claim 2, characterized in that, The process of processing the high-dimensional features using the reconstruction model in the deep multi-fusion autoencoder model to obtain the reconstruction result includes: The high-dimensional features are sparsely processed by the intercept function in the reconstruction model of the deep multi-fusion autoencoder model to obtain the first feature; The first feature is processed by the transformation matrix in the reconstruction model of the deep multi-fusion autoencoder model to obtain the second feature; The second feature is decoded and reconstructed using the decoder in the reconstruction model of the deep multi-fusion autoencoder model to obtain the reconstruction result.

4. The method as described in claim 1, characterized in that, The process of obtaining the normal traffic set and the abnormal traffic set based on the pre-trained model includes: Input the traffic to be used into the pre-trained model to obtain the first result and the second result; A third result is obtained by performing a binary OR operation on the first result and the second result using a fusion judgment method. If the third result indicates that the traffic to be used is normal traffic, then the traffic to be used is added to the set of normal traffic. If the third result indicates that the traffic to be used is abnormal traffic, then the traffic to be used is added to the abnormal traffic set.

5. The method as described in claim 4, characterized in that, The step of performing an XOR judgment on the first result and the second result through a fusion judgment method to obtain a third result includes: If the first result indicates that the traffic to be used is normal traffic, and the second result indicates that the traffic to be used is normal traffic, then the third result indicating that the traffic to be used is normal traffic is obtained; If the first result indicates that the traffic to be used is abnormal traffic, and the second result indicates that the traffic to be used is abnormal traffic, then the third result indicating that the traffic to be used is abnormal traffic is obtained.

6. The method as described in claim 1, characterized in that, After obtaining the normal traffic set and the abnormal traffic set, the following is also included: At each first interval, the normal traffic set is processed using an election-based noise reduction method to obtain the most normal traffic, and then the normal traffic set is cleared; and Every second time interval, the abnormal traffic set is processed using the election noise reduction method to obtain the most abnormal traffic, and the abnormal traffic set is then cleared.

7. The method as described in claim 1, characterized in that, The joint calculation of the first result and the corresponding second result yields the joint value corresponding to the flow rate, including: Normal traffic and abnormal traffic that meet the ignore conditions in the normal traffic set and the abnormal traffic set are all ignored traffic; wherein, the ignore conditions are either a first result between a first threshold and a second threshold or a second result between a third threshold and a fourth threshold; For the traffic in the normal traffic set and the abnormal traffic set excluding the ignored traffic, perform a joint calculation of the corresponding first result and second result to obtain the joint value corresponding to the traffic.

8. An anomaly detection device, characterized in that, include: The first partitioning module is used to obtain the normal traffic set and the abnormal traffic set based on the pre-trained model; The second partitioning module is used to obtain the first result and second result corresponding to each normal traffic in the normal traffic set, and to obtain the first result and second result corresponding to each abnormal traffic in the abnormal traffic set; wherein, the first result corresponding to the normal traffic and the first result corresponding to the abnormal traffic are both abnormal results output by the classification model in the pre-trained model; the second result corresponding to the normal traffic and the second result corresponding to the abnormal traffic are both abnormal results output by the reconstruction model in the pre-trained model; the first result and the corresponding second result are jointly calculated to obtain the joint value corresponding to the corresponding traffic; the joint values ​​corresponding to the normal traffic in the normal traffic set are sorted in ascending order, and the normal traffic corresponding to the first joint value is selected as the most normal traffic; and the joint values ​​corresponding to the abnormal traffic in the abnormal traffic set are sorted in descending order, and the abnormal traffic corresponding to the first joint value is selected as the most abnormal traffic; wherein, the most normal traffic is the most normal normal traffic in the normal traffic set; and the most abnormal traffic is the most abnormal abnormal traffic in the abnormal traffic set. The processing module is used to feed back the most normal traffic and the most abnormal traffic to the pre-trained model for enhanced training every third time interval to obtain the target model, and to perform anomaly detection on the traffic using the target model.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a computer program stored in the memory, implements the method steps of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-7.

Images