Abnormality detection method and device and electronic equipment

Abstract

The invention discloses an anomaly detection method and device and electronic equipment, and the method comprises the steps: obtaining a normal traffic set and an abnormal traffic set based on a pre-training model; respectively processing the normal flow set and the abnormal flow set by electing a noise reduction mode to obtain the most normal flow and the most abnormal flow; and obtaining a target model based on the most normal traffic and the most abnormal traffic, and performing anomaly detection on the traffic through the target model. Through the technical scheme provided by the embodiment of the invention, the problem that the boundary between the normal traffic and the abnormal traffic is fuzzy is solved, and the false alarm rate of anomaly detection is reduced.

Description

Technical Field

[0001] The present application relates to the field of network and information security technology, and in particular to an anomaly detection method, device and electronic equipment. Background Art

[0002] Anomaly detection refers to finding patterns in data that do not conform to expected behavior. Therefore, a simple approach to anomaly detection is to define a region that represents normal behavior and declare any observation in the data that does not fall into this normal region as an anomaly.

[0003] However, in traffic anomaly detection, the means of disguising abnormal traffic attacks as normal traffic access (that is, disguising abnormal traffic as normal traffic, that is, disguising abnormal behavior as normal behavior) have been improving, making it difficult to define truly normal traffic, resulting in a blurred boundary between normal traffic and abnormal traffic, which in turn leads to a high false alarm rate in traffic anomaly detection. Summary of the invention

[0004] The present application provides an anomaly detection method, device and electronic device to solve the problem that the boundary between normal traffic and abnormal traffic is blurred, resulting in a high false alarm rate in traffic anomaly detection. The specific implementation scheme is as follows:

[0005] In a first aspect, the present application provides an anomaly detection method, the method comprising:

[0006] Based on the pre-trained model, obtain the normal traffic set and the abnormal traffic set;

[0007] By means of election noise reduction, the normal traffic set and the abnormal traffic set are processed respectively to obtain the most normal traffic and the most abnormal traffic; wherein the most normal traffic is the most normal normal traffic in the normal traffic set; and the most abnormal traffic is the most abnormal abnormal traffic in the abnormal traffic set;

[0008] Based on the most normal flow and the most abnormal flow, a target model is obtained, and abnormality detection is performed on the flow through the target model.

[0009] Through the above application embodiment, a normal traffic set and an abnormal traffic set are obtained based on a pre-trained model, thereby achieving a preliminary division of normal traffic and abnormal traffic, and then the normal traffic set and the abnormal traffic set are processed by an election noise reduction method to obtain the most normal traffic in the normal traffic set and the most abnormal traffic in the abnormal traffic set, thereby achieving a further division of normal traffic and abnormal traffic, so that the boundary between normal traffic and abnormal traffic can be clarified, thereby solving the problem of blurred boundaries between normal traffic and abnormal traffic. Then, based on the most normal traffic and the most abnormal traffic after clear division, a target model is obtained, and anomaly detection is performed through the target model, so that the false alarm rate of traffic anomaly detection can be reduced.

[0010] In a possible implementation manner, before acquiring the normal traffic set and the abnormal traffic set based on the pre-trained model, the method further includes:

[0011] Acquire the flow to be trained, and pre-process the normal flow to obtain input data; wherein the flow to be trained is the normal flow;

[0012] Inputting the input data into an encoder in a deep multi-fusion autoencoder model to obtain high-dimensional features;

[0013] Processing the high-dimensional features through the classification model in the deep multi-fusion autoencoder model to obtain a classification result; and performing loss calculation based on the classification result and the input data to obtain a classification loss value; and

[0014] The high-dimensional features are processed by a reconstruction model in the deep multi-fusion autoencoder model to obtain a reconstruction result; and a loss calculation is performed based on the reconstruction result and the input data to obtain a reconstruction loss value;

[0015] Combining the classification loss value and the reconstruction loss value to obtain a combined loss value;

[0016] Based on the combined loss value, the deep multi-fusion autoencoder model is updated to obtain the pre-trained model.

[0017] Through the above-mentioned application embodiments, the deep multi-fusion autoencoder model is jointly updated based on the classification model in the deep multi-fusion autoencoder model and the reconstruction model in the deep multi-fusion autoencoder model, so that the obtained pre-trained model takes into account both the situation of the classification model and the situation of the reconstruction model, which is beneficial to improving the model performance of the pre-trained model, thereby facilitating improving the accuracy of the normal traffic set and the abnormal traffic set obtained based on the pre-trained model, and enabling the pre-trained model to identify the distribution characteristics of the given normal traffic, and preliminarily identify the normal traffic based on the characteristics.

[0018] In a possible implementation, the processing of the high-dimensional features by a reconstruction model in the deep multi-fusion autoencoder model to obtain a reconstruction result includes:

[0019] Performing sparse processing on the high-dimensional feature by using an intercept function in the reconstruction model in the deep multi-fusion autoencoder model to obtain a first feature;

[0020] Processing the first feature by using the transformation matrix in the reconstructed model to obtain a second feature;

[0021] The second feature is decoded and reconstructed by a decoder in the reconstruction model to obtain the reconstruction result.

[0022] Through the above application embodiment, the high-dimensional features are first sparsely processed by the intercept function in the reconstruction model, so as to filter out the redundant information features in the high-dimensional features, thereby avoiding the harmful effects of the redundant information features. Then, the first feature after sparse processing is processed by the transformation matrix in the reconstruction model, so as to convert the dimension of the first feature into a second feature that meets the preset dimension, and combines the key information so that the second feature meets the requirements. Then, the second feature is decoded and reconstructed by the decoder in the reconstruction model, and a more accurate reconstruction result is obtained, which is conducive to improving the accuracy of the reconstruction loss value obtained based on the reconstruction result.

[0023] In a possible implementation, the obtaining of a normal traffic set and an abnormal traffic set based on a pre-trained model includes:

[0024] Input the traffic to be used into the pre-trained model to obtain a first result and a second result; wherein the first result is an abnormal result output by the classification model in the pre-trained model, and the second result is an abnormal result output by the reconstruction model in the pre-trained model;

[0025] Perform an identical or identical (OR) judgment on the first result and the second result by a fusion judgment method to obtain a third result;

[0026] If it is determined that the third result indicates that the traffic to be used is normal traffic, adding the traffic to be used to the normal traffic set;

[0027] If it is determined that the third result indicates that the traffic to be used is abnormal traffic, the traffic to be used is added to the abnormal traffic set.

[0028] Through the above-mentioned application embodiment, the traffic to be used is input into the pre-trained model, and the abnormal result (i.e., the first result) output by the classification model in the pre-trained model and the abnormal result (i.e., the second result) output by the reconstruction model in the pre-trained model are obtained. Then, the first result and the second result are judged to be the same or the same through a fusion judgment method, so that the abnormal result (i.e., the first result) output by the classification model and the abnormal result (i.e., the second result) output by the reconstruction model are corrected, and the preliminary boundary between normal traffic and abnormal traffic is obtained, and the credibility of the third result is made higher, so that the credibility of the normal traffic set and the abnormal traffic set obtained based on the third result is further improved.

[0029] In a possible implementation manner, performing an identical or identical judgment on the first result and the second result by a fusion judgment method to obtain a third result includes:

[0030] If the first result indicates that the flow rate to be used is normal flow rate, and the second result indicates that the flow rate to be used is normal flow rate, then the third result indicating that the flow rate to be used is normal flow rate is obtained;

[0031] If the first result indicates that the traffic to be used is abnormal traffic, and the second result indicates that the traffic to be used is abnormal traffic, then the third result indicating that the traffic to be used is abnormal traffic is obtained.

[0032] Through the above-mentioned application embodiments, the third result that the traffic to be used is normal traffic is determined only when the first result and the second result both indicate that the traffic to be used is normal traffic, and the third result that the traffic to be used is abnormal traffic is determined only when the first result and the second result both indicate that the traffic to be used is abnormal traffic, so that the credibility of the third result is further improved.

[0033] In a possible implementation manner, the process of respectively processing the normal traffic set and the abnormal traffic set by means of election noise reduction to obtain the most normal traffic and the most abnormal traffic includes:

[0034] At first intervals, the normal traffic set is processed by the election noise reduction method to obtain the most normal traffic, and the normal traffic set is cleared; and

[0035] At second intervals, the abnormal traffic set is processed by the election noise reduction method to obtain the most abnormal traffic, and the abnormal traffic set is cleared.

[0036] Through the above application embodiment, the most normal flow is selected from the normal flow set every first time, and the most abnormal flow is selected from the abnormal flow set every second time, so that the boundary between normal flow and abnormal flow is not blurred. In addition, the most normal flow and the most abnormal flow are selected in a regular manner (i.e., every first time, every second time), so that when the most normal flow and the most abnormal flow are selected, a certain amount of flow is included in the normal flow set and the abnormal flow set. In addition, the most normal flow and the most abnormal flow are selected in a regular manner (i.e., every first time, every second time), and after selecting the most normal flow from the normal flow set, the normal flow set is cleared, and after selecting the most abnormal flow from the abnormal flow set, the abnormal flow set is cleared, so that a new round of normal flow sets and abnormal flow sets can be continuously obtained in the future, so that the most normal flow and the most abnormal flow can be obtained continuously, and then the positive and negative flow samples (i.e., normal flow and abnormal flow) can be continuously and automatically increased, providing abnormal flow data capabilities and solving the problem of manually obtaining samples.

[0037] In a possible implementation manner, the process of respectively processing the normal traffic set and the abnormal traffic set by means of election noise reduction to obtain the most normal traffic and the most abnormal traffic includes:

[0038] Respectively obtain the first result and the second result corresponding to each normal flow in the normal flow set, and respectively obtain the first result and the second result corresponding to each abnormal flow in the abnormal flow set;

[0039] The first result and the corresponding second result are jointly calculated to obtain a joint value corresponding to the corresponding flow rate;

[0040] The joint values ​​corresponding to the normal traffic in the normal traffic set are sorted in ascending order, and the normal traffic corresponding to the first joint value is selected as the most normal traffic; and the joint values ​​corresponding to the abnormal traffic in the abnormal traffic set are sorted in descending order, and the abnormal traffic corresponding to the first joint value is selected as the most abnormal traffic.

[0041] Through the above application embodiment, the first result and the second result corresponding to each normal flow in the normal flow set are jointly calculated, and then the obtained joint values ​​are sorted in ascending order, and then the normal flow corresponding to the first joint value is selected as the most normal flow, so that the selected most normal flow has the highest credibility of being normal. Similarly, the first result and the second result corresponding to each abnormal flow in the abnormal flow set are jointly calculated, and then the obtained joint values ​​are sorted in descending order, and then the abnormal flow corresponding to the first joint value is selected as the most abnormal flow, so that the selected most abnormal flow has the highest credibility of being abnormal. Therefore, through the selection of the most normal flow and the most abnormal flow with the highest credibility, the problem of blurred boundary between normal flow and abnormal flow is solved.

[0042] In a possible implementation manner, the jointly calculating the first result and the corresponding second result to obtain a joint value corresponding to the corresponding flow includes:

[0043] The normal traffic and abnormal traffic that meet the ignore condition in the normal traffic set and the abnormal traffic set are both regarded as ignore traffic; wherein the ignore condition is that the first result is between the first threshold and the second threshold or the second result is between the third threshold and the fourth threshold;

[0044] For the traffic in the normal traffic set and the abnormal traffic set except the ignored traffic, a corresponding joint calculation of the first result and the second result is performed to obtain a joint value corresponding to the corresponding traffic.

[0045] Through the above-mentioned application embodiment, the ignored traffic that meets the ignored conditions is first determined in the normal traffic set and the abnormal traffic set, so that the ignored traffic can be ignored in the subsequent joint calculation, thereby performing a preliminary screening of the normal traffic set and the abnormal traffic set, and then performing a joint calculation of the first result and the second result for the traffic other than the ignored traffic in the normal traffic set and the abnormal traffic set, so that the credibility of the selected most normal traffic and most abnormal traffic is further improved.

[0046] In a possible implementation manner, obtaining a target model based on the most normal flow and the most abnormal flow includes:

[0047] Every third time period, the most normal flow and the most abnormal flow are fed back to the pre-training model for enhanced training to obtain the target model.

[0048] Through the above-mentioned application embodiment, the most normal traffic and the most abnormal traffic are fed back to the pre-trained model every third time for enhanced training, so that the model is updated regularly, and then the model is continuously improved, making the results of the model more credible, maintaining the timeliness of the model, and prompting the accuracy of the model in detecting traffic anomalies.

[0049] In a second aspect, the present application further provides an abnormality detection device, the device comprising:

[0050] The first partitioning module is used to obtain a normal traffic set and an abnormal traffic set based on a pre-trained model;

[0051] The second division module is used to process the normal traffic set and the abnormal traffic set respectively through the election noise reduction method to obtain the most normal traffic and the most abnormal traffic; wherein the most normal traffic is the most normal normal traffic in the normal traffic set; the most abnormal traffic is the most abnormal abnormal traffic in the abnormal traffic set;

[0052] The processing module is used to obtain a target model based on the most normal flow and the most abnormal flow, and perform abnormality detection on the flow through the target model.

[0053] In a possible implementation, the device further includes a pre-training module, which is used to obtain the traffic to be trained and pre-process the normal traffic to obtain input data; wherein the traffic to be trained is normal traffic; the input data is input into the encoder in the deep multi-fusion autoencoder model to obtain high-dimensional features; the high-dimensional features are processed by the classification model in the deep multi-fusion autoencoder model to obtain a classification result; and a loss is calculated based on the classification result and the input data to obtain a classification loss value; and the high-dimensional features are processed by the reconstruction model in the deep multi-fusion autoencoder model to obtain a reconstruction result; and a loss is calculated based on the reconstruction result and the input data to obtain a reconstruction loss value; the classification loss value and the reconstruction loss value are merged to obtain a merged loss value; based on the merged loss value, the deep multi-fusion autoencoder model is updated to obtain the pre-training model.

[0054] In one possible implementation, the pre-training module is specifically used to perform sparse processing on the high-dimensional features through the intercept function in the reconstruction model in the deep multi-fusion autoencoder model to obtain a first feature; process the first feature through the transformation matrix in the reconstruction model to obtain a second feature; decode and reconstruct the second feature through the decoder in the reconstruction model to obtain the reconstruction result.

[0055] In a possible implementation, the first division module is specifically used to input the traffic to be used into the pre-trained model to obtain a first result and a second result; wherein the first result is an abnormal result output by the classification model in the pre-trained model, and the second result is an abnormal result output by the reconstruction model in the pre-trained model; the first result and the second result are subjected to an identical or identical judgment by a fusion judgment method to obtain a third result; if it is determined that the third result indicates that the traffic to be used is normal traffic, the traffic to be used is added to the normal traffic set; if it is determined that the third result indicates that the traffic to be used is abnormal traffic, the traffic to be used is added to the abnormal traffic set.

[0056] In a possible implementation, the first division module is specifically used to obtain the third result indicating that the traffic to be used is normal traffic if the first result indicates that the traffic to be used is normal traffic and the second result indicates that the traffic to be used is normal traffic; if the first result indicates that the traffic to be used is abnormal traffic and the second result indicates that the traffic to be used is abnormal traffic, then obtain the third result indicating that the traffic to be used is abnormal traffic.

[0057] In a possible implementation, the second division module is specifically used to process the normal traffic set through the election noise reduction method at first intervals to obtain the most normal traffic and clear the normal traffic set; and to process the abnormal traffic set through the election noise reduction method at second intervals to obtain the most abnormal traffic and clear the abnormal traffic set.

[0058] In a possible implementation, the second partitioning module is specifically used to respectively obtain the first result and the second result corresponding to each normal flow in the normal flow set, and respectively obtain the first result and the second result corresponding to each abnormal flow in the abnormal flow set; jointly calculate the first result and the corresponding second result to obtain a joint value corresponding to the corresponding flow; sort the joint values ​​corresponding to the normal flows in the normal flow set in ascending order, select the normal flow corresponding to the first joint value as the most normal flow, and sort the joint values ​​corresponding to the abnormal flows in the abnormal flow set in descending order, and select the abnormal flow corresponding to the first joint value as the most abnormal flow.

[0059] In a possible implementation, the second division module is specifically used to treat the normal traffic and abnormal traffic in the normal traffic set and the abnormal traffic set that meet the ignoring condition as ignored traffic; wherein the ignoring condition is that the first result is between the first threshold and the second threshold or the second result is between the third threshold and the fourth threshold; for the traffic in the normal traffic set and the abnormal traffic set except the ignored traffic, the corresponding first result and the second result are jointly calculated to obtain the joint value corresponding to the corresponding traffic.

[0060] In a possible implementation, the processing module is specifically configured to feed back the most normal flow and the most abnormal flow to the pre-training model every third time period for enhanced training to obtain the target model.

[0061] In a third aspect, the present application provides an electronic device, including:

[0062] Memory, used to store computer programs;

[0063] The processor is used to implement the above-mentioned abnormality detection method steps when executing the computer program stored in the memory.

[0064] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned anomaly detection method steps are implemented.

[0065] For each aspect from the second to the fourth aspect and the technical effects that may be achieved by each aspect, please refer to the above description of the technical effects that can be achieved by the first aspect or various possible schemes in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic diagram of a deep multi-fusion autoencoder model provided in an embodiment of the present application;

[0067] Figure 2 A flowchart of an anomaly detection method provided in an embodiment of the present application;

[0068] Figure 3 A schematic diagram of a pre-training process of a deep multi-fusion autoencoder model provided in an embodiment of the present application;

[0069] Figure 4 A schematic diagram of a normal flow set and an abnormal flow set provided in an embodiment of the present application;

[0070] Figure 5 A schematic diagram of a processing process of an anomaly detection method provided in an embodiment of the present application;

[0071] Figure 6 A schematic diagram of an abnormality detection device provided in an embodiment of the present application;

[0072] Figure 7 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The specific operating methods in the method embodiments can also be applied to device embodiments or system embodiments. It should be noted that in the description of the present application, "multiple" is understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A is connected to B, which can represent: A is directly connected to B and A is connected to B through C. In addition, in the description of the present application, words such as "first" and "second" are only used to distinguish the purpose of description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0074] The embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0075] Most of the existing traffic anomaly detection methods use machine learning or deep learning automatic learning methods to identify abnormal traffic. However, this method does not solve the problem of blurred boundaries between normal and abnormal traffic, and a large amount of pseudo-abnormal traffic and pseudo-normal traffic will be generated during the detection process, resulting in a high false alarm rate for traffic anomaly detection.

[0076] Therefore, the present application proposes an anomaly detection method, which first obtains a normal traffic set and an abnormal traffic set through a pre-training model, thereby achieving a preliminary division of normal traffic and abnormal traffic. Then, through the election noise reduction method, the normal traffic set and the abnormal traffic set are processed separately to obtain the most normal normal traffic in the normal traffic set (i.e., the most normal traffic) and the most abnormal traffic in the abnormal traffic set (i.e., the abnormal traffic set), thereby further accurately dividing the normal traffic and the abnormal traffic, solving the problem of blurred boundaries between normal traffic and abnormal traffic. Then, based on the target model obtained from the most normal traffic and the most abnormal traffic, the traffic is detected for anomalies, which significantly reduces the false alarm rate of traffic anomaly detection.

[0077] In an embodiment of the present application, the target model is obtained after enhanced training based on a pre-trained model; the pre-trained model is obtained after pre-training a deep multi-fusion autoencoder model.

[0078] The above deep multi-fusion autoencoder model can be Figure 1 The deep multi-fusion autoencoder model includes an encoder, a classification model, a sparse transform, a decoder, a fusion judgment module, an election denoising module, etc. Among them, the sparse transform and the decoder can form a reconstruction model.

[0079] The encoder in the above deep multi-fusion autoencoder model is used to encode the data (i.e. Figure 1 The input in is processed to obtain high-dimensional features (i.e. Figure 1 The input is the data preprocessed by the preprocessing module. The preprocessing methods in the preprocessing module include common preprocessing methods, such as data cleaning, normalization processing, etc., which will not be repeated here.

[0080] The classification model in the above deep multi-fusion autoencoder model is used to process high-dimensional features and obtain classification results (i.e. Figure 1 y' in ). The classification model may be a conventional deep learning classification. The classification result may be expressed in the form of probability. Based on the classification result, it may be determined whether the corresponding input is abnormal, that is, whether the corresponding input is normal traffic or abnormal traffic.

[0081] Exemplarily, the A traffic preprocessed by the preprocessing module is input into the encoder in the deep multi-fusion autoencoder, and then the output of the encoder (i.e., high-dimensional features) is input into the classification model in the deep multi-fusion autoencoder. The classification model outputs a classification result (i.e., y'), which is the output probability of the A traffic. The output probability is then compared with the probability threshold. If it is determined that the output probability is less than the probability threshold, the A traffic is determined to be abnormal traffic. If it is determined that the output probability is not less than the probability threshold, the A traffic is determined to be normal traffic.

[0082] The above output probability is usually between 0 and 1, and the probability threshold may be 0.5, but is not limited thereto. In other words, if the output probability is less than 0.5, traffic A is abnormal traffic; if the output probability is not less than 0.5, traffic A is normal traffic.

[0083] The reconstruction model in the above-mentioned deep multi-fusion autoencoder model is used to process high-dimensional features and obtain the reconstruction result (i.e. Figure 1 x' in the

[0084] The fusion judgment module in the above-mentioned deep multi-fusion autoencoder model is used to perform the same or same judgment on the abnormal results output by the classification model and the abnormal results output by the reconstruction model to obtain more accurate results.

[0085] Exemplarily, if the abnormal result output by the classification model indicates that the corresponding traffic is normal traffic, and the abnormal result output by the reconstruction model also indicates that the corresponding traffic is normal traffic, then it is determined that the corresponding traffic is normal traffic.

[0086] If the abnormal result output by the classification model indicates that the corresponding traffic is abnormal traffic, and the abnormal result output by the reconstruction model also indicates that the corresponding traffic is abnormal traffic, then it is determined that the corresponding traffic is abnormal traffic.

[0087] If the abnormal result output by the classification model indicates that the corresponding traffic is abnormal traffic, while the abnormal result output by the reconstruction model indicates that the corresponding traffic is normal traffic; or if the abnormal result output by the classification model indicates that the corresponding traffic is normal traffic, while the abnormal result output by the reconstruction model also indicates that the corresponding traffic is abnormal traffic, the corresponding traffic is discarded.

[0088] Therefore, the corresponding traffic is determined to be normal traffic only when the abnormal results output by the classification model and the reconstruction model both indicate that the traffic to be used is normal traffic, and the corresponding traffic is determined to be abnormal traffic only when the abnormal results output by the classification model and the reconstruction model both indicate that the traffic to be used is abnormal traffic, thereby further improving the credibility of whether the traffic is normal traffic or abnormal traffic.

[0089] The abnormal result output by the above classification model is the classification result of the aforementioned classification model.

[0090] However, it should be noted that the abnormal result output by the above reconstruction model is not the aforementioned reconstruction result, but the reconstruction loss value obtained after the loss calculation is performed based on the reconstruction result and x in the input. Specifically, if it is determined that the reconstruction loss value is greater than the score threshold, the corresponding traffic is determined to be abnormal, that is, the corresponding traffic is abnormal traffic. If it is determined that the reconstruction loss value is not greater than the score threshold, the corresponding traffic is determined to be normal, that is, the corresponding traffic is normal traffic. The score threshold can be 200, but is not limited to this, and can be adjusted according to the specific application scenario.

[0091] The above reconstruction loss value can be calculated by using the L2 norm error method. Specifically, the calculation formula for calculating the reconstruction loss value by using the L2 norm error method can be as follows:

[0092] L(θ2)=‖x ′ -x‖2

[0093] Among them, L(θ2) represents the loss value of the reconstruction model, that is, the reconstruction loss value.

[0094] The election denoising module in the above-mentioned deep multi-fusion autoencoder model is used to regularly select the most normal traffic from the normal traffic set and the most abnormal traffic from the abnormal traffic set.

[0095] The normal traffic set includes normal traffic, and the normal traffic is the traffic determined as normal traffic by the fusion judgment module; the abnormal traffic set includes abnormal traffic, and the abnormal traffic is the traffic determined as abnormal traffic by the fusion judgment module. The most normal traffic is the most normal traffic in the normal traffic set; the most abnormal traffic is the most abnormal traffic in the abnormal traffic set.

[0096] Reference Figure 2 The figure is a flow chart of a method for abnormality detection provided by an embodiment of the present application, the method comprising:

[0097] S201, based on the pre-trained model, obtain a normal traffic set and an abnormal traffic set.

[0098] Before obtaining a normal traffic set and an abnormal traffic set based on a pre-trained model, the pre-trained model needs to be obtained first.

[0099] Optionally, the acquisition process of the pre-trained model (i.e., the pre-training process of the deep multi-fusion autoencoder model) can be as follows: Figure 3 As shown in steps S301-S308:

[0100] S301, obtaining the traffic to be trained, and preprocessing the traffic to be trained to obtain input data.

[0101] The traffic to be trained is normal traffic. The traffic to be trained can also be normal traffic that is easy to obtain, so as to facilitate better pre-training. The pre-processing method of the traffic to be trained can be the aforementioned Figure 1 The preprocessing method of the preprocessing module in will not be repeated here.

[0102] The above-mentioned traffic to be trained can be obtained in the form of data packets.

[0103] Optionally, when the traffic to be trained is obtained in the form of data packets, the preprocessing method of the traffic to be trained may also include the following methods:

[0104] Specifically, the acquired data packet (pcap) is first divided into session controls (session) according to five-tuples, and then a fixed-size length is cut for each session to obtain a binary fragment, which is then normalized to obtain preprocessed input data to improve the data quality of the input data, thereby helping to improve the performance of the model pre-trained based on the input data.

[0105] S302, input the input data into the encoder in the deep multi-fusion autoencoder model to obtain high-dimensional features.

[0106] In an embodiment of the present application, in the process of inputting input data into the encoder in the deep multi-fusion autoencoder model, the input data can be input into the encoder in a batch training manner to obtain high-dimensional features. The high-dimensional features can be 1×n features; n is a positive integer.

[0107] S303, processing the high-dimensional features through the classification model in the deep multi-fusion autoencoder model to obtain a classification result.

[0108] S304: Perform loss calculation based on the classification result and the input data to obtain a classification loss value.

[0109] In an embodiment of the present application, a cross entropy loss function may be used to calculate the classification loss value.

[0110] However, since the input data in the pre-training process only includes normal traffic, when the classification model uses the cross entropy loss function to calculate the classification loss value, the cross entropy loss function is as follows:

[0111]

[0112] Among them, L(θ1) represents the loss value of the classification model, that is, the classification loss value; y represents the label corresponding to the i-th data in the input data; y ′ It represents the predicted value corresponding to the i-th data in the input data, that is, the classification result corresponding to the i-th data; m represents the number of samples (that is, the number of traffic).

[0113] Since the input data only includes normal traffic, y represents normal traffic. For example, normal traffic is represented by 1 and abnormal traffic is represented by 0, then y is 1.

[0114] S305, processing the high-dimensional features through the reconstruction model in the deep multi-fusion autoencoder model to obtain a reconstruction result.

[0115] In order to improve the accuracy of the reconstruction results, the high-dimensional features are not directly reconstructed, but first converted into second features through sparse transformation, and then the second features are decoded and reconstructed.

[0116] Therefore, the specific process of processing high-dimensional features through the reconstruction model in the deep multi-fusion autoencoder model to obtain the reconstruction result can be shown as follows:

[0117] First, the high-dimensional features are sparsely processed by the intercept function in the reconstruction model to obtain the first feature, so as to filter out the redundant information features. Then, the first feature is converted into the second feature by the transformation matrix in the reconstruction model, so as to convert the dimension of the first feature into a preset dimension. The preset dimension can be 1×n, that is, the dimension of the second feature is 1×n. Finally, the high-dimensional features are reconstructed by the reconstruction model to obtain the reconstruction result.

[0118] In an embodiment of the present application, the intercept function may be a rectified linear unit (Rectified Linear Unit, abbreviated as relu) function. The transformation matrix may be an n×n matrix of the same dimension as the high-dimensional feature. The specific method of the sparse processing and dimensional conversion may be as shown in the following formula:

[0119]

[0120] z ′ =temp*N

[0121] Among them, z i represents the i-th high-dimensional feature; λ represents the intercept of the relu function, which is a preset filtering threshold to filter out redundant information features; δ represents the deviation value, which is a very small deviation value used to prevent calculation problems caused by the denominator being 0; temp i represents the intermediate value corresponding to the i-th high-dimensional feature; the dimension of temp is 1×n; N represents the change matrix, the dimension of the change matrix is ​​n×n; z ′ Indicates the second feature.

[0122] S306, performing loss calculation based on the reconstruction result and the input data to obtain a reconstruction loss value.

[0123] S307, merging the classification loss value and the reconstruction loss value to obtain a combined loss value.

[0124] Specifically, the following formula can be used to combine the classification loss value and the reconstruction loss value.

[0125] L(θ1, θ2)=αL(θ1)+(1-α)L(θ2)

[0126] Among them, L(θ1, θ2) represents the combined loss value; α represents the combined weight.

[0127] Optionally, in order to achieve the purpose of making the reconstruction loss value more important than the classification loss value, the merging weight (i.e., α) can be set to 0.3 so that the weight of the reconstruction loss value is greater than the weight of the classification loss value, but it is not limited to this, and the merging weight (i.e., α) can be adjusted according to the specific application scenario.

[0128] S308, pre-training a deep multi-fusion autoencoder model based on the combined loss value to obtain a pre-trained model.

[0129] The above process of pre-training the deep multi-fusion autoencoder model based on the merged loss value can be pre-trained by using the method of pre-training the model based on the loss value in the prior art, which will not be repeated here.

[0130] However, it should be noted that in the process of pre-training the deep multi-fusion autoencoder model based on the merged loss value to obtain the pre-trained model, the pre-training is performed on the encoder, classification model and reconstruction model in the deep multi-fusion autoencoder model, and the fusion judgment module and election denoising module in the deep multi-fusion autoencoder model are not pre-trained.

[0131] Therefore, the optimal pre-training model at the current stage is obtained through the above steps S301-S308, so as to obtain the combination of normal traffic set and abnormal traffic based on the pre-training model.

[0132] The specific process of obtaining the normal traffic set and the abnormal traffic set based on the pre-training model can be as follows:

[0133] First, the original traffic to be used is obtained, and then the original traffic to be used is preprocessed to obtain the traffic to be used. The preprocessing process of the original traffic to be used is consistent with the preprocessing method of the traffic to be trained in the above step S301, and will not be repeated here.

[0134] Then, the traffic to be used is input into the pre-trained model to obtain the first result and the second result. The first result is the abnormal result output by the classification model in the pre-trained model; the second result is the abnormal result output by the reconstruction model in the pre-trained model. Then, the first result and the second result are judged by the fusion judgment method to obtain the third result.

[0135] If the third result is determined to be a normal result, the traffic to be used is added to the normal traffic set.

[0136] If the third result is determined to be an abnormal result, the traffic to be used is added to the abnormal traffic set.

[0137] Through the above method, the abnormal results output by the classification model (i.e., the first result) and the abnormal results output by the reconstruction model (i.e., the second result) are corrected, so that the credibility of the third result obtained is higher, thereby further improving the credibility of the normal traffic set and the abnormal traffic set obtained based on the third result.

[0138] The above-mentioned fusion judgment method can be the fusion judgment module in the aforementioned deep multi-fusion autoencoder model. Therefore, the specific process of the above-mentioned fusion judgment method to judge the first result and the second result is consistent with the specific process of the fusion judgment module in the aforementioned deep multi-fusion autoencoder model for judging the abnormal results output by the classification model and the abnormal results output by the reconstruction model, and will not be repeated here.

[0139] S202, by means of election noise reduction, respectively processing the normal traffic set and the abnormal traffic set to obtain the most normal traffic and the most abnormal traffic.

[0140] Due to the means of forging abnormal traffic, it is inevitable that there will be a small amount of abnormal traffic in the normal traffic area (i.e., normal traffic set) and a small amount of normal traffic in the abnormal traffic area (i.e., abnormal traffic set). In other words, the normal traffic in the normal traffic set may be pseudo-normal traffic, and the abnormal traffic in the abnormal traffic set may be pseudo-abnormal traffic. Figure 4 As shown, there is pseudo-normal traffic in the normal traffic set, and there is pseudo-abnormal traffic in the abnormal traffic set; among them, the dotted circle in the normal traffic set represents pseudo-normal traffic (i.e. abnormal traffic), and the solid circle represents normal traffic; the dotted circle in the abnormal traffic set represents abnormal traffic, and the solid circle represents pseudo-abnormal traffic (i.e. normal traffic).

[0141] To solve this problem, the normal traffic set and the abnormal traffic set can be processed separately by electing noise reduction, and the most normal normal traffic in the normal traffic set (i.e. the most normal traffic) and the most abnormal abnormal traffic in the abnormal traffic set (i.e. the most abnormal traffic) are obtained, thereby solving the problem of blurred boundaries between normal traffic and abnormal traffic. For example, Figure 4 As shown, the most normal traffic (i.e., circle 1 in the normal traffic set) can be selected from the normal traffic set through the election noise reduction method, and the most abnormal traffic (i.e., circle 1' in the abnormal traffic set) can be selected from the abnormal traffic set.

[0142] Specifically, after obtaining the normal flow set and the abnormal flow set in step S101, the normal flow set is processed by the election noise reduction method at first intervals to obtain the most normal flow, and the abnormal flow set is processed by the election noise reduction method at second intervals to obtain the most abnormal flow, so that the most normal flow is selected from the normal flow set and the most abnormal flow is selected from the abnormal flow set by the election noise reduction method regularly, so that the boundary between normal flow and abnormal flow is not blurred. In addition, after selecting the most normal flow from the normal flow set, the normal flow set is cleared, and after selecting the most abnormal flow from the abnormal flow set, the abnormal flow set is cleared, so as to continue to obtain a new round of normal flow sets and abnormal flow sets in the future, so as to continuously obtain the most normal flow and the most abnormal flow, and solve the problem of manually obtaining samples.

[0143] The first time and the second time may be consistent or inconsistent.

[0144] The above-mentioned election denoising method may be the election denoising module in the aforementioned deep multi-fusion autoencoder model.

[0145] The specific process of processing the normal traffic set through the election noise reduction method to obtain the most normal traffic can be shown as follows:

[0146] First, a first result and a second result corresponding to each normal flow in the normal flow set are obtained.

[0147] Since each normal flow in the normal flow set comes from the flow to be used, the abnormal result output by the classification model in the pre-trained model for each normal flow in the normal flow set is the aforementioned first result, and the abnormal result output by the reconstruction model in the pre-trained model is the aforementioned second result.

[0148] Therefore, when obtaining the abnormal results output by the classification model and the abnormal results output by the reconstruction model corresponding to each normal flow in the normal flow set, the first result and the second result corresponding to each normal flow in the normal flow set are obtained.

[0149] Then, the first result and the corresponding second result of each normal flow in the normal flow set are jointly calculated to obtain the joint value corresponding to each normal flow in the normal flow set. Then, the obtained joint values ​​are sorted in ascending order, and the normal flow corresponding to the first joint value in the sorted joint values ​​is selected as the most normal flow, so that the selected most normal flow has the highest credibility as normal, which solves the problem of blurred boundaries between normal flow and abnormal flow.

[0150] In order to further make the most normal traffic selected as a more accurate normal traffic, the first result and the corresponding second result of each normal traffic in the normal traffic set are jointly calculated to obtain the joint value corresponding to each normal traffic in the normal traffic set, which can be calculated in the following way:

[0151] First, in the normal traffic set, the normal traffic that meets the ignoring condition is used as the ignored traffic, so as to determine the traffic that needs to be ignored. The ignored traffic is the traffic that needs to be ignored. The ignoring condition can be that the first result is between the first threshold and the second threshold or the second result is between the third threshold and the fourth threshold. Then, for other normal traffic in the normal traffic set except the ignored traffic, the first result and the second result are jointly calculated to obtain the joint value corresponding to the corresponding traffic. In this way, the normal traffic between the first threshold and the second threshold of the first result is ignored, and the normal traffic between the third threshold and the fourth threshold of the second result is ignored, and then the normal traffic in the normal traffic set is preliminarily screened, and then the normal traffic in the normal traffic set except the ignored traffic is jointly calculated, so as to further improve the credibility of the most normal traffic selected.

[0152] The first threshold and the second threshold may be probability thresholds, and the third threshold and the fourth threshold may be score thresholds.

[0153] Exemplarily, the normal traffic included in the normal traffic set is C1 traffic, C2 traffic, C3 traffic, C4 traffic and C5 traffic. The first result corresponding to C1 traffic is 0.1, and the corresponding second result is 20; the first result corresponding to C2 traffic is 0.05, and the corresponding second result is 7; the first result corresponding to C3 traffic is 0.8, and the corresponding second result is 80; the first result corresponding to C4 traffic is 0.9, and the corresponding second result is 180; the first result corresponding to C5 traffic is 0.6, and the corresponding second result is 101; the first probability threshold is 0.3, the second probability threshold is 0.7, the first score threshold is 12, and the second score threshold is 100.

[0154] Therefore, the ignored traffic is C1 traffic, C3 traffic, and C5 traffic. Then, the first result and the second result corresponding to the C2 traffic are jointly calculated to obtain a joint value corresponding to the C2 traffic, and the first result and the second result corresponding to the C4 traffic are jointly calculated to obtain a joint value corresponding to the C4 traffic.

[0155] The above-mentioned joint calculation of the first result and the corresponding second result can be calculated by logarithmic conversion and summation, but is not limited thereto.

[0156] Specifically, the first result is first logarithmically transformed to obtain a first logarithmic result. The second result is then logarithmically transformed to obtain a second logarithmic result. The first logarithmic result and the second logarithmic result are then summed, and the sum obtained is the joint value, so that the calculated joint value is more accurate, which helps to make the most normal traffic selected based on the joint value more credible.

[0157] The specific process of processing the abnormal traffic set through the election noise reduction method to obtain the most abnormal traffic is consistent with the specific process of processing the normal traffic set through the election noise reduction method to obtain the most normal traffic, and will not be repeated here.

[0158] However, it should be noted that in the process of processing the abnormal traffic set through the election noise reduction method to obtain the most abnormal traffic, after obtaining the joint value, the joint value is sorted in descending order, and then the abnormal traffic corresponding to the first joint value after sorting is selected as the most abnormal traffic, so as to select the most abnormal abnormal traffic.

[0159] For example, Figure 4 As shown in , the joint values ​​corresponding to the normal traffic in the normal traffic set are sorted in ascending order. The order represented by this sorting is as follows Figure 4 As shown in the circled numbers in the normal traffic set in , the normal traffic corresponding to the first joint value is circle 1; the joint values ​​corresponding to the abnormal traffic in the abnormal traffic set are sorted in descending order, and the order represented by the sorting is as follows Figure 4 As shown in the circled numbers in the abnormal traffic set, the abnormal traffic corresponding to the first joint value is circle 1'.

[0160] S203, obtaining a target model based on the most normal traffic and the most abnormal traffic, and performing abnormality detection on the traffic through the target model.

[0161] After obtaining the most normal flow and the most abnormal flow in step S102, relatively reliable normal flow and abnormal flow can be obtained, so that the data completeness is gradually enhanced, and then the most normal flow and the most abnormal flow are fed back to the pre-training model for enhanced training every third time to obtain the target model, so as to detect abnormal flow through the target model. In this way, the model has continuous self-learning ability, which improves the timeliness and accuracy of the detection of increasingly updated abnormal flow patterns.

[0162] Furthermore, by feeding back the most normal traffic and the most abnormal traffic to the pre-training model every third time, the number of the most normal traffic and the most abnormal traffic is not limited to one, thereby facilitating enhanced training of the pre-training model.

[0163] The training method of the above-mentioned enhanced training is consistent with the pre-training method in the aforementioned step S101, and will not be repeated here.

[0164] However, it should be noted that, after the most normal traffic and the most abnormal traffic are fed back to the pre-training model, the new training data set contains abnormal traffic in addition to normal traffic, so that the label is not limited to normal traffic. In the process of using the cross entropy loss function to calculate the classification loss value in the aforementioned step S304, the cross entropy loss function can be a complete cross entropy loss function, which is as follows:

[0165]

[0166] The specific process of detecting traffic anomalies through the target model can be as follows:

[0167] After obtaining the original flow to be detected, the original flow to be detected is preprocessed to obtain the flow to be detected. The preprocessing process of the original flow to be detected is the same as the preprocessing process of the flow to be trained in the aforementioned step S301, and will not be repeated here.

[0168] Then, the flow to be detected is input into the target model, and the target model processes the flow to be detected to obtain a detection result, which includes the result that the flow to be detected is abnormal flow or the result that the flow to be detected is normal flow, thereby achieving accurate abnormal detection of flow.

[0169] In addition, since the most normal traffic and the most abnormal traffic will be fed back to the pre-training model for enhanced training every third period of time, the target model will also be updated after a period of time. At this time, the updated target model will be used to detect traffic anomalies.

[0170] In summary, the anomaly detection method proposed in this application learns the characteristics of normal traffic through the deep autoencoder (i.e., the encoder and the reconstruction model) in the deep multi-fusion autoencoder, so that it can reconstruct the characteristics of normal traffic. And in the encoder of the deep multi-fusion autoencoder model, the classification judgment (i.e., the classification model) is branched, so that pre-training of only normal traffic (such as pre-supervised training) and post-correction training of normal traffic and abnormal traffic are performed on high-dimensional features, so that the data completeness is gradually enhanced. In addition, the anomaly detection method provided in the embodiment of the present application is based on incomplete data, does not rely on the detection results of the pre-training model, and as the model is used for a longer time, the completeness of the data is gradually enhanced.

[0171] In addition, the high-dimensional features between the encoder and decoder of the deep multi-fusion autoencoder model are sparsely transformed (that is, the high-dimensional features are sparsely transformed in the reconstruction model), filtering out the harmful effects of redundant information features.

[0172] Then, a fusion judgment method (i.e., the fusion judgment module in the deep multi-fusion autoencoder model) is used to combine the side branch classification results (i.e., the abnormal results output by the classification model) and the reconstruction results (i.e., the abnormal results output by the reconstruction model) to judge the preliminary boundary between abnormal traffic and normal traffic, so as to obtain a set of normal traffic that is preliminarily divided into normal traffic and a set of abnormal traffic that is preliminarily divided into abnormal traffic.

[0173] The election denoising method (i.e., the election denoising module in the deep multi-fusion autoencoder model) is then used to further process the most normal normal traffic in the normal traffic set (i.e., the most normal traffic) and the most abnormal traffic in the abnormal traffic set (i.e., the most abnormal traffic), thereby solving the problem of blurred boundaries between normal traffic and abnormal traffic, and helping to improve the accuracy of traffic anomaly detection.

[0174] Next, the obtained maximum values ​​(i.e., the most normal flow and the most abnormal flow) are regularly fed back to the pre-trained model for enhanced training. That is, the obtained multiple most normal flows and the most abnormal flows are fed back to the pre-trained model for enhanced training, thereby realizing the continuous self-learning ability of the model and improving the timeliness and accuracy of the detection of increasingly updated abnormal flow patterns.

[0175] In addition, through the anomaly detection method provided in the embodiment of the present application, positive and negative flow samples (ie, normal flow and abnormal flow) are continuously and automatically added, providing abnormal flow data capabilities.

[0176] The technical solution of this application is further explained below in conjunction with a specific application process.

[0177] like Figure 5 The figure shows a processing diagram of the anomaly detection method. First, in the preprocessing unit, normal traffic is obtained and preprocessed to obtain input data, and then the input data is transmitted to the pretraining unit.

[0178] In the pre-training unit, the input data is processed by the encoder in the deep multi-fusion autoencoder model to obtain high-dimensional features. Then, the high-dimensional features are processed by the classification model in the deep multi-fusion autoencoder model to obtain the classification result. The loss is calculated based on the classification result and the input data to obtain the classification loss value. At the same time, the high-dimensional features are processed by the reconstruction model in the deep multi-fusion autoencoder model to obtain the reconstruction result. The loss is calculated based on the reconstruction result and the input data to obtain the reconstruction loss value. Then, the classification loss value and the reconstruction loss value are merged to obtain the merged loss value. Then, the deep multi-fusion autoencoder model is updated by the merged loss value to obtain the pre-trained model. The pre-trained model is transmitted to the traffic division unit.

[0179] In the traffic division unit, the received traffic to be used is processed by the received pre-trained model through the model processing module to obtain the first result and the second result. Then, the first result and the second result are judged by the fusion judgment module in the deep multi-fusion autoencoder model to obtain the third result. If the third result indicates that the traffic to be used is abnormal traffic, the traffic to be used is added to the abnormal traffic set. If the third result indicates that the traffic to be used is normal traffic, the traffic to be used is added to the normal traffic set. After obtaining the normal traffic set and the abnormal traffic set, the election noise reduction module in the deep multi-fusion autoencoder model is used to regularly select the most normal traffic from the normal traffic set and the most abnormal traffic from the abnormal traffic set. And the most abnormal traffic and the most normal traffic are transmitted to the feedback unit.

[0180] In the feedback unit, the most normal traffic and the most abnormal traffic are regularly fed back to the pre-training model for enhanced training to obtain the target model, thereby performing traffic anomaly detection based on the target model.

[0181] Based on the same inventive concept, an abnormality detection device is also provided in the embodiment of the present application, such as Figure 6 The figure is a schematic diagram of the structure of an abnormality detection device provided by the present application, the device comprising:

[0182] The first division module 601 is used to obtain a normal traffic set and an abnormal traffic set based on a pre-trained model;

[0183] The second division module 602 is used to process the normal traffic set and the abnormal traffic set respectively by electing a noise reduction method to obtain the most normal traffic and the most abnormal traffic; wherein the most normal traffic is the most normal normal traffic in the normal traffic set; and the most abnormal traffic is the most abnormal abnormal traffic in the abnormal traffic set;

[0184] The processing module 603 is used to obtain a target model based on the most normal traffic and the most abnormal traffic, and perform abnormality detection on the traffic through the target model.

[0185] In a possible implementation, the device further includes a pre-training module, which is used to obtain the traffic to be trained and pre-process the normal traffic to obtain input data; wherein the traffic to be trained is normal traffic; the input data is input into the encoder in the deep multi-fusion autoencoder model to obtain high-dimensional features; the high-dimensional features are processed by the classification model in the deep multi-fusion autoencoder model to obtain a classification result; and a loss is calculated based on the classification result and the input data to obtain a classification loss value; and the high-dimensional features are processed by the reconstruction model in the deep multi-fusion autoencoder model to obtain a reconstruction result; and a loss is calculated based on the reconstruction result and the input data to obtain a reconstruction loss value; the classification loss value and the reconstruction loss value are merged to obtain a merged loss value; based on the merged loss value, the deep multi-fusion autoencoder model is updated to obtain the pre-training model.

[0186] In one possible implementation, the pre-training module is specifically used to perform sparse processing on the high-dimensional features through the intercept function in the reconstruction model in the deep multi-fusion autoencoder model to obtain a first feature; process the first feature through the transformation matrix in the reconstruction model to obtain a second feature; decode and reconstruct the second feature through the decoder in the reconstruction model to obtain the reconstruction result.

[0187] In one possible implementation, the first division module 601 is specifically used to input the traffic to be used into the pre-trained model to obtain a first result and a second result; wherein, the first result is an abnormal result output by the classification model in the pre-trained model, and the second result is an abnormal result output by the reconstruction model in the pre-trained model; the first result and the second result are subjected to an identical or identical judgment by a fusion judgment method to obtain a third result; if it is determined that the third result indicates that the traffic to be used is normal traffic, the traffic to be used is added to the normal traffic set; if it is determined that the third result indicates that the traffic to be used is abnormal traffic, the traffic to be used is added to the abnormal traffic set.

[0188] In a possible implementation, the first division module 601 is specifically used to obtain the third result indicating that the traffic to be used is normal traffic if the first result indicates that the traffic to be used is normal traffic and the second result indicates that the traffic to be used is normal traffic; if the first result indicates that the traffic to be used is abnormal traffic and the second result indicates that the traffic to be used is abnormal traffic, then obtain the third result indicating that the traffic to be used is abnormal traffic.

[0189] In a possible implementation, the second division module 602 is specifically used to process the normal traffic set every first time period through the election noise reduction method to obtain the most normal traffic and clear the normal traffic set; and to process the abnormal traffic set every second time period through the election noise reduction method to obtain the most abnormal traffic and clear the abnormal traffic set.

[0190] In one possible implementation, the second division module 602 is specifically used to respectively obtain the first result and the second result corresponding to each normal flow in the normal flow set, and respectively obtain the first result and the second result corresponding to each abnormal flow in the abnormal flow set; jointly calculate the first result and the corresponding second result to obtain a joint value corresponding to the corresponding flow; sort the joint values ​​corresponding to the normal flows in the normal flow set in ascending order, select the normal flow corresponding to the first joint value as the most normal flow, and sort the joint values ​​corresponding to the abnormal flows in the abnormal flow set in descending order, and select the abnormal flow corresponding to the first joint value as the most abnormal flow.

[0191] In one possible implementation, the second division module 602 is specifically used to treat the normal traffic and abnormal traffic in the normal traffic set and the abnormal traffic set that meet the ignoring condition as ignored traffic; wherein the ignoring condition is that the first result is between the first threshold and the second threshold or the second result is between the third threshold and the fourth threshold; for the traffic in the normal traffic set and the abnormal traffic set except the ignored traffic, the corresponding first result and the second result are jointly calculated to obtain the joint value corresponding to the corresponding traffic.

[0192] In a possible implementation, the processing module 603 is specifically configured to feed back the most normal flow and the most abnormal flow to the pre-training model every third time period for enhanced training to obtain the target model.

[0193] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application. The electronic device can realize the functions of the aforementioned abnormality detection device. Figure 7 , the electronic equipment includes:

[0194] At least one processor 701, and a memory 702 connected to the at least one processor 701. The specific connection medium between the processor 701 and the memory 702 is not limited in the embodiment of the present application. Figure 7 In the example, the processor 701 and the memory 702 are connected via a bus 700. Figure 7The connections between other components are shown in bold lines, and are not intended to be limiting. The bus 700 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 7 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 701 can also be called a controller, and there is no limitation on the name.

[0195] In the embodiment of the present application, the memory 702 stores instructions that can be executed by at least one processor 701. The at least one processor 701 can execute the anomaly detection method discussed above by executing the instructions stored in the memory 702. The processor 701 can implement Figure 6 The functions of each module in the device shown.

[0196] Among them, the processor 701 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 702 and calling data stored in the memory 702, the various functions of the device and process data, the device can be monitored as a whole.

[0197] In one possible design, the processor 701 may include one or more processing units, and the processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 701. In some embodiments, the processor 701 and the memory 702 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0198] The processor 701 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the anomaly detection method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0199] The memory 702 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 702 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (English: Random Access Memory, abbreviated as RAM), a static random access memory (English: Static Random Access Memory, abbreviated as SRAM), a programmable read-only memory (English: Programmable Read Only Memory, abbreviated as PROM), a read-only memory (English: Read Only Memory, abbreviated as ROM), an electrically erasable programmable read-only memory (English: Electrically Erasable Programmable Read-Only Memory, abbreviated as EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 702 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 702 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0200] By programming the processor 701, the code corresponding to the abnormality detection method described in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 2 The steps of the abnormality detection method of the embodiment shown are as follows: How to design and program the processor 701 is a technique well known to those skilled in the art and will not be described in detail here.

[0201] Based on the same inventive concept, an embodiment of the present application further provides a storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the anomaly detection method discussed above.

[0202] In some possible implementations, various aspects of the anomaly detection method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on an apparatus, the program code is used to enable the control device to execute the steps of the anomaly detection method according to various exemplary implementations of the present application described above in this specification.

[0203] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0204] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0205] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0206] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0207] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An anomaly detection method, characterized in that: include: Based on the pre-trained model, obtain the normal traffic set and the abnormal traffic set; By means of election noise reduction, the normal traffic set and the abnormal traffic set are processed respectively to obtain the most normal traffic and the most abnormal traffic; wherein the most normal traffic is the most normal normal traffic in the normal traffic set; and the most abnormal traffic is the most abnormal abnormal traffic in the abnormal traffic set; Based on the most normal flow and the most abnormal flow, a target model is obtained, and abnormality detection is performed on the flow through the target model.

2. The method according to claim 1, characterized in that Before obtaining the normal traffic set and the abnormal traffic set based on the pre-trained model, the method further includes: Acquire the flow to be trained, and pre-process the normal flow to obtain input data; wherein the flow to be trained is the normal flow; Inputting the input data into an encoder in a deep multi-fusion autoencoder model to obtain high-dimensional features; Processing the high-dimensional features through the classification model in the deep multi-fusion autoencoder model to obtain a classification result; and performing loss calculation based on the classification result and the input data to obtain a classification loss value; and The high-dimensional features are processed by a reconstruction model in the deep multi-fusion autoencoder model to obtain a reconstruction result; and a loss calculation is performed based on the reconstruction result and the input data to obtain a reconstruction loss value; Combining the classification loss value and the reconstruction loss value to obtain a combined loss value; Based on the combined loss value, the deep multi-fusion autoencoder model is updated to obtain the pre-trained model.

3. The method according to claim 2, characterized in that The high-dimensional features are processed by the reconstruction model in the deep multi-fusion autoencoder model to obtain a reconstruction result, including: Performing sparse processing on the high-dimensional feature by using an intercept function in the reconstruction model in the deep multi-fusion autoencoder model to obtain a first feature; Processing the first feature by using the transformation matrix in the reconstructed model to obtain a second feature; The second feature is decoded and reconstructed by a decoder in the reconstruction model to obtain the reconstruction result.

4. The method according to claim 1, characterized in that The obtaining of a normal traffic set and an abnormal traffic set based on a pre-trained model includes: Input the traffic to be used into the pre-trained model to obtain a first result and a second result; wherein the first result is an abnormal result output by the classification model in the pre-trained model, and the second result is an abnormal result output by the reconstruction model in the pre-trained model; Perform an identical or identical (OR) judgment on the first result and the second result by a fusion judgment method to obtain a third result; If it is determined that the third result indicates that the traffic to be used is normal traffic, adding the traffic to be used to the normal traffic set; If it is determined that the third result indicates that the traffic to be used is abnormal traffic, the traffic to be used is added to the abnormal traffic set.

5. The method according to claim 4, characterized in that The performing the same or same judgment on the first result and the second result by the fusion judgment method to obtain the third result includes: If the first result indicates that the flow rate to be used is normal flow rate, and the second result indicates that the flow rate to be used is normal flow rate, then the third result indicating that the flow rate to be used is normal flow rate is obtained; If the first result indicates that the traffic to be used is abnormal traffic, and the second result indicates that the traffic to be used is abnormal traffic, then the third result indicating that the traffic to be used is abnormal traffic is obtained.

6. The method according to claim 1, characterized in that The method of selecting noise reduction to process the normal traffic set and the abnormal traffic set respectively to obtain the most normal traffic and the most abnormal traffic includes: At first intervals, the normal traffic set is processed by the election noise reduction method to obtain the most normal traffic, and the normal traffic set is cleared; and At second intervals, the abnormal traffic set is processed by the election noise reduction method to obtain the most abnormal traffic, and the abnormal traffic set is cleared.

7. The method according to claim 1, characterized in that The method of selecting noise reduction to process the normal traffic set and the abnormal traffic set respectively to obtain the most normal traffic and the most abnormal traffic includes: Respectively obtain the first result and the second result corresponding to each normal flow in the normal flow set, and respectively obtain the first result and the second result corresponding to each abnormal flow in the abnormal flow set; The first result and the corresponding second result are jointly calculated to obtain a joint value corresponding to the corresponding flow rate; The joint values ​​corresponding to the normal traffic in the normal traffic set are sorted in ascending order, and the normal traffic corresponding to the first joint value is selected as the most normal traffic; and the joint values ​​corresponding to the abnormal traffic in the abnormal traffic set are sorted in descending order, and the abnormal traffic corresponding to the first joint value is selected as the most abnormal traffic.

8. The method according to claim 7, characterized in that The joint calculation of the first result and the corresponding second result to obtain a joint value corresponding to the corresponding flow includes: The normal traffic and abnormal traffic that meet the ignore condition in the normal traffic set and the abnormal traffic set are both regarded as ignore traffic; wherein the ignore condition is that the first result is between the first threshold and the second threshold or the second result is between the third threshold and the fourth threshold; For the traffic in the normal traffic set and the abnormal traffic set except the ignored traffic, a corresponding joint calculation of the first result and the second result is performed to obtain a joint value corresponding to the corresponding traffic.

9. The method according to claim 1, characterized in that The obtaining of a target model based on the most normal flow and the most abnormal flow includes: Every third time period, the most normal flow and the most abnormal flow are fed back to the pre-training model for enhanced training to obtain the target model.

10. An abnormality detection device, characterized in that: include: The first partitioning module is used to obtain a normal traffic set and an abnormal traffic set based on a pre-trained model; The second division module is used to process the normal traffic set and the abnormal traffic set respectively through the election noise reduction method to obtain the most normal traffic and the most abnormal traffic; wherein the most normal traffic is the most normal normal traffic in the normal traffic set; the most abnormal traffic is the most abnormal abnormal traffic in the abnormal traffic set; The processing module is used to obtain a target model based on the most normal flow and the most abnormal flow, and perform abnormality detection on the flow through the target model.

11. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to implement the method steps of any one of claims 1 to 9 when executing a computer program stored in the memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 9 are implemented.

Images