Training method and device for abnormal flow detection model, monitoring method and equipment

Abstract

The present application relates to the technical field of traffic detection, and discloses a training method and device, a monitoring method, and equipment for an abnormal traffic detection model. The method includes: obtaining a network traffic data set; the network traffic data set includes multiple network traffic features; according to the importance of each network traffic feature, the network traffic data set is subjected to feature redundancy elimination and an alternative feature data set is formed; the alternative feature data set is subjected to balancing processing and a target feature data set is formed; and the preset network model is trained using the target feature data set to obtain an abnormal traffic detection model. In this way, by screening the network traffic features by importance, only important network traffic features can be retained, so that the abnormal traffic detection model trained by the screened network traffic features is more targeted. At the same time, balancing processing can improve the detection recall rate of small sample network traffic feature types without increasing additional model complexity.

Description

Technical Field

[0001] The present application relates to the field of flow detection technology, and in particular to a training method and device, a monitoring method and equipment for an abnormal flow detection model. Background Art

[0002] As a carrier of information transmission and interaction in cyberspace, network traffic contains a lot of important information. With the continuous development of science and technology, network viruses and attack methods are becoming more and more complex, resulting in some abnormal network traffic, which may threaten network security. Therefore, how to automatically and accurately identify abnormal network traffic data is an urgent problem to be solved.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical elements or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0005] The embodiments of the present application provide a training method and apparatus, a monitoring method, and equipment for an abnormal traffic detection model to automatically and accurately identify abnormal network traffic.

[0006] An embodiment of the present application provides a training method for an abnormal traffic detection model, comprising: obtaining a network traffic data set; the network traffic data set includes multiple network traffic features; eliminating feature redundancy of the network traffic data set according to the importance of each of the network traffic features and forming an alternative feature data set; performing balancing processing on the alternative feature data set and forming a target feature data set; and using the target feature data set to train a preset network model to obtain an abnormal traffic detection model.

[0007] In the above-mentioned implementation, by screening the network traffic features by importance, the network traffic features can be streamlined, and only important network traffic features are retained, so that the abnormal traffic detection model trained by the screened network traffic features is more targeted. By balancing the alternative feature data sets and obtaining the target feature data sets, the problem of imbalanced data types in the alternative feature data sets can be reduced, so that the detection recall rate of small sample network traffic feature types can be improved without adding additional model complexity by training the model with the optimized alternative feature data sets. Thereby, the abnormal traffic detection model trained by the method of the present application can automatically and accurately identify abnormal network traffic. And the abnormal traffic detection model trained by the present application can process larger-scale data sets, with faster processing speed in practical applications and real-time response.

[0008] Furthermore, obtaining a network traffic data set includes: obtaining network raw data and a data source of each of the network raw data; extracting key information from each of the network raw data according to the data source; analyzing the correlation between each of the data sources, and fusing the correlation with each of the key information to obtain a number of network traffic features; each of the network traffic features constitutes the network traffic data set.

[0009] In the above implementation, by fusing the association relationship with each key information to obtain the network traffic feature, the information contained in the network traffic feature can be richer, so that the abnormal traffic detection model trained using the network traffic feature can be more accurate.

[0010] Furthermore, according to the importance of each of the network traffic features, feature redundancy of the network traffic data set is eliminated and an alternative feature data set is formed, including: step S11, inputting the network traffic data set into a preset first extreme tree model to obtain the importance score of each of the network traffic features; step S12, removing the network traffic feature with the lowest importance score in the network traffic data set to obtain a new network traffic data set; looping steps S11 to S12 until a preset condition is met, and the remaining network traffic features form the alternative feature data set; the preset condition includes: the number of the network traffic features in the alternative feature data set is less than or equal to a preset number.

[0011] In the above implementation, important network traffic features often have a greater impact on whether the predicted traffic is abnormal. By screening important network traffic features to form a candidate feature data set, the accuracy of the trained abnormal traffic detection model can be improved. At the same time, by screening important network traffic, the features used for training can be reduced, thereby reducing the resources and time required for model training.

[0012] Furthermore, according to the importance of each of the network traffic features, feature redundancy of the network traffic data set is eliminated and an alternative feature data set is formed, which also includes: generating shadow features with the same data distribution characteristics as the network traffic features; using a preset second extreme tree model to construct a number of decision trees, calculating the Gini index reduction of each of the network traffic features when a node of each decision tree is split, and selecting the network traffic feature with the largest Gini index reduction for splitting, and calculating the importance score of each of the network traffic features according to all the Gini index reductions of each of the network traffic features in the second extreme tree model; using a preset third extreme tree model to construct a number of decision trees, calculating the Gini index reduction of each of the shadow features when a node of each decision tree is split, and selecting the shadow feature with the largest Gini index reduction for splitting, and calculating the importance score of each of the shadow features according to all the Gini index reductions of each of the shadow features in the third extreme tree model; screening some of the network traffic features according to the network traffic features and the importance scores of the shadow features to form the alternative feature data set.

[0013] In the above implementation, the importance scores of network traffic features reflect their actual role in the model, while the importance scores of shadow features should be close to zero because shadow features do not contain any actual information. Therefore, by comparing the importance scores of network traffic features and shadow features, the significance of each network traffic feature can be evaluated, which is conducive to the accuracy of the abnormal traffic detection model obtained by subsequent training.

[0014] Furthermore, based on the importance scores of the network traffic features and the shadow features, some of the network traffic features are screened and the alternative feature data set is formed, including: obtaining the maximum value of the importance scores of all the shadow features; comparing the importance score of each network traffic feature with the maximum value, retaining the network traffic features with an importance score greater than the maximum value and forming the alternative feature data set.

[0015] In the above implementation, network traffic features that are significantly more important than shadow features are considered global features that contribute significantly to model prediction. Therefore, network traffic features with importance scores greater than the maximum value of shadow features are retained and form a candidate feature data set. The network traffic features that can be used for subsequent model training are all those that contribute significantly to model prediction, thereby improving the accuracy of the abnormal traffic detection model after training.

[0016] Furthermore, the candidate feature data sets are balanced and a target feature data set is formed, including: for each of the network traffic features: determining the distance between the network traffic feature and other network traffic features; determining the network traffic features whose distance is less than a preset threshold as neighboring samples; counting the types of each of the neighboring samples and the number of each type, and defining the type with the largest number as the dominant type; if the dominant type is consistent with the type of the network traffic feature, retaining the network traffic feature; if the dominant type is inconsistent with the type of the network traffic feature, deleting the network traffic feature or adjusting the preset weight of the network traffic feature, and the remaining network traffic features form the target feature data set.

[0017] In the above implementation, if the dominant type is different from the type of the network traffic feature, the network traffic feature may be a boundary point or a noise point, which will cause data imbalance. Therefore, by determining the dominant type corresponding to the network traffic feature and deleting the network traffic feature that is inconsistent with the dominant type, the problem of data type imbalance in the candidate feature data set can be reduced, so that the model trained using the optimized candidate feature data set can improve the detection recall rate of small sample network traffic feature types without increasing additional model complexity.

[0018] Furthermore, the preset network model is a multi-layer perceptron model; the preset network model is trained using the target feature data set to obtain an abnormal traffic detection model, including: inputting the target feature data set into the multi-layer perceptron model for training, and using a multi-head attention mechanism to perform weighted summation on the features in the target feature data set to obtain an output result; calculating the loss value of the output result according to a preset loss function; optimizing the weights and biases of each layer in the multi-layer perceptron model according to the loss value and a preset optimization algorithm until convergence or reaching a set number of iterations to obtain the abnormal traffic detection model.

[0019] In the above implementation, by using the multi-head attention mechanism to perform weighted summation on the features in the target feature data set, the importance of different features can be dynamically adjusted using the attention weights, thereby adaptively focusing on the network traffic features that are most critical for detecting abnormal traffic. Thus, the trained abnormal traffic detection model can be more accurate.

[0020] An embodiment of the present application provides an abnormal traffic monitoring method, including: obtaining the traffic to be detected; inputting the traffic to be detected into a preset abnormal traffic detection model to obtain whether the traffic to be detected is abnormal; the abnormal traffic detection model is obtained by the above-mentioned abnormal traffic detection model training method.

[0021] In the above implementation, abnormal traffic is detected by an abnormal traffic detection model, which can effectively process large-scale data sets, identify abnormal traffic at a fast speed, and respond in real time.

[0022] The embodiment of the present application provides a training device for an abnormal traffic detection model, including: a data set acquisition module, used to acquire a network traffic data set; the network traffic data set includes multiple network traffic features; a first optimization module, used to eliminate feature redundancy of the network traffic data set according to the importance of each of the network traffic features and form an alternative feature data set; a second optimization module, used to perform balancing processing on the alternative feature data set and form a target feature data set; a training module, used to train a preset network model using the target feature data set to obtain an abnormal traffic detection model.

[0023] An embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the above-mentioned abnormal traffic detection model training method or the above-mentioned abnormal traffic monitoring method.

[0024] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0026] Figure 1 is a schematic diagram of a training method for an abnormal traffic detection model provided in an embodiment of the present application;

[0027] Figure 2 is a schematic diagram of an abnormal flow monitoring method provided in an embodiment of the present application;

[0028] Figure 3 is a schematic diagram of a training device for an abnormal flow detection model provided in an embodiment of the present application;

[0029] Figure 4 is a schematic diagram of an abnormal flow monitoring device provided in an embodiment of the present application;

[0030] Figure 5 It is a schematic diagram of an electronic device provided in an embodiment of the present application.

[0031] Reference numerals:

[0032] 1: Data set acquisition module; 2: First optimization module; 3: Second optimization module; 4: Training module; 5: Traffic acquisition module; 6: Abnormal determination module; 7: Memory; 8: Communication interface; 9: Bus; 10: Processor. DETAILED DESCRIPTION

[0033] In order to be able to understand the features and technical contents of the embodiments of the present application in more detail, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0034] The terms "first", "second", etc. in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present application described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0035] Unless otherwise stated, the term "plurality" means two or more.

[0036] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0037] Embodiment 1

[0038] The present application embodiment provides a method for training an abnormal traffic detection model. Figure 1 As shown, Figure 1 A basic flow chart of the training method of the abnormal traffic detection model provided in the embodiment of the present application includes:

[0039] Step S101, obtaining a network traffic data set.

[0040] Among them, the network traffic dataset includes multiple network traffic features.

[0041] In some embodiments, step S101 may include: obtaining network raw data and the data source of each network raw data; extracting key information from each network raw data according to the data source; analyzing the correlation between each data source, and integrating the correlation with each key information to obtain a number of network traffic characteristics; each network traffic characteristic constitutes a network traffic data set.

[0042] In the above embodiment, the network raw data may be network traffic data. Exemplarily, the network traffic data includes at least one of: IP address (Internet Protocol address), port number, protocol type, data packet content, session duration, data packet size distribution, log information of different network layers, security device alarms, and status information of other network security assets. The network layers include, for example, application layer, transport layer, and network layer.

[0043] In an optional manner of the above embodiment, obtaining the original network data may be: obtaining the pre-processed original network data.

[0044] In the above optional methods, the network raw data can be preprocessed in the following ways:

[0045] Method 1: Unify the format or unit of raw data from each network.

[0046] It is understandable that data from different network devices or at different time points may have different formats or units. The format or unit of each network raw data can be identified, and the format or unit of each network raw data can be unified through data conversion or standardization operations to ensure the consistency of each network raw data in the data set. Exemplarily, if the network raw data has different timestamp formats, the network raw data is converted into a unified time format. The unified time format can be a pre-set time format.

[0047] Method 2: Obtain the data type of the original network data, search for the type to be converted corresponding to the data type in a preset type database; and convert the original network data according to the type to be converted.

[0048] Exemplarily, the text log is converted into a structured data table. The hexadecimal value is converted into a decimal value. Non-numeric data is encoded into numeric data, wherein the non-numeric data is, for example, protocol type, service type, etc. In this way, the captured network raw data is converted into the necessary data type, which can facilitate better processing and analysis by the machine learning algorithm.

[0049] Method three: Determine whether there are outliers in the original network data, and correct the original network data with outliers.

[0050] For example, statistical analysis and data mining techniques can be applied to identify and process outliers. For example, use box plots, Z scores or IQR (interquartile range) methods to process raw network data and identify abnormal data that deviates significantly from the normal range. Correct, delete or replace abnormal data to prevent the abnormal data from adversely affecting subsequent analysis. Outliers can also be processed using outlier detection algorithms to improve the accuracy of subsequent analysis.

[0051] Method 4: When the network original data is time series data, the network original data are sorted in chronological order, and the time intervals between the network original data are made the same.

[0052] It is understandable that since the time intervals of the original network data may be different, the original network data can be increased or decreased through techniques such as interpolation, data resampling, or time window aggregation to ensure the continuity and comparability of the time series data.

[0053] In an optional manner of the above embodiment, extracting key information from each network original data according to the data source can be: filtering out available data sources from the data source as alternative data sources; and extracting key information from the network original data corresponding to the alternative data sources.

[0054] In the above optional manner, selecting an available data source from the data source as an alternative data source may be: comparing the data source with a preset data source, and confirming the data source as an alternative data source when the data source is consistent with the preset data source.

[0055] The preset data sources can be set by engineers based on their experience. Examples of preset data sources include network traffic logs, security event management system (SIEM) records, firewall and intrusion detection system (IDS / IPS) alerts, application logs, system logs, user behavior data, etc.

[0056] In the above optional manner, extracting key information from the network original data corresponding to the alternative data source may be: inputting the network original data corresponding to the alternative data source into a preset key information extraction model to obtain the key information in the network original data.

[0057] Among them, the key information extraction model can be obtained in the following way: obtaining the original data of the sample network with the key information label, inputting the original data of the sample network with the key information label into a preset training model for training, and obtaining the key information extraction model.

[0058] In the above optional methods, specific parsing technologies may also be applied to the alternative data sources, such as text parsing and regular expressions, event correlation and behavior analysis, data parsing and natural language processing, etc., to extract key information.

[0059] Among them, key information refers to the core data or content extracted from the network raw data that is crucial to understanding and analyzing network traffic characteristics. Exemplarily, key information can be: structured data, such as tabular data in the data source. Unstructured data, such as important text content in the log file. Traffic statistics, such as statistical information extracted from network traffic logs. Security event information, such as relevant information extracted from security event management system (SIEM) records. User behavior data, such as access patterns extracted from user behavior data. Device alarm information, such as alarms extracted from firewalls and intrusion detection systems (IDS / IPS). Key information in application and system logs.

[0060] In an optional manner of the above embodiment, analyzing the association relationship between the data sources may be: sending the data sources to a preset analysis platform to obtain the association relationship between the data sources fed back by the analysis platform, wherein the analysis platform is used to annotate the association relationship between the data sources in response to the user's operation.

[0061] In another optional method of the above embodiment, analyzing the correlation relationship between each data source can be: obtaining key information corresponding to each data source; constructing features that can characterize the attributes of each key information; and using the features to perform calculations according to a preset statistical algorithm to obtain the correlation relationship between each data source.

[0062] Among them, constructing features that can characterize the attributes of each key information may include: aligning the time series of each key information, and constructing features for the key information after the time series alignment. In this way, by aligning the time series of data from all data sources, the relationship between different data sources can be analyzed within the same time frame.

[0063] It can be understood that the time series alignment of each key information can be to standardize all key information to the same time granularity, such as every minute or every hour. In addition, for missing time points, an interpolation method can be used to fill in the key information or mark it as missing.

[0064] Exemplarily, a set of descriptive features is created for each data source, and these features can represent the attributes of the data source. For example, for network traffic data, the total traffic volume, number of independent IPs, protocol distribution and other attributes can be included; for security event data, the event type distribution, severity distribution and other attributes can be included. It can be understood that the descriptive features can be expressed in the form of feature vectors, which contain various contents that can represent the attributes of the data source. Engineers can pre-set the attributes corresponding to each data source and construct descriptive features according to the pre-set attributes.

[0065] Among them, the correlation relationship between the data sources is obtained by calculating the features according to the preset statistical algorithm, which can be: when the feature is a numerical feature, the linear correlation between the two data sources corresponding to the two features is obtained by calculating the Pearson correlation coefficient between the two features. When the feature is a categorical feature, the chi-square test can be used to evaluate the degree of correlation between the data sources corresponding to the two features. Mutual information can also be used to capture the nonlinear relationship between the data sources corresponding to the two features.

[0066] In the above optional manner, time lag analysis can also be performed on each data source, by using a cross-correlation function to detect the time dependency between different data sources and identifying the time lag corresponding to the maximum correlation. In this way, some events may cause changes in other data sources, and time lag analysis may reveal the causal relationship between the data sources.

[0067] Exemplarily, the correlation between various data sources is analyzed, for example, network traffic data is correlated with security alerts to identify possible attack patterns or malicious activities.

[0068] In the above optional manner, after obtaining the association relationship between each data source, a graphical model can also be constructed to represent the relationship between the data sources, for example: by using the data source as a node, using the correlation strength as the weight of the edge, and using the time lag information to determine the direction of the edge in the absence of conflict, thereby obtaining a graphical model. Apply a community detection algorithm on the constructed graphical model: use a Louvain algorithm (a community discovery algorithm) or a label propagation algorithm to identify closely connected groups of data sources, which may represent sets of data sources that are highly correlated with each other. Perform abnormal relationship detection on the constructed graphical model, determine abnormal relationships by identifying edges with abnormal correlation strengths, and / or, determine abnormal relationships by identifying connections that do not conform to the overall pattern. Among them, the abnormal correlation strength refers to a correlation that is higher than the first preset correlation or lower than the second preset correlation. The first preset correlation is greater than the second preset correlation.

[0069] In the above optional approach, considering that the relationship between data sources may change over time, a sliding window method can also be used to repeat the above analysis, track changes in relationship strength and pattern, and identify relationship mutation points, which may indicate the occurrence of important events.

[0070] In the above optional manner, a relationship summary can also be generated based on the above obtained relationship. The relationship summary can include the strongest relationship, key groups, abnormal relationships, etc., and automatically classify the identified relationships, such as "strong positive correlation", "weak negative correlation", "periodic correlation", etc. In this way, all analysis results can be integrated to form a comprehensive understanding of the relationship between data sources.

[0071] In the above embodiment, the association relationship is fused with each key information to obtain a number of network traffic characteristics, which can be achieved by integrating key information from different sources using data fusion technology.

[0072] It can be understood that the network traffic feature is a comprehensive representation, which includes key information extracted from a single or multiple related network raw data, as well as the association relationship between the data source to which these key information belongs and other data sources. Its expression can be a multi-dimensional matrix or tensor, and each dimension can represent different features or attributes. That is, the network traffic feature can be a comprehensive representation including a single key information and the association relationship between the data source to which the key information belongs and other data sources. The network traffic feature can also be a comprehensive representation including multiple related key information and the association relationship between the data source to which the multiple key information belongs and other data sources.

[0073] Exemplarily, it can be considered that there is a correlation between the network raw data corresponding to the linearly related data sources.

[0074] Exemplarily, network traffic characteristics may include information content, time dimension, and association strength. Information content: may include the core content of the original key information, as well as the derived information obtained from the association relationship analysis. Time dimension: may include time series information, reflecting the data status at different time points. Association strength: may include quantitative indicators representing the association strength between different data sources. Data fusion technology, such as data alignment, data fusion framework, and entity resolution. In the entire data fusion process, steps such as time synchronization, data format unification, entity recognition, and relationship construction may be involved.

[0075] In the above embodiment, after analyzing the correlation between various data sources and fusing the correlation with various key information, it also includes: constructing a comprehensive network traffic data view based on the fused data. In this way, the constructed network traffic data view will contain rich context information, such as user identity, device type, application behavior, network protocol usage, etc., as well as the correlation between them, etc., which can facilitate users to intuitively understand network traffic data.

[0076] For example, in order to improve the accuracy of subsequent feature extraction and analysis, the data view can be further enriched by introducing contextual information such as threat intelligence, historical security events, and user behavior benchmarks.

[0077] Step S102, eliminating feature redundancy of the network traffic data set according to the importance of each network traffic feature and forming an alternative feature data set.

[0078] In some embodiments, step S102 may include: step S11, inputting the network traffic data set into a preset first extreme tree model to obtain the importance score of each network traffic feature; step S12, removing the network traffic feature with the lowest importance score in the network traffic data set to obtain a new network traffic data set; looping steps S11 to S12 until the preset conditions are met, and the remaining network traffic features form an alternative feature data set; the preset conditions include: the number of network traffic features in the alternative feature data set is less than or equal to the preset number.

[0079] In the above embodiment, inputting the network traffic data set into the preset first extreme tree model may be: processing the network traffic data set using a PCA (principal component analysis) algorithm or an autoencoder algorithm, and inputting the processed network traffic data set into the preset first extreme tree model. In this way, the PCA algorithm or the autoencoder algorithm can reduce the dimension of the network traffic features in the network traffic data set, map the high-dimensional feature space to a lower-dimensional space, and retain key information.

[0080] In the above embodiment, the preset condition may also be that the model performance meets a specific standard. The model performance may be at least one of accuracy, recall, and F1 score. Exemplarily, the model performance meeting the specific standard may be that the accuracy reaches a set accuracy. The model performance meeting the specific standard may also be that the recall reaches a set recall. The model performance meeting the specific standard may also be that the F1 score reaches a set F1 score.

[0081] The network traffic data set can be divided into a training set and a test set, and the first limit tree model can be tested using the test set to obtain true positive examples, true negative examples, false positive examples, and false negative examples. Among them, accuracy = (true positive examples + true negative examples) / (true positive examples + true negative examples + false positive examples + false negative examples). Recall = true positive examples / (true positive examples + false negative examples). F1 score = 2 * (precision * recall rate) / (precision + recall rate). Among them, true positive examples are the number of samples correctly predicted as positive by the model. True negative examples are the number of samples correctly predicted as negative by the model. False positive examples are the number of samples that the model incorrectly predicts as positive from negative classes. False negative examples are the number of samples that the model incorrectly predicts as negative from positive classes. "*" is multiplication.

[0082] In some embodiments, step S102 may also include: generating shadow features with the same data distribution characteristics as network traffic features; constructing several decision trees using a preset second extreme tree model, calculating the Gini index reduction of each network traffic feature when a node of each decision tree is split, and selecting the network traffic feature with the largest Gini index reduction for splitting, and calculating the importance score of each network traffic feature based on all the Gini index reductions of each network traffic feature in the second extreme tree model; constructing several decision trees using a preset third extreme tree model, calculating the Gini index reduction of each shadow feature when a node of each decision tree is split, and selecting the shadow feature with the largest Gini index reduction for splitting, and calculating the importance score of each shadow feature based on all the Gini index reductions of each shadow feature in the third extreme tree model; screening some network traffic features based on the importance scores of network traffic features and shadow features to form an alternative feature data set.

[0083] The data distribution characteristics may refer to mean, median, mode, skewness coefficient, etc. For example, a shadow feature having the same data distribution characteristics as the network traffic feature may be generated by a random measurement algorithm.

[0084] For example, the Gini index can be calculated as follows: in, is the proportion of samples of the i-th class in the data set, and m is the total number of classes. The Gini index reduction can be calculated as follows:

[0085] ;

[0086] Where D is the data set of the current node, is the subset obtained after splitting, yes The number of samples, is the total number of samples in the current node dataset.

[0087] It can be understood that for a data set containing mixed categories, a higher Gini index indicates that the purity of the data set is lower; when all samples in the data set belong to the same category, the Gini index is the lowest, indicating that the purity of the data set is the highest. In the process of feature selection and decision tree construction, the reduction of the Gini index is an important indicator. The reduction of the Gini index helps evaluate the effect of splitting and can guide the growth direction of the decision tree to build an efficient classification model.

[0088] In the above embodiment, filtering some network traffic features according to the importance scores of network traffic features and shadow features and forming an alternative feature data set may include: obtaining the maximum value of the importance scores of all shadow features; comparing the importance score of each network traffic feature with the maximum value, and retaining the network traffic features with an importance score greater than the maximum value to form an alternative feature data set. In this way, since the shadow feature does not contain any actual information, the importance of the shadow feature should be close to zero. Therefore, the network traffic feature with an importance score greater than the maximum value of the importance score of the shadow feature is more important than other network traffic features, so that more important network traffic features can be selected.

[0089] Step S103: performing equalization processing on the candidate feature data sets to form a target feature data set.

[0090] In some embodiments, step S103 may include: for each network traffic feature: determining the distance between the network traffic feature and other network traffic features; determining the network traffic features whose distance is less than a preset threshold as neighboring samples; counting the types of each neighboring sample and the number of each type, and defining the type with the largest number as the dominant type; if the dominant type is consistent with the type of the network traffic feature, retaining the network traffic feature; if the dominant type is inconsistent with the type of the network traffic feature, deleting the network traffic feature or adjusting the preset weight of the network traffic feature, and the remaining network traffic features form the target feature data set.

[0091] In the above embodiment, the distance between two network traffic features may be determined by Euclidean distance, Chebyshev distance, or the like.

[0092] It is understandable that before calculating the distance for each network traffic feature, each network traffic feature may be re-encoded by one-hot encoding, and the distance is calculated using the encoded network traffic features.

[0093] Step S104, using the target feature data set to train the preset network model to obtain an abnormal traffic detection model.

[0094] In some embodiments, the preset network model is a multi-layer perceptron model; step S104 may include: inputting the target feature data set into the multi-layer perceptron model for training, and using a multi-head attention mechanism to perform weighted summation on the features in the target feature data set to obtain an output result; calculating the loss value of the output result according to a preset loss function; optimizing the weights and biases of each layer in the multi-layer perceptron model according to the loss value and a preset optimization algorithm until convergence or reaching a set number of iterations to obtain an abnormal traffic detection model.

[0095] Among them, loss function, for example: cross entropy loss function. Preset optimization algorithm, for example: gradient descent algorithm, Adam algorithm (optimization algorithm combining momentum method and adaptive learning rate), etc.

[0096] For example, a multi-layer perceptron is composed of multiple hidden layers, each of which contains several neurons. Each neuron receives input from the previous layer, performs a nonlinear transformation on the input data through an activation function, and then passes the result to the next layer. At the same time, a multi-head attention mechanism is introduced in each hidden layer, in which each attention head independently weights the input features. In this way, the stacking combination method enables the model to learn complex nonlinear relationships and improve the mapping ability of network traffic data.

[0097] Exemplarily, first, multiple data collection nodes are deployed at key nodes of the network to capture network traffic data in real time. Among them, data collection nodes, such as routers, switches, firewalls and intrusion detection systems. The collected network traffic data includes but is not limited to key traffic features such as IP address, port number, protocol type, packet size, flow duration, etc. The collected network traffic data is preprocessed, and the preprocessing may include removing duplicate records, filling missing values, data type conversion, normalization and normalization processing, etc. Specifically, the consistency and comparability of network traffic data can be ensured by minimum-maximum standardization and Z-score standardization methods. An outlier detection algorithm is used to process outliers to improve the accuracy of subsequent analysis. Then, according to the data source of the network traffic data, the key information in each network traffic data is extracted respectively, and the correlation between the key information and each data source is used as the network traffic feature, and the network traffic feature is formed into a network traffic data set. Afterwards, the network traffic multidimensional feature redundancy elimination method (ET-RFE) based on the extreme tree recursive elimination is adopted, and the importance of the network traffic features is evaluated by the extreme tree algorithm. The candidate feature data set is screened out by the number of network traffic features in the network traffic data set, or the cross-validation (CV) method. Subsequently, the full correlation significant feature selection method based on ET-Boruta (a feature selection method based on random forest) is applied to generate shadow features through randomness strategy, and the importance of shadow features is compared with network traffic features in the network traffic dataset to evaluate the importance of network traffic features. The more important network traffic features are retained as candidate feature datasets. Next, the data balancing method based on All-KNN (K nearest neighbor algorithm) is implemented. By reconstructing the network traffic features by one-hot encoding, the neighbor samples are determined for each reconstructed network traffic feature, and the type of the neighbor samples of each network traffic feature is used to determine whether to retain the network traffic feature. The retained network traffic features constitute the target feature dataset. Finally, an abnormal traffic detection model based on multi-layer perceptron (MHA-MLP) with multi-head attention mechanism is constructed. The target feature dataset is input into the first hidden layer of the MHA-MLP model, and multiple attention heads are applied to independently weight the features. The weights and biases of the model are optimized by back-propagation algorithm and gradient descent to construct the final abnormal traffic detection model.

[0098] Embodiment 2

[0099] The present application provides an abnormal flow monitoring method. Figure 2 As shown, Figure 2 The following is a basic flow chart of the abnormal flow monitoring method provided in the embodiment of the present application, including:

[0100] Step S201, obtaining the flow to be detected.

[0101] In some embodiments, the traffic to be detected is network traffic data.

[0102] For example, data collection nodes can be deployed at key nodes of the network to capture network traffic data in real time. The data collection nodes include, for example, routers, switches, firewalls, and intrusion detection systems. The data collection nodes are used to collect network traffic data.

[0103] Step S202: input the flow to be detected into a preset abnormal flow detection model to obtain whether the flow to be detected is abnormal.

[0104] The abnormal traffic detection model is obtained by the method of the first embodiment.

[0105] In some embodiments, inputting the flow to be detected into a preset abnormal flow detection model to obtain whether the flow to be detected is abnormal can be: preprocessing the flow to be detected; inputting the preprocessed flow to be detected into a preset abnormal flow detection model to obtain whether the flow to be detected is abnormal.

[0106] The manner of preprocessing the traffic to be detected may refer to the technical means of preprocessing the original network data in the first embodiment.

[0107] Embodiment 3

[0108] Based on the same inventive concept, the present application embodiment provides a training device for an abnormal flow detection model, such as Figure 3 As shown, the training device of the abnormal traffic detection model includes: a data set acquisition module 1, a first optimization module 2, a second optimization module 3 and a training module 4. Among them, the data set acquisition module 1 is used to acquire a network traffic data set; the network traffic data set includes multiple network traffic features; the first optimization module 2 is used to eliminate feature redundancy of the network traffic data set according to the importance of each network traffic feature and form an alternative feature data set; the second optimization module 3 is used to balance the alternative feature data set and form a target feature data set; the training module 4 is used to train a preset network model using the target feature data set to obtain an abnormal traffic detection model.

[0109] In some embodiments, the data set acquisition module 1 is used to acquire a network traffic data set in the following manner: acquiring network raw data and a data source of each network raw data; extracting key information from each network raw data according to the data source; analyzing the correlation between each data source, and integrating the correlation with each key information to obtain a number of network traffic features; each network traffic feature constitutes a network traffic data set.

[0110] In some embodiments, the first optimization module 2 is used to eliminate feature redundancy of the network traffic data set and form an alternative feature data set according to the importance of each network traffic feature in the following manner: step S11, input the network traffic data set into a preset first extreme tree model to obtain the importance score of each network traffic feature; step S12, remove the network traffic feature with the lowest importance score in the network traffic data set to obtain a new network traffic data set; loop steps S11 to S12 until the preset conditions are met, and the remaining network traffic features form the alternative feature data set; the preset conditions include: the number of network traffic features in the alternative feature data set is less than or equal to the preset number.

[0111] In some embodiments, the first optimization module 2 is used to eliminate feature redundancy of the network traffic data set and form an alternative feature data set according to the importance of each network traffic feature in the following manner: generate shadow features with the same data distribution characteristics as the network traffic features; use the preset second extreme tree model to construct several decision trees, calculate the Gini index reduction of each network traffic feature when the node of each decision tree is split, and select the network traffic feature with the largest Gini index reduction for splitting, and calculate the importance score of each network traffic feature according to all the Gini index reductions of each network traffic feature in the second extreme tree model; use the preset third extreme tree model to construct several decision trees, calculate the Gini index reduction of each shadow feature when the node of each decision tree is split, and select the shadow feature with the largest Gini index reduction for splitting, and calculate the importance score of each shadow feature according to all the Gini index reductions of each shadow feature in the third extreme tree model; select some network traffic features according to the network traffic features and the importance scores of the shadow features to form an alternative feature data set.

[0112] In some embodiments, the first optimization module 2 is used to filter some network traffic features and form an alternative feature data set according to the importance scores of network traffic features and shadow features in the following manner: obtain the maximum value of the importance scores of all shadow features; compare the importance score of each network traffic feature with the maximum value, and retain the network traffic features with importance scores greater than the maximum value to form an alternative feature data set.

[0113] In some embodiments, the second optimization module 3 is used to balance the alternative feature data sets and form a target feature data set in the following manner: for each network traffic feature: determine the distance between the network traffic feature and other network traffic features; determine the network traffic features whose distance is less than a preset threshold as neighbor samples; count the types of each neighbor sample and the number of each type, and define the type with the largest number as the dominant type; if the dominant type is consistent with the type of the network traffic feature, retain the network traffic feature; if the dominant type is inconsistent with the type of the network traffic feature, delete the network traffic feature or adjust the preset weight of the network traffic feature, and the remaining network traffic features form the target feature data set.

[0114] In some embodiments, the preset network model is a multi-layer perceptron model; the training module 4 is used to train the preset network model using the target feature data set in the following manner to obtain an abnormal traffic detection model: input the target feature data set into the multi-layer perceptron model for training, and use a multi-head attention mechanism to perform weighted summation on the features in the target feature data set to obtain an output result; calculate the loss value of the output result according to a preset loss function; optimize the weights and biases of each layer in the multi-layer perceptron model according to the loss value and a preset optimization algorithm until convergence or reaching a set number of iterations to obtain an abnormal traffic detection model.

[0115] It can be understood that the various embodiments described in the first embodiment are also applicable to the third embodiment if there is no conflict. Based on the consideration of brevity of description, they will not be repeated here.

[0116] Embodiment 4

[0117] Based on the same inventive concept, the present application embodiment provides an abnormal flow monitoring device, such as Figure 4 As shown, the abnormal flow monitoring device includes: a flow acquisition module 5 and an abnormality determination module 6. The flow acquisition module 5 is used to acquire the flow to be detected; the abnormality determination module 6 is used to input the flow to be detected into a preset abnormal flow detection model to obtain whether the flow to be detected is abnormal.

[0118] Embodiment 5

[0119] Combination Figure 5 As shown, an embodiment of the present application provides an electronic device, including a processor 10 and a memory 7. Optionally, the device may also include a communication interface 8 and a bus 9. Among them, the processor 10, the communication interface 8, and the memory 7 can communicate with each other through the bus 9. The communication interface 8 can be used for information transmission. The processor 10 can call the logic instructions in the memory 7 to execute the training method of the abnormal flow detection model or the abnormal flow monitoring method of the above embodiment.

[0120] In addition, the logic instructions in the above-mentioned memory 7 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0121] The memory 7 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present application. The processor 10 executes the function application and data processing by running the program instructions / modules stored in the memory 7, that is, the training method of the abnormal flow detection model or the abnormal flow monitoring method in the above embodiment is realized.

[0122] The memory 7 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 7 may include a high-speed random access memory and may also include a non-volatile memory.

[0123] The electronic device may be a computer or a server, etc.

[0124] An embodiment of the present application provides a storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above-mentioned abnormal traffic detection model training method or abnormal traffic monitoring method.

[0125] An embodiment of the present application provides a computer program product, which includes a computer program stored on a storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the above-mentioned abnormal traffic detection model training method or abnormal traffic monitoring method.

[0126] The computer-readable storage medium mentioned above may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0127] The technical solution of the embodiment of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present application. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and other media that can store program codes, or a transient storage medium.

[0128] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0129] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. At the same time, the above embodiments can be combined with each other to form new embodiments without conflict.

Claims

1. A training method for an abnormal traffic detection model, characterized in that: include: Get the network traffic dataset; The network traffic data set includes a plurality of network traffic features; Eliminate feature redundancy of the network traffic data set according to the importance of each of the network traffic features and form an alternative feature data set; Performing equalization processing on the candidate feature data sets to form a target feature data set; Using the target feature data set to train a preset network model to obtain an abnormal traffic detection model; The candidate feature data sets are balanced and formed into a target feature data set, including: for each of the network traffic features: determining the distance between the network traffic feature and other network traffic features; determining the network traffic features whose distance is less than a preset threshold as neighboring samples; counting the types of each of the neighboring samples and the number of each type, and defining the type with the largest number as the dominant type; if the dominant type is consistent with the type of the network traffic feature, retaining the network traffic feature; if the dominant type is inconsistent with the type of the network traffic feature, deleting the network traffic feature or adjusting the preset weight of the network traffic feature, and the remaining network traffic features form the target feature data set; According to the importance of each of the network traffic features, feature redundancy of the network traffic data set is eliminated and an alternative feature data set is formed, including: step S11, inputting the network traffic data set into a preset first extreme tree model to obtain the importance score of each of the network traffic features; step S12, removing the network traffic feature with the lowest importance score in the network traffic data set to obtain a new network traffic data set; looping steps S11 to S12 until a preset condition is met, and the remaining network traffic features form the alternative feature data set; the preset condition includes: the number of the network traffic features in the alternative feature data set is less than or equal to a preset number.

2. The method according to claim 1, characterized in that Get network traffic datasets, including: Acquire network original data and the data source of each of the network original data; Extracting key information from each of the network raw data according to the data source; The correlation relationship between each of the data sources is analyzed, and the correlation relationship is integrated with each of the key information to obtain a number of network traffic features; each of the network traffic features constitutes the network traffic data set.

3. The method according to claim 1, characterized in that Eliminating feature redundancy of the network traffic data set according to the importance of each of the network traffic features and forming an alternative feature data set, further comprising: generating a shadow feature having the same data distribution characteristics as the network traffic feature; Constructing a plurality of decision trees using the preset second extreme tree model, calculating the Gini index reduction of each network traffic feature when each decision tree node is split, and selecting the network traffic feature with the largest Gini index reduction for splitting, and calculating the importance score of each network traffic feature according to all Gini index reductions of each network traffic feature in the second extreme tree model; Constructing several decision trees using the preset third extreme tree model, calculating the Gini index reduction of each shadow feature when each decision tree node is split, and selecting the shadow feature with the largest Gini index reduction for splitting, and calculating the importance score of each shadow feature according to all the Gini index reductions of each shadow feature in the third extreme tree model; According to the importance scores of the network traffic features and the shadow features, some of the network traffic features are screened to form the candidate feature data set.

4. The method according to claim 3, characterized in that Filtering part of the network traffic features according to the importance scores of the network traffic features and the shadow features and forming the candidate feature data set includes: Obtaining the maximum value of importance scores of all the shadow features; The importance score of each of the network traffic features is compared with the maximum value, and the network traffic features with importance scores greater than the maximum value are retained to form the candidate feature data set.

5. The method according to claim 1, characterized in that The preset network model is a multi-layer perceptron model; the preset network model is trained using the target feature data set to obtain an abnormal traffic detection model, including: Inputting the target feature data set into the multi-layer perceptron model for training, and performing weighted summation on the features in the target feature data set using a multi-head attention mechanism to obtain an output result; Calculating the loss value of the output result according to a preset loss function; The weights and biases of each layer in the multi-layer perceptron model are optimized according to the loss value and a preset optimization algorithm until convergence or reaching a set number of iterations, thereby obtaining the abnormal traffic detection model.

6. An abnormal flow monitoring method, characterized in that: include: Get the traffic to be tested; Inputting the flow to be detected into a preset abnormal flow detection model to obtain whether the flow to be detected is abnormal; The abnormal flow detection model is obtained by the method described in any one of claims 1 to 5 above.

7. A training device for an abnormal flow detection model, characterized in that: include: A data set acquisition module, used to acquire network traffic data sets; The network traffic data set includes a plurality of network traffic features; A first optimization module, configured to eliminate feature redundancy of the network traffic data set and form an alternative feature data set according to the importance of each of the network traffic features; A second optimization module, used for performing equalization processing on the candidate feature data set to form a target feature data set; A training module, used to train a preset network model using the target feature data set to obtain an abnormal traffic detection model; Among them, the second optimization module is used to balance the candidate feature data sets and form a target feature data set in the following manner: for each network traffic feature: determine the distance between the network traffic feature and other network traffic features; determine the network traffic features whose distance is less than a preset threshold as neighboring samples; count the types of each neighboring sample and the number of each type, and define the type with the largest number as the dominant type; if the dominant type is consistent with the type of the network traffic feature, retain the network traffic feature; if the dominant type is inconsistent with the type of the network traffic feature, delete the network traffic feature or adjust the preset weight of the network traffic feature, and the remaining network traffic features form the target feature data set; The first optimization module is used to eliminate feature redundancy of the network traffic data set and form an alternative feature data set according to the importance of each of the network traffic features in the following manner, including: step S11, inputting the network traffic data set into a preset first extreme tree model to obtain the importance score of each of the network traffic features; step S12, removing the network traffic feature with the lowest importance score in the network traffic data set to obtain a new network traffic data set; looping steps S11 to S12 until a preset condition is met, and the remaining network traffic features form the alternative feature data set; the preset condition includes: the number of the network traffic features in the alternative feature data set is less than or equal to a preset number.

8. An electronic device, characterized in that: It includes a processor and a memory, the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the training method of the abnormal flow detection model described in any one of claims 1 to 5 or the abnormal flow monitoring method described in claim 6.

Images