Self-adaptive flow identification method based on dynamic QoS strategy

Abstract

The invention discloses a self-adaptive flow identification method based on a dynamic QoS strategy, and relates to the technical field of flow data identification, and the self-adaptive flow identification method comprises the following steps: S1, collecting flow data; s2, flow data preprocessing; s3, inputting the traffic data features into a known network traffic classification model for prediction, and outputting a corresponding category prediction result; and S4, performing preliminary judgment on the traffic data of the category prediction result according to a self-adaptive adjustment confidence coefficient principle to obtain unknown traffic data. According to the method, new representative data is acquired through a multi-period random sampling method to ensure the diversity and variability of the coverage traffic, and new malicious traffic data is simulated and generated by using historical malicious traffic data and adopting a data augmentation method based on numerical disturbance to expand the diversity of malicious encrypted traffic, so that the traffic coverage is more diversified. The malicious traffic can be identified and found more effectively, so that accurate and effective identification of unknown malicious traffic by the model is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of flow data identification, and in particular to an adaptive flow identification method based on dynamic QoS strategy. Background Art

[0002] Langqiao SDWAN routing gateway management system mainly includes: device management, policy configuration, VPN management, performance monitoring, alarm management, log management, permission management, report center, configuration backup and recovery, license management, update management, network topology, multi-tenant management, QoS management, API management and other functional sections. Among them, policy configuration includes routing policy, QoS policy, security policy, etc. It is mainly used in telecommunications, finance, government, education and other industries to provide enterprises with safe and reliable network connection and management solutions. The system realizes centralized management and intelligent path selection through advanced software-defined network technology, supports multi-line automatic load balancing and rapid fault recovery, and ensures efficient operation and stability of the network. In the network environment of dynamic QoS policy, the growth of network traffic makes unknown traffic mixed in the network traffic data stream. There may be threatening malicious traffic in the unknown traffic, which will pose a huge threat to the network security of the routing gateway management system. Its hidden dangers should not be underestimated. In the field of traffic detection, due to the increasing application of encryption technology, the traditional plaintext detection method has become ineffective. However, machine learning algorithms can use non-encrypted content data as training data to discover patterns and effectively identify malicious traffic. The detection method based on machine learning technology usually uses pre-collected normal business traffic (white traffic) and malicious encrypted traffic (black traffic) to build a training set, and then detects through the training model. Such a model is called a solidified model. However, after actual verification, it was found that due to the difference between the pre-collected white traffic and the white traffic in the customer's specific scenario network environment, the flexibility and adaptability of the solidified model are insufficient, which will cause some false alarms in the model. Therefore, an adaptive traffic identification method based on dynamic QoS strategy came into being.

[0003] The existing technology has the following problems:

[0004] In the real world, network application data is vast and constantly updated with the development of the network. New applications are unknown to existing models. Whenever new applications emerge, existing models cannot effectively identify and discover malicious traffic in unknown traffic, which greatly affects the accuracy of model recognition. The efficiency of identifying unknown traffic will also decrease, and even threaten network security. Summary of the invention

[0005] The present invention provides an adaptive traffic identification method based on dynamic QoS strategy to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] An adaptive traffic identification method based on a dynamic QoS strategy, the adaptive traffic identification method comprising the following steps:

[0008] S1: flow data collection;

[0009] S2: flow data preprocessing;

[0010] S3: Input the traffic data features into the known network traffic classification model for prediction, and output the corresponding category prediction results;

[0011] S4: Preliminary judgment is made on the traffic data of the category prediction results according to the confidence principle of adaptive adjustment to obtain unknown traffic data;

[0012] S5: Divide the unknown traffic data into different categories according to the adaptive clustering method and label them according to the categories;

[0013] S6: Identify unknown traffic data;

[0014] S7: Use the malicious traffic data and normal traffic data that have been identified in the unknown traffic data as known traffic data to train and update the known network traffic classification model.

[0015] Preferably, the network traffic data set includes a plurality of known traffic data and unknown traffic data, wherein the known traffic data and the unknown traffic data both include normal traffic data and malicious traffic data.

[0016] Preferably, normal traffic data is obtained through a multi-time period random sampling method to obtain new representative data, and historical malicious traffic data is used to simulate and generate new malicious traffic data using a data augmentation method based on numerical perturbations.

[0017] Preferably, S2 includes:

[0018] S21: Obtain a known traffic data training set;

[0019] S22: extracting the behavior features from each known traffic data set respectively, and performing preprocessing operations such as deduplication and missing value processing on the known traffic data training set by using time sequence priority, random sampling, load descending order, and load ascending order processing methods;

[0020] S23: Inputting each processed known traffic data set into a known network traffic classification model for iterative training until it has the ability to classify known traffic data;

[0021] S24: Parameter adjustment, adjust the category weights based on cost-sensitive learning to eliminate bias.

[0022] Preferably, the known traffic data training set includes known normal traffic data, known malicious traffic data, and mixed traffic data of known normal traffic data and known malicious traffic data, and the known network traffic classification model includes three sub-traffic classification network models.

[0023] Preferably, S5 includes: using a principal component analysis method to perform feature dimension reduction on the unknown flow data, and using a clustering method to obtain a category label for the unknown flow data after dimension reduction.

[0024] Preferably, S6 includes: identifying the category of unknown traffic data according to a similarity coefficient estimation method, estimating the similarity of the traffic data by calculating the KL divergence index of two traffic data features, that is, normal traffic data and malicious traffic data.

[0025] Preferably, S4 includes: if the category prediction result is less than the confidence threshold, then the flow data corresponding to the category prediction result is judged to be unknown flow data; if the category prediction result is greater than or equal to the confidence threshold, then the flow data corresponding to the category prediction result is judged to be known flow data; when the flow data exceeding a preset number in the network flow data set is preliminarily judged to be unknown flow data, then the flow data with the lowest preset number of category prediction results is judged to be unknown flow data;

[0026] When a preliminary judgment is made on unknown traffic data based on the category prediction results and the confidence threshold: the initial value of the adaptive confidence threshold is a preset value. When the traffic data in the network traffic data set that contains less than a preset amount is preliminarily judged to be unknown traffic data, the preset value is increased at a ratio of (1+a), where a is a preset parameter.

[0027] Due to the adoption of the above technical solution, the present invention has the following technical advances compared with the prior art:

[0028] 1. The present invention provides an adaptive traffic identification method based on a dynamic QoS strategy, which first classifies known traffic in network data using a known network traffic classification model, then makes a preliminary judgment based on the classification prediction result, uses an adaptive clustering method to classify the data initially judged as unknown traffic, and then identifies each category using a similarity coefficient estimation method to identify the category of malicious traffic and the category of normal traffic, further recognizes and learns the unknown traffic data, converts it into known traffic data, and inputs the new known traffic data into the known network traffic classification model again for training and updating, so that the known network traffic classification model can learn and train the new network data that constantly appears during actual application so that it has the ability to identify more network traffic data, and continuously improves the ability to identify unknown traffic data, so that it is suitable for the current rapidly changing network environment of dynamic QoS strategies.

[0029] 2. The present invention provides an adaptive traffic identification method based on dynamic QoS strategy. New representative data is obtained through a multi-time random sampling method to ensure the diversity and variability of the coverage traffic. The historical malicious traffic data is used to simulate and generate new malicious traffic data using a data augmentation method based on numerical perturbation to expand the diversity of malicious encrypted traffic. The malicious traffic can be more effectively identified and discovered, thereby ensuring that the model can accurately and effectively identify unknown malicious traffic.

[0030] 3. The present invention provides an adaptive traffic identification method based on dynamic QoS strategy. After marking the unknown network traffic, these traffic data can be added to the earliest known network traffic data set to train and update the known network traffic classification model. Through this process, the model can continuously identify and mark the newly appeared unknown traffic in the network. In the actual network environment based on dynamic QoS strategy, the model always has efficient and accurate identification capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the adaptive traffic identification process of the present invention. DETAILED DESCRIPTION

[0032] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0033] like Figure 1 As shown, an adaptive traffic identification method based on a dynamic QoS policy comprises the following steps:

[0034] S1: flow data collection;

[0035] S2: flow data preprocessing;

[0036] S3: Input the traffic data features into the known network traffic classification model for prediction, and output the corresponding category prediction results;

[0037] S4: Preliminary judgment is made on the traffic data of the category prediction results according to the confidence principle of adaptive adjustment to obtain unknown traffic data;

[0038] S5: Divide the unknown traffic data into different categories according to the adaptive clustering method and label them according to the categories;

[0039] S6: Identify unknown traffic data;

[0040] S7: Use the malicious traffic data and normal traffic data that have been identified in the unknown traffic data as known traffic data to train and update the known network traffic classification model.

[0041] First, the known traffic in the network traffic data is classified using a known network traffic classification model, and then a preliminary judgment is made based on the classification prediction results. The adaptive clustering method is used to classify the data that is initially judged to be unknown traffic. Then, the categories are identified through the similarity coefficient estimation method, and the categories of malicious traffic and normal traffic are identified. The unknown traffic data is further recognized and learned, and converted into known traffic data. The new known traffic data is input into the known network traffic classification model again for training and updating. In the process of actual application, the known network traffic classification model can learn and train the new network data that continues to appear, so that it has the ability to identify more network traffic data, and continuously improves the ability to identify unknown traffic data, making it suitable for the current rapidly changing network environment of dynamic QoS policies.

[0042] The network traffic data set includes a plurality of known traffic data and unknown traffic data, wherein the known traffic data and the unknown traffic data both include normal traffic data and malicious traffic data.

[0043] Normal traffic data is obtained through random sampling methods in multiple time periods, and new representative data is obtained to ensure the diversity and variability of coverage traffic. Historical malicious traffic data is used, and a data augmentation method based on numerical perturbation is adopted to simulate the generation of new malicious traffic data to expand the diversity of malicious encrypted traffic.

[0044] S2 includes:

[0045] S21: Obtain a known traffic data training set;

[0046] S22: extracting the behavior features from each known traffic data set respectively, and performing preprocessing operations such as deduplication and missing value processing on the known traffic data training set by using time sequence priority, random sampling, load descending order, and load ascending order processing methods;

[0047] S23: Inputting each processed known traffic data set into a known network traffic classification model for iterative training until it has the ability to classify known traffic data;

[0048] S24: Parameter adjustment, adjust the category weights based on cost-sensitive learning to eliminate bias.

[0049] By adjusting parameters, the model can treat different types of traffic in a balanced manner, thereby ensuring that the model can accurately and effectively identify unknown malicious traffic.

[0050] The known traffic data training set includes known normal traffic data, known malicious traffic data, and mixed traffic data of known normal traffic data and known malicious traffic data. The known network traffic classification model includes three sub-traffic classification network models.

[0051] S5 includes: using a principal component analysis method to perform feature dimension reduction on unknown traffic data, and using a clustering method to obtain a category label for the unknown traffic data after dimension reduction.

[0052] After the unknown network traffic is labeled, the traffic data can be added to the earliest known network traffic data set to train and update the known network traffic classification model. Through this process, the model can continuously identify and label new unknown traffic in the network. In the actual network environment based on dynamic QoS policies, the model always has efficient and accurate recognition capabilities.

[0053] S6 includes: identifying the category of unknown traffic data according to a similarity coefficient estimation method, estimating the similarity of the traffic data by calculating the KL divergence index of two traffic data features, that is, normal traffic data and malicious traffic data.

[0054] S4 includes: if the category prediction result is less than the confidence threshold, then the traffic data corresponding to the category prediction result is judged to be unknown traffic data; if the category prediction result is greater than or equal to the confidence threshold, then the traffic data corresponding to the category prediction result is judged to be known traffic data; when traffic data exceeding a preset number in the network traffic data set is preliminarily judged to be unknown traffic data, then the traffic data with the lowest preset number of category prediction results is judged to be unknown traffic data;

[0055] When a preliminary judgment is made on unknown traffic data based on the category prediction results and the confidence threshold: the initial value of the adaptive confidence threshold is a preset value. When the traffic data in the network traffic data set that contains less than a preset amount is preliminarily judged to be unknown traffic data, the preset value is increased at a ratio of (1+a), where a is a preset parameter.

[0056] In summary, traffic data is collected, traffic data features are extracted, and a known traffic data training set is obtained. The traffic data training set is preprocessed using time priority, random sampling, load descending order, and load ascending order processing methods. Each processed known traffic data set is input into a known network traffic classification model for iterative training until it has the ability to classify known traffic data. The traffic data features are input into the known network traffic classification model for prediction and classification. Then, the traffic data of the category prediction results is preliminarily judged based on the confidence principle of adaptive adjustment. The traffic data preliminarily judged as unknown is classified using an adaptive clustering method and labeled according to the category. After the unknown network traffic is labeled, these traffic data can be added to the earliest known network traffic data set to train and update the known network traffic data set. Network traffic classification model. Through this process, the model can continuously identify and mark the newly emerging unknown traffic in the network. The model always has efficient and accurate recognition capabilities. Then, each category is identified through the similarity coefficient estimation method to identify the category of malicious traffic and the category of normal traffic. The unknown traffic data is further recognized and learned, and it is converted into known traffic data. The new known traffic data is input into the known network traffic classification model again for training and updating. In the actual application process, the known network traffic classification model can learn and train the new network data that continues to emerge, so that it has the ability to identify more network traffic data, and continuously improves the ability to identify unknown traffic data, making it suitable for the current rapidly changing dynamic QoS policy network environment.

[0057] The above generally describes the present invention in detail, but it is obvious to a person skilled in the art that some modifications or improvements can be made to the present invention. Therefore, modifications or improvements that do not depart from the spirit of the present invention are within the scope of protection of the present invention.

Claims

1. An adaptive traffic identification method based on dynamic QoS strategy, characterized by: The adaptive traffic identification method comprises the following steps: S1: flow data collection; S2: flow data preprocessing; S3: Input the traffic data features into the known network traffic classification model for prediction, and output the corresponding category prediction results; S4: Preliminary judgment is made on the traffic data of the category prediction results according to the confidence principle of adaptive adjustment to obtain unknown traffic data; S5: Divide the unknown traffic data into different categories according to the adaptive clustering method and label them according to the categories; S6: Identify unknown traffic data; S7: Use the malicious traffic data and normal traffic data that have been identified in the unknown traffic data as known traffic data to train and update the known network traffic classification model.

2. The method for adaptive traffic identification based on dynamic QoS strategy according to claim 1, characterized in that: The network traffic data set includes a plurality of known traffic data and unknown traffic data, wherein the known traffic data and the unknown traffic data both include normal traffic data and malicious traffic data.

3. The method for adaptive traffic identification based on dynamic QoS strategy according to claim 1, characterized in that: Normal traffic data is obtained through random sampling methods in multiple time periods to obtain new representative data. Historical malicious traffic data is used to simulate and generate new malicious traffic data using a data augmentation method based on numerical perturbations.

4. The method for adaptive traffic identification based on dynamic QoS strategy according to claim 1, characterized in that: The S2 includes: S21: Obtain a known traffic data training set; S22: extracting the behavior features from each known traffic data set respectively, and performing preprocessing operations such as deduplication and missing value processing on the known traffic data training set by using time sequence priority, random sampling, load descending order, and load ascending order processing methods; S23: Inputting each processed known traffic data set into a known network traffic classification model for iterative training until it has the ability to classify known traffic data; S24: Parameter adjustment, adjust the category weights based on cost-sensitive learning to eliminate bias.

5. The method for adaptive traffic identification based on dynamic QoS strategy according to claim 4 is characterized in that: The known traffic data training set includes known normal traffic data, known malicious traffic data, and mixed traffic data of known normal traffic data and known malicious traffic data. The known network traffic classification model includes three sub-traffic classification network models.

6. The method for adaptive traffic identification based on dynamic QoS strategy according to claim 1, characterized in that: The S5 includes: using a principal component analysis method to perform feature dimension reduction on the unknown flow data, and using a clustering method to obtain a category label for the unknown flow data after dimension reduction.

7. The method for adaptive traffic identification based on dynamic QoS strategy according to claim 1, characterized in that: The S6 includes: identifying the category of unknown traffic data according to a similarity coefficient estimation method, estimating the similarity of the traffic data by calculating the KL divergence index of two traffic data features, that is, normal traffic data and malicious traffic data.

8. The method for adaptive traffic identification based on dynamic QoS strategy according to claim 1, characterized in that: The S4 includes: if the category prediction result is less than the confidence threshold, then the flow data corresponding to the category prediction result is judged to be unknown flow data; if the category prediction result is greater than or equal to the confidence threshold, then the flow data corresponding to the category prediction result is judged to be known flow data; when the flow data exceeding a preset number in the network flow data set is preliminarily judged to be unknown flow data, then the flow data with the lowest preset number of category prediction results is judged to be unknown flow data; When a preliminary judgment is made on unknown traffic data based on the category prediction results and the confidence threshold: the initial value of the adaptive confidence threshold is a preset value. When the traffic data in the network traffic data set that contains less than a preset amount is preliminarily judged to be unknown traffic data, the preset value is increased at a ratio of (1+a), where a is a preset parameter.

Images