Open set scene expansion method and system of closed set network flow classification model

Abstract

The invention discloses an open set scene expansion method and system for a closed set network traffic classification model, and belongs to the technical field of network traffic classification. Under the condition that an existing model does not need to be redesigned or retrained, a network traffic classification model trained in a closed set scene is expanded to an open set scene, and unknown traffic is effectively recognized. According to the method, a trained network traffic classification model in a closed set scene is used as a feature extractor, feature extraction is carried out on known traffic, then extracted deep-level features are mapped to a hyperspherical embedding space, spatial distribution is optimized by using aggregation loss and discrete loss, and a hyperspherical model is obtained. According to the method, close aggregation of known categories and discretization between the categories in a hypersphere embedding space are ensured, and then unknown flow is identified based on the distance between a to-be-detected sample and a space range of a known category sample. According to the method, any trained network traffic classification model can be seamlessly inserted, unknown traffic is detected, and the method is suitable for a network traffic classification task in an open set scene.

Description

Technical Field

[0001] The present invention involves extracting features based on the sequence information of network traffic using a trained network traffic classification model, and optimizing the feature space based on artificial intelligence-related technologies, so that unknown traffic can be detected without affecting normal classification. Specifically, it relates to an open-set scenario expansion method and system based on a closed-set network traffic classification model. Background Art

[0002] Network traffic classification is the process of associating network traffic with the application protocols or traffic-generating applications. By analyzing the characteristics of network traffic, it can be accurately classified into specific categories. Network traffic classification plays a crucial role in network performance management and security areas such as resource allocation and network intrusion detection. Quality of Service (QoS) and network anomaly detection both require accurate network traffic classification.

[0003] In recent years, significant progress has been made in classifying network traffic in closed-set scenarios, where all possible categories are known and included in the training dataset. However, with the advancement of network technology, new applications and devices continue to emerge, and the variety of network traffic has also increased. In real-world scenarios, in addition to traffic samples with known categories consistent with the training data, there are also a large number of traffic samples with unknown categories. Therefore, how to accurately classify network traffic in open-set scenarios has become an important problem that needs to be solved urgently.

[0004] Most existing network traffic classification methods are only applicable to closed-set scenarios, meaning they can only identify samples of known classes seen during training. When applied to open-set scenarios, these methods often encounter a large number of traffic of unknown classes. Without intervention, they assume these unknown classes are known and subsequently classify them, significantly reducing classification performance and posing a significant security risk to network management. Previous research has introduced several network traffic classification methods specifically for open-set scenarios. The Max Softmax method relies on the maximum probability value output by the Softmax function as a decision basis. However, the model often outputs high-confidence incorrect predictions for unknown samples, resulting in a high false positive rate. The MaxLogit method uses the logit value before the final classification to distinguish between known and unknown classes, but struggles with complex scenarios with overlapping feature distributions. MaxGrad and SquareGrad are gradient-based methods that detect unknown classes by backpropagating the gradient magnitude. However, they are only applicable to models with gradients (deep learning models) and are expensive. Some methods periodically retrain models to ensure the model's ability to identify and predict unknown traffic. However, retraining has limitations in practical applications and results in excessive cost and resource waste. These methods either rely on specific model structures or are limited to static threshold strategies, making it difficult to achieve both efficiency and robustness.

[0005] Therefore, achieving unknown traffic detection without modifying existing models trained for closed-set scenarios has become an important research direction. To address this issue, this paper proposes a plug-and-play approach. This approach eliminates the need to redesign or retrain network traffic classification models; instead, it can be plugged into any trained model, making it applicable to open-set network traffic classification. Summary of the Invention

[0006] The purpose of the present invention is to design and implement an open set scenario expansion method and system for a closed set network traffic classification model. Without the need to redesign or retrain the existing model, the network traffic classification model trained in the closed set scenario is extended to the open set scenario to effectively identify unknown traffic. The method uses the known traffic of the network traffic classification model that has been trained in the closed set scenario for feature extraction, and then maps the extracted deep-level features to the hypersphere embedding space, and optimizes the spatial distribution using aggregation loss and discrete loss, and then identifies unknown traffic based on the distance between the spatial range of the sample to be detected and the sample of the known category. In order to achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] A method and system for expanding open-set scenarios of a closed-set network traffic classification model, characterized by comprising a traffic feature space reconstruction phase and an unknown traffic detection phase;

[0008] The traffic feature space reconstruction phase includes the following steps:

[0009] 1) Preprocess the raw network traffic data of known traffic, reorganize the flow samples based on the quintuples to generate unique flow samples, and extract the sequence information of the flow samples to generate a flow sample set;

[0010] 2) Input the traffic samples preprocessed in step 1) into the network traffic classification model (feature extractor) for feature extraction to obtain deep-level features of the traffic;

[0011] 3) Initialize a hypersphere mapping model to map the deep features obtained in step 2) to the low-dimensional hypersphere embedding space, and perform strict normalization on the mapped feature vectors to complete the initialization mapping;

[0012] 4) Use the Von Mises-Fisher (VMF) distribution to model the characteristic distribution of multi-category network traffic in the initial hypersphere embedding space in step 3). Define the category prototype and concentration parameter for each category. The category prototype points to the center direction of the traffic category on the hypersphere space, and the concentration parameter controls the density of the distribution.

[0013] 5) By designing a discrete loss function and an aggregate loss function, the initialized hypersphere mapping model in step 3) is optimized, and the feature distribution of the hypersphere embedding space is dynamically adjusted to ensure that intra-class features are tightly clustered and inter-class features are discrete. Finally, the optimized hypersphere mapping model is output;

[0014] The unknown traffic detection phase includes the following steps:

[0015] 6) The traffic to be tested undergoes the same preprocessing and feature extraction process as steps 1) and 2) to ensure consistency in the input space and obtain the deep features of the traffic to be tested. The deep features of the traffic to be tested are mapped to the reshaped hypersphere embedding space using the hypersphere mapping model optimized in step 5), and the feature vectors are normalized to preserve directional information.

[0016] 7) Based on the feature vector of the hypersphere embedding space obtained in step 6), the K-nearest neighbor method is used to calculate the cosine distance between the traffic sample to be detected and the nearest known category sample as the distance score. Based on the dynamic judgment threshold rule and combined with the AUROC evaluation indicator, the network traffic classification model is evaluated to detect unknown traffic in the open set scenario.

[0017] An open-set scenario expansion system for a closed-set network traffic classification model is characterized by comprising a traffic feature space reconstruction module and an unknown traffic detection module.

[0018] The traffic feature space reconstruction module is used to preprocess known traffic and use the traffic classification model trained under the known traffic set to extract features to obtain deep features. The deep features are then mapped to the hypersphere embedding space and VMF distribution modeling is performed. The feature space is reshaped by optimizing the discrete loss and the aggregate loss through the joint loss.

[0019] The unknown traffic detection module is used to preprocess and extract features of the traffic to be detected, map the traffic to be detected to the reshaped hypersphere embedding space, calculate the distance score, and detect the unknown traffic based on the dynamic judgment threshold rule.

[0020] The key technical points of the present invention are:

[0021] 1. Hypersphere embedding space mapping technology is used to map the deep features of known traffic into a low-dimensional hypersphere embedding space and model the feature distribution using Von Mises-Fisher distribution.

[0022] 2. Design a combination of aggregation loss and discrete loss to optimize the feature distribution of the hypersphere embedding space.

[0023] 3. By introducing the hypersphere embedding space mapping technology, the present invention can extend the network traffic classification model in closed scenarios to open scenarios without redesigning or retraining the existing model, significantly reducing the complexity and resource consumption of model expansion.

[0024] Compared with the disclosed related technologies, it has the following advantages:

[0025] 1. This invention can extend the network traffic classification model in closed scenarios to open scenarios without changing the existing model structure, and has efficient open scenario adaptability.

[0026] 2. The method proposed in this invention does not require redesigning or retraining existing models. Instead, it only requires adding a hypersphere embedding space mapping module to the existing model to achieve unknown traffic detection in open-set scenarios, significantly reducing the consumption of computing resources and time.

[0027] 3. The method of the present invention is applicable to various network traffic classification models based on machine learning and deep learning, whether gradient-based or non-gradient models. It can realize traffic classification in open scenarios through a simple plug-in method and has wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the traffic feature space reconstruction phase of the open set scenario expansion method of the closed set network traffic classification model.

[0029] Figure 2 It is a flow chart of the unknown traffic detection phase of the open set scenario expansion method of the closed set network traffic classification model.

[0030] Figure 3 It is an open-set scenario expansion system architecture diagram of a closed-set network traffic classification model. DETAILED DESCRIPTION

[0031] The workflow of the present invention can be divided into a traffic feature space reconstruction phase and an unknown traffic detection phase. In the traffic feature space reconstruction phase, deep feature extraction is performed on known traffic flows and mapped to a hypersphere embedding space modeled using a VMF distribution. This hypersphere embedding space is then optimized using a combination of discrete and aggregate losses to reshape the hypersphere embedding space. In the unknown traffic detection phase, distance scores are used to detect unknown traffic flows using the reshaped hypersphere embedding space, extending the closed-set network traffic classification model to open-set scenarios.

[0032] In the traffic feature space reconstruction stage, the key technical part of the present invention is: based on the quintuple to reorganize the original traffic, generate a unique flow sample, extract attributes such as byte sequence or packet length as required, input to the traffic classification model trained in the closed set scenario to extract deep-level features, then initialize the hypersphere embedding space, map the deep-level features to the low-dimensional hypersphere embedding space through the projection head, retain only the direction information after normalization, use the VMF probability distribution model to define the category prototype and distribution parameters, preliminarily construct a compact embedding space within the class, and output an optimized hypersphere embedding projection head. The construction process is as follows: Figure 1 The specific implementation steps are as follows:

[0033] 1. The input raw network traffic pcap file is first preprocessed. Stream reassembly and session identification are performed based on the standard five-tuple (source IP, source port, destination IP, destination port, and transport protocol). This ensures the uniqueness of each network session and forms a flow sample set. Based on the architectural characteristics of the subsequent feature extractor, targeted traffic sample sequence information is extracted. For example, for models such as ET-BERT that analyze packet content, payload byte sequences are extracted, while for models such as FS-Net that rely on statistical features, packet length timing features are extracted. The preprocessing process also includes key steps such as appropriate data cleaning and abnormal flow filtering (such as filtering DNS traffic), laying the foundation for feature extraction.

[0034] 2. After completing traffic preprocessing, based on the feature extractor trained in the closed scenario (that is, the network traffic classification model obtained by training, verification and testing using a known traffic data set in the closed scenario), , the traffic sample sequence information is input into it, and the network traffic classification model is made to make real predictions, but in fact the prediction results are not obtained, but it is used to fully extract traffic features. This design retains the high-level semantic information of the traffic, and the feature extraction process can extract the deep features of the traffic samples. These deep features, typically from the penultimate layer of the model and often referred to as high-level semantic features, reflect high-level representations and contextual information of traffic. These extracted deep features serve as input for subsequent hypersphere embedding space initialization. The feature extraction process supports both traditional machine learning models and a variety of deep learning models, including CNNs, RNNs, and Transformers, enhancing key feature dimensions through deep neural networks.

[0035] 3. After completing the traffic feature extraction, we build an initialized hypersphere mapping model to map the deep features to the low-dimensional (128-dimensional) hypersphere embedding space. The hypersphere mapping model uses a two-layer fully connected network structure, and the weight matrix uses the Xavier initialization method to maintain the stability of the feature distribution. The input of the hypersphere mapping model is the deep traffic features obtained in the traffic feature extraction, and the output is the feature vector in the hypersphere embedding space. The feature representation is ,in It's a projection head. It is a deep-seated feature.

[0036] 4. Use Von Mises-Fisher (VMF) distribution to model the feature distribution in the hypersphere embedding space. VMF distribution is a distribution model applicable to data points on a sphere, especially suitable for directional data in high-dimensional space. When considering multiple categories, the embedding space can be modeled as a mixture of category-conditional VMF distributions, where each category c∈{1,2,…,C}. The feature distribution of each category is represented by a unit vector (category prototype vector) and a concentration parameter Control. The probability density function of VMF distribution is:

[0037] in, is a single network flow sample point in the hypersphere embedding space, is the category prototype of the category to which the traffic sample belongs, is a concentration parameter that controls the tightness of the distribution. is the dimension of the feature space, is the normalization factor. The larger The value indicates that the classification is more concentrated in the direction of the mean Around. When = 0, the sample points are evenly distributed on the hypersphere. Under this probability model, the input vector Assigned to the category to which it belongs The probability is:

[0038]

[0039] in , is the temperature parameter, which is used to control the smoothness of the distribution.

[0040] 5. Eigenvectors in the initialization hypersphere embedding space after mapping After strict L2 normalization, it is converted into a unit vector In order to better utilize the directional information in the VMF distribution, it is necessary to ensure that all eigenvectors are located on the unit hypersphere and only retain the directional information of the eigenvectors to avoid the influence of the size of the eigenvectors on subsequent analysis. A small perturbation term is introduced in the normalization process to prevent numerical instability and ensure computational robustness.

[0041] 6. The initialized hypersphere embedding space alone cannot achieve tight clustering within a class and discretization between classes. It needs to be reshaped to generate a feature representation space with high discriminability (distinguishing known and unknown traffic). Discrete loss is used to optimize the distance between prototypes of different categories in the hypersphere embedding space to achieve discretization between classes. Design a discrete loss function By maximizing the cosine similarity between prototypes of different categories, each category is uniformly distributed on the hypersphere, ensuring that the prototype vectors of different categories maintain a large angular separation in the hypersphere space. The calculation formula for discrete loss is:

[0042]

[0043] in, is the total number of traffic classes, It is The class prototype to which the traffic class belongs, It is The class prototype to which the traffic class belongs.

[0044] 7. Use aggregation loss to optimize the distance between data points of the same category and their category prototypes in the hypersphere embedding space. The aggregation loss function ensures that the feature vectors of the same category are closely clustered by minimizing the cosine similarity between similar samples and category prototypes. Design aggregation loss function , ensuring that the feature vectors of the same category are tightly clustered around their category prototypes in the hypersphere space. The calculation formula of the compactness loss is:

[0045]

[0046] in, is the total number of training samples, It is The embedding vector of each sample, It is The categories to which the samples belong The prototype vector of is a temperature parameter that adjusts the smoothness of the distribution.

[0047] 8. Combine aggregation loss and discrete loss to jointly optimize the feature distribution of the hypersphere embedding space, ensure tight aggregation within known categories and dispersion between categories, and design the total loss function :

[0048]

[0049] in, is the weight parameter for the compactness loss. By optimizing the total loss function, the feature distribution of the hypersphere embedding space can be reshaped. The two losses are weighted together using the balance coefficient λ. End-to-end training is performed using the AdamW optimizer. After sufficient iterations, the resulting feature space exhibits ideal discriminative properties: dense clustering within classes and clear separation between classes. Ultimately, an optimized hypersphere mapping model is obtained, which is used for hypersphere embedding space mapping in subsequent unknown traffic detection processes.

[0050] In the unknown traffic detection stage, the key technical part of the present invention is to use the optimized hypersphere embedding space to achieve efficient unknown traffic detection. Similar to the first stage, the traffic to be detected is first preprocessed and traffic features are extracted. Then, the deep features extracted from the traffic features are mapped to the reshaped hypersphere embedding space, and the distance scores between the samples to be detected and the known samples in the hypersphere embedding space are calculated, thereby distinguishing known traffic from unknown traffic and realizing unknown traffic detection. The construction process is as follows: Figure 3 The specific implementation steps are as follows:

[0051] 1. First, the traffic to be tested undergoes the same traffic preprocessing and feature extraction as the first stage: The flow samples are reassembled based on the same five-tuple rule, and their sequence information is extracted. Depending on the feature extractor type, either byte sequences or packet length sequences are extracted to ensure consistency across the input space. The traffic sample sequence information is fed into the feature extractor trained in a closed-set scenario, allowing the network traffic classification model to make realistic predictions and fully extract traffic features. This design preserves high-level semantic information of the traffic, allowing the feature extraction process to extract deep-level features of the traffic samples to be tested.

[0052] 2. The deep features of the traffic to be detected are mapped to the hypersphere embedding space through the hypersphere mapping model trained in the first stage. The mapping process disables gradient calculation and fixes the model parameters. After mapping, the feature vectors in the hypersphere embedding space are normalized to retain only the directional information.

[0053] 3. Use the K-nearest neighbor method (K=1) to calculate the distance between the feature vector of the traffic sample to be tested and the nearest known class traffic sample. The calculated distance is used as the detection score. The distance is used to determine whether the sample belongs to the unknown class. A larger distance score indicates that the traffic sample to be tested is farther away from the known class traffic sample, has a lower similarity, and is more likely to belong to the unknown class. A smaller distance score indicates that the traffic sample to be tested is closer to the known class traffic sample, is close to the distribution area of a known class traffic sample, has a higher similarity with the known traffic, and is more likely to belong to the known class.

[0054] 4. Based on the obtained scores of the samples to be tested, combined with their labels (known categories are '0', location categories are '1'), and automatically set thresholds, the AUROC metric is calculated to assess the unknown traffic detection capability. This step classifies traffic samples as known or unknown, providing a basis for subsequent predictions, thus extending the closed-set network traffic classification model to open-set scenarios.

[0055] Combined with the above-mentioned open set scenario expansion method of a closed set network traffic classification model, this patent also discloses an open set scenario expansion system for a closed set network traffic classification model. This system mainly consists of two important stages: the traffic feature space reconstruction stage and the unknown traffic detection stage. The system diagram architecture is as follows: Figure 3 As shown:

[0056] 1. Traffic feature space reconstruction stage: Using a known traffic set as input, the raw network traffic data is preprocessed to obtain traffic sequence features, and the traffic classification model trained under the known traffic set is used to extract features to obtain deep-level features. The present invention maps the deep-level features of the known traffic to a low-dimensional hypersphere embedding space through an initialized hypersphere mapping model, and then normalizes the feature vectors to retain directional information. The feature distribution of the hypersphere space is then dynamically optimized by jointly optimizing the aggregation loss and the discrete loss, achieving the reshaping of the hypersphere embedding space and providing a clear decision boundary for unknown traffic detection. The output of this stage is the optimized hypersphere mapping model.

[0057] 2. Unknown traffic detection stage: Using the traffic set to be detected as input, the original network traffic data is preprocessed to obtain traffic sequence features, and the traffic classification model trained under the known traffic set is used to extract features to obtain deep-level features. The present invention maps the deep-level features of the traffic sample to be tested to the reshaped low-dimensional hypersphere space through the optimized hypersphere mapping model. Finally, the similarity distance between the normalized features of the traffic sample to be tested and the nearest known category sample point is calculated, and the unknown traffic is detected based on the dynamic judgment threshold rule. The output result of this stage is whether the traffic to be tested is unknown.

[0058] This paper systematically validates its approach on four open network traffic datasets. The AndroidApp dataset contains 28,000 traffic samples from 28 mainstream Android applications, covering typical mobile application scenarios such as social networking, video, and shopping. Each traffic flow includes the complete protocol interaction process. The CSNET-TLS1.3 dataset, collected from real network environments, contains 46,372 samples from 120 TLS 1.3 encrypted services, accurately annotated using the SNI field. The USTC-TFC dataset consists of 5,312 encrypted traffic samples from 10 types of malware traffic and 10 types of normal application traffic, making it a benchmark for malicious traffic detection. The ISCX-VPN dataset records 29 types of application traffic in VPN and non-VPN environments, containing 3,163 samples. It covers multiple protocol types such as OpenVPN and PPTP and exhibits complex encryption characteristics. Details of the four datasets are shown in Table 1.

[0059] Table 1: Four open network traffic datasets

[0060]

[0061] Based on the aforementioned datasets, four cross-dataset open-set scenarios were designed to simulate the challenges of unknown traffic detection in real-world network environments, as shown in Table 2. In Scenario A, all 28 categories of Android apps were used as known categories, and the 20 categories of malicious / normal traffic from USTC-TFC were used as the unknown test set. This focused on verifying the ability to detect the distributional migration from normal applications to malicious traffic. Scenario B continued to use Android apps as known categories, but replaced the unknown set with 29 categories of VPN traffic from ISCX-VPN, examining the cross-environmental adaptability of mobile applications to encrypted services. Scenario C used the 120 categories of TLS services from CSNET-TLS1.3 as known categories and USTC-TFC as the unknown set to test the anomaly detection performance in large-scale service identification. Scenario D was conducted entirely within the context of encrypted traffic, using CSNET-TLS1.3 as the known category and ISCX-VPN as the unknown set, challenging the ability to distinguish between encrypted traffic in a fine-grained manner.

[0062] Table 2: Opening scene settings

[0063]

[0064] The experimental part uses three core evaluation indicators: FPR95 (false alarm rate at 95% recall) measures the false alarm control ability, AUROC (area under the ROC curve) evaluates the overall detection performance, and AUPR (area under the precision-recall curve) focuses on detection stability in class imbalance scenarios.

[0065] Using the FS-Net network traffic classification model trained under a known traffic dataset as a feature extractor, the performance of the present invention in unknown traffic detection is compared with four mainstream methods: MSP, MaxLogit, MaxGrad, and SquareGrad, in four scenarios, as shown in Table 3. MSP sets a threshold based on the maximum probability value output by Softmax. If the maximum probability of a sample is low, it is judged to be an unknown class. MaxLogit uses the Logit value (confidence) output by the classification model as the basis for judgment, and low-confidence samples are regarded as unknown classes. MaxGrad uses the maximum value of the first-step gradient of the model backpropagation as the detection score. A high gradient value corresponds to an unknown class and is only applicable to deep learning models. SquareGrad is similar to MaxGrad, but calculates the square root of the sum of the squared gradients as a score to comprehensively evaluate the overall change in the gradient. It is also only applicable to deep learning models.

[0066] Table 3: Comparison of the present invention with four baseline methods

[0067]

[0068] As shown in Table 3, in malware traffic detection in scenario A, the FPR95 of the present invention is as low as 8.79%, a decrease of 15.1% compared with the suboptimal method MaxGrad (23.89%), the AUROC reaches 95.84%, an increase of 7.43% compared with the suboptimal method MaxGrad (88.41%), and the AUPR reaches 96.41%, an increase of 8.49% compared with the suboptimal method MaxGrad (87.92%), demonstrating the ability to accurately identify malicious traffic; in VPN traffic detection in scenario B, the FPR95 of the present invention is as low as 15.67%, a decrease of 15.9% compared with the suboptimal method Msp (31.57%), the AUROC index of the present invention reaches 93.26%, a significant increase of 8.86% compared with the suboptimal MaxGrad (84.40%), and the AUPR reaches 92.87%, an increase of 8.99% compared with the suboptimal method MaxGrad (83.88%), demonstrating the ability to accurately identify VPN traffic.

[0069] In the more challenging scenario C, faced with a complex environment of 120 types of known TLS services and 20 types of unknown traffic, the FPR95 of the present invention was only 7.29%, while the AUROC increased to 97.89% and the AUPR was 97.32%. These results were suboptimal and slightly behind the MaxLogit method. In the unknown traffic detection task of scenario D, the three indicators of the present invention were comprehensively ahead of the four comparison methods, with an FPR95 as low as 1.48%, an AUROC of 99.48%, and an AUPR of 99.44%, achieving extremely high unknown traffic detection performance on the ISCX-VPN dataset.

[0070] Experimental data show that the present invention significantly outperforms existing methods in different types of open set scenarios, especially in the detection of malicious traffic and VPN traffic, providing a breakthrough technical solution for network security protection, and successfully expanding the network traffic classification model trained in closed set scenarios to open set scenarios, thereby enhancing the adaptability of the network traffic classification model to the open network environment.

Claims

1. A method for expanding open-set scenarios of a closed-set network traffic classification model, characterized in that: It includes a traffic feature space reconstruction phase and an unknown traffic detection phase; the traffic feature space reconstruction phase includes the following steps: Step 1) Preprocess the original network traffic data of known traffic, reorganize the flow samples based on the quintuples to generate unique flow samples, and extract the sequence information of the flow samples to generate a flow sample set; Step 2) Input the flow samples preprocessed in step 1) into the network traffic classification model for feature extraction to obtain deep-level features of the traffic; Step 3) Initialize a hypersphere mapping model to map the deep features obtained in step 2) to a low-dimensional hypersphere embedding space, and perform strict normalization on the mapped feature vectors to complete the initialization mapping; Step 4) Use the VMF distribution to model the characteristic distribution of multi-category network traffic in the initial hypersphere embedding space in step 3). Define the category prototype and concentration parameter for each category. The category prototype points to the center direction of the traffic category in the hypersphere space, and the concentration parameter controls the density of the distribution. Step 5) By designing a discrete loss function and an aggregate loss function, the initialized hypersphere mapping model in step 3) is optimized, and the feature distribution of the hypersphere embedding space is dynamically adjusted to ensure that intra-class features are tightly clustered and inter-class features are discrete. Finally, the optimized hypersphere mapping model is output. The unknown traffic detection phase includes the following steps: Step 6) The network flow to be tested undergoes the same preprocessing and feature extraction process as steps 1) and 2) to ensure consistency in the input space and obtain the deep features of the flow to be tested. The deep features of the flow to be tested are mapped to the reshaped hypersphere embedding space using the hypersphere mapping model optimized in step 5), and the feature vectors are normalized to preserve directional information. Step 7) Based on the feature vector of the hypersphere embedding space obtained in step 6), the K-nearest neighbor method is used to calculate the cosine distance between the traffic sample to be detected and the nearest known category sample as the distance score. Based on the dynamic judgment threshold rule and combined with the AUROC evaluation indicator, the network traffic classification model is evaluated to detect unknown traffic in the open set scenario.

2. The open set scenario expansion method of the closed set network traffic classification model according to claim 1 is characterized in that: The specific operation method of the pretreatment in step 1) is: Step 1-1) Reconstruct the original network traffic based on the five-tuple information of the network traffic to generate a unique flow sample. At the same time, perform data cleaning and abnormal flow filtering to ensure the quality of the input data. Step 1-2) uses the reconstructed flow sample obtained in step 1-1) to extract sequence information of the flow sample according to the architectural characteristics of the network traffic classification model trained in the closed set scenario.

3. The open set scenario expansion method of the closed set network traffic classification model according to claim 1 is characterized in that: The specific operation method of feature extraction in step 2) is: Step 2-1) Obtain a network traffic classification model trained using the same dataset as in step 1), treat the traffic in the specific dataset as known traffic, and use the trained model as a basic feature extractor for subsequent feature extraction; In step 2-2, the features of the flow sample sequence preprocessed in step 1-2) are input into the feature extractor in step 2-1). The deep network structure trained with a known flow set in a closed-set scenario is used to extract layer-by-layer features of the flow samples, extracting deep features for subsequent hypersphere embedding space mapping.

4. The open set scenario expansion method of the closed set network traffic classification model according to claim 1, characterized in that: The specific operation method for initializing the mapping in step 3) is: Step 3-1) Construct a two-layer fully connected neural network as the hypersphere mapping model. The first layer uses the ReLU activation function to implement nonlinear transformation, and the second layer is a linear mapping layer. The weight matrix uses the Xavier initialization method to ensure the stability of the feature distribution of each layer. Step 3-2) Input the deep features extracted in step 2-2) into the initialized hypersphere mapping model in step 3-1) and map them to a low-dimensional hypersphere embedding space through an output layer of multiple neurons; Step 3-3) Normalize the feature vectors mapped in step 3-2) and introduce a batch processing mechanism to convert them into unit vectors by dividing them by the L2 norm of the vectors. Add a small perturbation to prevent numerical instability caused by zero vectors.

5. The open set scenario expansion method of the closed set network traffic classification model according to claim 1 is characterized in that: The specific operation method of VMF distribution modeling in step 4) is: Step 4-1) Based on the traffic feature vector in the multi-category hypersphere embedding space obtained in step 3-3), a mixed VMF distribution is used to perform probabilistic modeling, and a unique category prototype vector is defined for each known category; Step 4-2) Based on the category prototypes obtained in step 4-1), a learnable centralized parameter is set for each category, and the parameter is automatically optimized through the back-propagation algorithm to control the compactness of the category distribution on the hypersphere; In step 4-3), the category prototypes and lumped parameters of steps 4-1) and 4-2) are used to establish a probability density function to describe the distribution of sample points on the hypersphere. The smoothness of the distribution is adjusted by the temperature parameter so that it can adapt to the distribution characteristics of different categories.

6. The open set scenario expansion method of the closed set network traffic classification model according to claim 1, characterized in that: The specific operation method for optimizing the hypersphere embedding space in step 5) is: Step 5-1) Design a discrete loss function based on cosine similarity. By maximizing the angular distance between prototype vectors of different categories, the categories are uniformly distributed on the hypersphere, avoiding overlapping areas between categories. Step 5-2) Construct an aggregation loss function based on cosine similarity. By minimizing the directional deviation between similar samples and the category prototype, similar samples are forced to cluster tightly on the hypersphere to form compact category clusters. Step 5-3) By combining the two losses in step 5-1) and step 5-2), an optimizer is used for end-to-end training to optimize the feature distribution of the hypersphere embedding space.

7. The open set scenario expansion method of the closed set network traffic classification model according to claim 1, characterized in that: The specific operation method of calculating the distance score and detecting unknown traffic in step 7) is: Step 7-1) Build an efficient K-nearest neighbor search algorithm. Based on the feature distribution in the hypersphere embedding space optimized in Step 5-3), use the spherical distance metric to calculate the cosine similarity between the sample to be tested and all known class samples, and find the K known samples closest to the sample to be tested, thereby improving search efficiency. In step 7-2, a dynamic threshold determination mechanism is designed. By analyzing the distance score distribution obtained in step 6-1), the optimal determination threshold is automatically determined. Simultaneously, multiple indicators, including FPR95, AUROC, and AUPR, are calculated to implement a comprehensive multi-performance indicator evaluation framework. This allows for a comprehensive evaluation of the unknown traffic detection effect. The detection results are presented as a distance score-density map, providing visual analysis results.

8. The open set scenario expansion method of the closed set network traffic classification model according to claim 2, characterized in that: The specific operation method of the pretreatment described in step 1-1) is: Step 1-1-1) Use specialized network traffic parsing software to quickly read and parse the original network traffic pcap format file, reconstruct the original network traffic based on the standard five-tuple, and generate unique flow samples. Each flow sample is identified by the five-tuple information, ensuring the uniqueness and traceability of the network session and achieving accurate traffic reassembly processing; Step 1-1-2) adds an abnormal traffic filtering module to the process of step 1-1-1) to automatically identify and filter irregular and abnormal DNS query traffic through a predefined rule set; Step 1-1-3) uses a multi-threaded parallel processing architecture to distribute the traffic reassembly task in step 1-1-1) to multiple computing cores for simultaneous execution, thereby improving processing throughput.

9. The open set scenario expansion method of the closed set network traffic classification model according to claim 3, characterized in that: The specific operation method of feature extraction described in step 2-2) is: Step 2-2-1) uses the flow sample obtained in step 1-2) as input, performs a true prediction of the network traffic classification model during the feature extraction process, extracts features layer by layer from the flow sample, intercepts the intermediate layer of the network traffic classification model feature extraction, and stops the calculation before obtaining the final classification result. Without affecting the original network traffic classification model structure, the feature output of the second-to-last layer is accurately obtained, and this output is used as the deep-level feature of the flow sample after passing through the feature extractor; Step 2-2-2) uses the feature dimension alignment component for the feature extraction in step 2-2-1) to automatically adjust the feature dimensions output by different models to ensure compatibility with the subsequent hypersphere mapping model; Step 2-2-3) implements cache management for the deep features obtained in step 2-2-2) and compresses and stores them to support fast reading and subsequent processing.

10. An open set scenario expansion system for a closed set network traffic classification model implementing the method according to any one of claims 1 to 9, characterized in that: Including traffic feature space reconstruction module and unknown traffic detection module; The traffic feature space reconstruction module is used to pre-process known traffic and use the network traffic classification model trained on the known traffic set to extract features to obtain deep features. The deep features are then mapped to the hypersphere embedding space and VMF distribution modeling is performed. The discrete loss and aggregate loss are optimized through joint loss to obtain a hypersphere mapping model, completing the feature space reconstruction. The unknown traffic detection module is used to preprocess and extract features of the traffic to be detected, and map the traffic to be detected to the reshaped hypersphere embedding space. By calculating the distance score, the unknown traffic is detected based on the dynamic judgment threshold rule.

Images