A flow detection method and apparatus
By combining a semi-supervised learning framework with supervised and unsupervised training to generate a masked autoencoder, the problem of poor robustness of existing traffic detection methods to a large number of labeled samples and adversarial samples is solved, and efficient and robust traffic detection is achieved.
Patent Information
- Application Number
- CN202311153499.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-07
- Publication Date
- 2026-02-24
- Estimated Expiration
- 2043-09-07
AI Technical Summary
Existing traffic detection methods require a large number of high-quality labeled samples for training, which is labor-intensive and has poor robustness to adversarial attacks, resulting in low detection efficiency.
A semi-supervised learning framework is adopted, which combines supervised and unsupervised training. Labeled and unlabeled images are used to train samples to generate a masked autoencoder. By combining mask matrix processing and loss function optimization, traffic features are extracted and classified.
It reduces manpower consumption, improves the robustness of the model against adversarial attacks, and enhances the efficiency and accuracy of traffic detection.
Smart Images
Figure CN117173510B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and more specifically, to a flow detection method and apparatus. Background Technology
[0002] Currently, network traffic classification based on machine learning is a popular research direction. Supervised training models require a large number of high-quality labeled samples for training, and extracting effective traffic features also has a significant impact on the model. Existing traffic detection methods typically first construct a data dimension feature value table based on the data dimensions and dimensional feature values of each sample data. Then, an initial autoencoder is trained based on the sample data in the data dimension feature value table to obtain an autoencoder that meets the model's accuracy requirements. Next, a classifier is trained based on the dimensionality-reduced feature data output by the autoencoder. Finally, the trained classifier is used for traffic classification. However, in practice, it has been found that existing methods require a large number of accurate labeled samples for training, consuming a significant amount of manpower. Furthermore, they have poor robustness against adversarial attacks, thus reducing the efficiency of traffic detection. Summary of the Invention
[0003] The purpose of this application is to provide a traffic detection method and apparatus that can train a model using a small number of labeled samples, reduce manpower consumption, and improve the robustness of the model under adversarial sample attacks, thereby improving the efficiency of traffic detection.
[0004] The first aspect of this application provides a traffic detection method, including:
[0005] Obtain the original encoder model, training samples, and a preset classifier; wherein the training samples include labeled first image training samples and unlabeled second image training samples;
[0006] The first encoder is obtained by supervising the original encoder model using the first image training samples.
[0007] The second encoder is obtained by unsupervised training using the second image training samples.
[0008] The second encoder is optimized to obtain a mask autoencoder;
[0009] Obtain the raw traffic to be detected;
[0010] The original flow rate is preprocessed to obtain the preprocessed flow rate;
[0011] The preprocessed traffic is identified by the mask autoencoder and the classifier to obtain the traffic identification result.
[0012] A traffic detection report is generated based on the traffic identification results.
[0013] In the above implementation process, this method can prioritize acquiring the original encoder model, labeled first image training samples, unlabeled second image training samples, and a pre-defined classifier. Therefore, the first step of this method is to acquire basic data and the model to facilitate subsequent steps. After obtaining the labeled first image training samples and the unlabeled second image training samples, the method can use the first image training samples to perform supervised training on the original encoder model to obtain the first encoder; and use the second image training samples to perform unsupervised training on the first encoder to obtain the second encoder. Thus, this method can obtain semi-supervised training results by supervising the pre-trained encoder and then further training it again through unsupervised training. At this point, optimizing the second encoder yields a mask autoencoder, achieving a high-quality mask autoencoder generation effect. After obtaining the aforementioned mask autoencoder, the method proceeds to the formal operation. First, it acquires the raw traffic to be detected. Then, it preprocesses the raw traffic to obtain preprocessed traffic. Next, it uses the mask autoencoder and classifier to identify the preprocessed traffic, obtaining the traffic identification result. Finally, it generates a traffic detection report based on the traffic identification result. It is evident that this method can more efficiently extract key features of traffic based on a trained mask autoencoder. Facing misleading and deceptive traffic features, it identifies traffic categories based on limited key features, thus exhibiting good robustness and anti-interference capabilities, thereby ensuring effective traffic classification.
[0014] Further, the step of using the first image training samples to perform supervised training on the original encoder model to obtain the first encoder includes:
[0015] The first image training sample is partially randomly masked using a preset mask matrix to obtain the first processed sample.
[0016] Obtain the image portion sample that is not covered in the first processed sample;
[0017] The original encoder model is trained using partial samples of the image to obtain the first encoder.
[0018] In the above implementation process, during supervised training, this method can perform partial random masking on the first image training samples using a preset mask matrix to obtain the first processed samples; then, it obtains the unmasked image portions from the first processed samples; finally, it trains the original encoder model using these image portions to obtain the first encoder. It is evident that this method enables the original encoder to learn image features from the unmasked portions, while simultaneously using content padding and reconstruction to determine the final supervised loss, thereby determining the pre-trained first encoder.
[0019] Further, the step of using the second image training samples to perform unsupervised training on the first encoder to obtain the second encoder includes:
[0020] The second image training samples are masked to obtain the second processed samples; wherein, the second processed samples include weakly masked samples and strongly masked samples.
[0021] The weak masking samples and the strong masking samples are input into the first encoder for unsupervised training to obtain the second encoder.
[0022] In the above implementation process, during unsupervised training based on the first encoder, this method can preferentially mask the second image training samples to obtain second processed samples including weakly masked samples and strongly masked samples. Then, the weakly masked samples and strongly masked samples are input into the first encoder for unsupervised training to obtain the second encoder. It can be seen that this method can perform bidirectional prediction based on weak and strong masks, and select the cross-entropy loss function as the loss function at the end of the prediction, thereby determining the second encoder.
[0023] Further, the optimization of the second encoder to obtain a mask autoencoder includes:
[0024] Obtain the supervised loss function for supervised training of the original encoder model and the unsupervised loss function for unsupervised training of the original encoder model;
[0025] Based on the supervised loss function and the unsupervised loss function, construct the overall loss function for semi-supervised learning;
[0026] The second encoder is optimized based on the overall loss function to obtain a masked autoencoder.
[0027] In the above implementation process, when constructing a semi-supervised learning mask autoencoder, this method can combine supervised and unsupervised loss functions to complete the encoder optimization process and obtain the mask autoencoder. It is evident that this method automatically adjusts the two loss functions, enabling the mask autoencoder determined by the adjusted loss functions to acquire higher-quality features.
[0028] Further, the preprocessing of the original traffic to obtain preprocessed traffic includes:
[0029] The original traffic is subjected to grouping processing to obtain a first processed traffic file;
[0030] The first processed traffic file is processed by deleting redundant data packets to obtain the second processed traffic file;
[0031] The second processed traffic file is truncated according to the preset traffic length to obtain multiple truncated traffic files;
[0032] The truncated traffic file and the Xining byte sequence are normalized to obtain preprocessed traffic; wherein, the preprocessed traffic is a grayscale image of a preset size.
[0033] In the above implementation process, when the original traffic is preprocessed to obtain preprocessed traffic, the method can perform preprocessing such as grouping and deleting redundant data packets. Then, each stream is truncated to a fixed byte length. Finally, the traffic with the uniform length is normalized according to the byte sequence and converted into a grayscale image of size M×M, thereby obtaining a standardized image suitable for processing.
[0034] Further, the step of performing traffic identification on the preprocessed traffic using the mask autoencoder and the classifier to obtain traffic identification results includes:
[0035] The preprocessed traffic is masked to obtain masked traffic data to be processed;
[0036] The target traffic features are obtained by extracting features from the masked traffic data using the masked autoencoder.
[0037] The target traffic features are input into a pre-built classifier for traffic identification, and the traffic identification results are obtained.
[0038] In the above implementation process, when the method identifies the preprocessed traffic using the mask autoencoder and the classifier to obtain the traffic identification result, it can first perform masking processing on the preprocessed traffic to obtain masked traffic data to be processed. Then, the mask autoencoder extracts features from the masked traffic data to obtain hidden layer features. Finally, flow identification is performed based on the hidden layer features to obtain the traffic identification result. It is evident that this method can achieve better traffic classification results based on hidden layer features.
[0039] A second aspect of this application provides a flow detection device, the flow detection device comprising:
[0040] The first acquisition unit is used to acquire the original encoder model, training samples, and a preset classifier; wherein, the training samples include labeled first image training samples and unlabeled second image training samples;
[0041] The first training unit is used to perform supervised training on the original encoder model using the first image training samples to obtain the first encoder.
[0042] The second training unit is used to perform unsupervised training on the first encoder using the second image training samples to obtain the second encoder.
[0043] An optimization unit is used to optimize the second encoder to obtain a mask autoencoder;
[0044] The second acquisition unit is used to acquire the raw traffic to be detected;
[0045] A preprocessing unit is used to preprocess the raw traffic to obtain preprocessed traffic;
[0046] A traffic identification unit is used to identify the preprocessed traffic through the mask autoencoder and the classifier to obtain a traffic identification result.
[0047] The generation unit is used to generate a traffic detection report based on the traffic identification results.
[0048] In the above implementation process, the device can first construct a mask autoencoder, and then perform traffic detection based on the mask autoencoder, thereby obtaining higher quality traffic classification results more stably.
[0049] Furthermore, the first training unit includes:
[0050] The first processing subunit is used to perform partial random masking on the first image training sample using a preset mask matrix to obtain the first processed sample.
[0051] The first acquisition subunit is used to acquire the image portion sample that is not covered in the first processed sample;
[0052] The first training subunit is used to train the original encoder model using partial samples of the image to obtain the first encoder.
[0053] Furthermore, the second training unit includes:
[0054] The second processing subunit is used to perform masking processing on the second image training samples to obtain a second processed sample; wherein, the second processed sample includes a weakly masked sample and a strongly masked sample.
[0055] The second training subunit is used to input the weak masking samples and the strong masking samples into the first encoder for unsupervised training to obtain the second encoder.
[0056] Furthermore, the optimization unit includes:
[0057] The second acquisition subunit is used to acquire the supervised loss function when performing supervised training on the original encoder model and the unsupervised loss function when performing unsupervised training on the original encoder model.
[0058] Construct sub-units to build an overall loss function for semi-supervised learning based on the supervised loss function and the unsupervised loss function;
[0059] An optimization subunit is used to optimize the second encoder based on the overall loss function to obtain a masked autoencoder.
[0060] Furthermore, the preprocessing unit is specifically used to perform grouping processing on the original traffic to obtain a first processed traffic file;
[0061] The preprocessing unit is further configured to perform redundant data packet deletion on the first processed traffic file to obtain a second processed traffic file;
[0062] The preprocessing unit is further configured to perform stream truncation processing on the second processed stream file according to a preset stream length to obtain multiple truncated stream files;
[0063] The preprocessing unit is further configured to normalize the truncated traffic file and the Xining byte sequence to obtain preprocessed traffic; wherein the preprocessed traffic is a grayscale image of a preset size.
[0064] Furthermore, the traffic identification unit includes:
[0065] The third processing subunit is used to perform masking on the preprocessed traffic to obtain masked traffic data to be processed.
[0066] The feature extraction subunit is used to extract features from the masked traffic data through the mask autoencoder to obtain target traffic features;
[0067] The traffic identification subunit is used to input the target traffic features into a pre-built classifier for traffic identification and obtain the traffic identification result.
[0068] A third aspect of this application provides an electronic device including a memory and a processor, the memory storing a computer program, and the processor running the computer program to cause the electronic device to perform the traffic detection method described in any one of the first aspects of this application.
[0069] The fourth aspect of this application provides a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the flow detection method described in any one of the first aspects of this application. Attached Figure Description
[0070] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0071] Figure 1 A flowchart illustrating a traffic detection method provided in an embodiment of this application;
[0072] Figure 2 A flowchart illustrating another traffic detection method provided in an embodiment of this application;
[0073] Figure 3 This is a schematic diagram of the structure of a flow detection device provided in an embodiment of this application;
[0074] Figure 4 This is a schematic diagram of another flow detection device provided in an embodiment of this application;
[0075] Figure 5 A flowchart illustrating an example of semi-supervised traffic classification based on mask autoencoder and image recognition technology, provided for embodiments of this application;
[0076] Figure 6 An example flowchart of traffic preprocessing provided in this application embodiment;
[0077] Figure 7 A schematic diagram illustrating the training process of a labeled sample pre-trained masked autoencoder provided in this application embodiment;
[0078] Figure 8 This is a schematic diagram of an unsupervised training phase provided in an embodiment of this application. Detailed Implementation
[0079] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.
[0080] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0081] Example 1
[0082] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a traffic detection method provided in this embodiment. The traffic detection method includes:
[0083] S101. Obtain the original encoder model, training samples, and a preset classifier; wherein, the training samples include labeled first image training samples and unlabeled second image training samples.
[0084] S102. Supervised training of the original encoder model is performed using the first image training samples to obtain the first encoder.
[0085] S103. Use the second image training samples to perform unsupervised training on the first encoder to obtain the second encoder.
[0086] S104. Optimize the second encoder to obtain the mask autoencoder.
[0087] S105. Obtain the raw traffic to be detected.
[0088] S106. Preprocess the raw flow to obtain preprocessed flow.
[0089] S107. Traffic identification is performed on the preprocessed traffic using a mask autoencoder and a classifier to obtain the traffic identification result.
[0090] S108. Generate a traffic detection report based on the traffic identification results.
[0091] In this embodiment, the method actually provides a semi-supervised traffic classification method based on mask autoencoder and image recognition technology. This method, after converting traffic into grayscale images, uses a mask autoencoder to extract traffic features based on a semi-supervised learning framework, and finally inputs these features into a classifier to achieve traffic classification.
[0092] For example, this method mainly includes the following steps:
[0093] (1) Data preprocessing;
[0094] (2) Supervised training;
[0095] (3) Unsupervised training;
[0096] (4) Overall model optimization;
[0097] (5) Classifier identifies traffic.
[0098] It is easy to see that the above content corresponds to steps S101 to S108. For details, please refer to... Figure 5 , Figure 5 An example flowchart of a semi-supervised traffic classification based on mask autoencoder and image recognition technology is shown.
[0099] This implementation addresses the problem that current supervised learning-based traffic classification techniques rely on large amounts of accurate labeled data for training models. It proposes a method to train models using a small number of labeled samples, thereby reducing manpower consumption. Furthermore, to address the vulnerability of existing traffic classification techniques to adversarial attacks due to erroneous features, this proposed method focuses on improving the model's robustness against adversarial attacks.
[0100] In this embodiment, the subject executing the method can be a computing device such as a computer or server, and no limitation is made in this embodiment.
[0101] In this embodiment, the subject executing the method can also be a smart device such as a smartphone or tablet, and no limitation is made in this embodiment.
[0102] As can be seen, the traffic detection method described in this embodiment can efficiently extract key features of traffic. Faced with misleading or deceptive traffic features, this method can identify traffic categories based on limited key features, thus exhibiting good robustness and resistance to interference. Simultaneously, this method can be trained using a small amount of labeled data, reducing manpower consumption. Specifically, this method can employ a semi-supervised learning framework for traffic classification, using labeled data for supervised training and unlabeled data for unsupervised training to further optimize the model, thereby reducing the cost of manually labeling data. Furthermore, this method can also utilize the advanced FixMatch semi-supervised learning framework, allowing the autoencoder model to be adjusted and optimized by predicting unlabeled traffic samples after changes in strength, enabling the model to identify higher-quality, more representative, and more discriminative features.
[0103] Example 2
[0104] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a traffic detection method provided in this embodiment. The traffic detection method includes:
[0105] S201. Obtain the original encoder model, training samples, and a preset classifier; wherein, the training samples include labeled first image training samples and unlabeled second image training samples.
[0106] In this embodiment, the training samples can be pcap traffic files.
[0107] In this embodiment, the method can first preprocess the original pcap traffic file, including stream grouping and deleting redundant data packets. Then, stream truncation is performed, unifying each stream to a fixed byte length, padding with zeros at the end if necessary. Finally, the traffic with uniform length is normalized according to the byte sequence and converted into a grayscale image of size M×M.
[0108] In this implementation, the method can convert traffic data into grayscale images and extract features using the ViT model, a leading model in computer vision.
[0109] Please refer to Figure 6 , Figure 6 An example flowchart of traffic preprocessing is shown.
[0110] In this embodiment, the original encoder model is a labeled image pre-trained masked autoencoder (MAE). Specifically, the MAE selects the ViT model (Vision Transformer, a visual self-attention model) as its encoder.
[0111] S202. The first image training sample is partially randomly masked using a preset mask matrix to obtain the first processed sample.
[0112] In this embodiment, the method uses an M×M mask matrix Mask to randomly cover a portion of the image. A matrix of 0s and 1s is randomly generated with probability Pa according to a Bernoulli distribution, where Pa represents the probability of 1 appearing (which can be set to 65%). The portion of the image corresponding to 0 in the matrix is masked. The input image X and the mask matrix Mask are multiplied by a matrix multiplication operation ⊙ to obtain the randomly masked image X', i.e., X' = X⊙Mask.
[0113] S203. Obtain the image portion sample that is not covered in the first processing sample.
[0114] S204. Train the original encoder model using partial image samples to obtain the first encoder.
[0115] In this embodiment, the method can retain only the undisturbed portion of the image and input it into the ViT encoder so that ViT can learn the features of the image from the undisturbed portion.
[0116] In this embodiment, the method can use the masking described above for padding, merging the encoder output and the masking while preserving the original order, and using this as the decoder input; then, the masked portion is reconstructed by the decoder to restore the input traffic image, and the error between the reconstructed image and the original image is denoted as the supervised loss L. label The first encoder can be determined based on this supervised loss. By implementing this method, the ViT encoder is able to learn labeled data from the unmasked portion of the input.
[0117] The key characteristics of the data flow enable the decoder to effectively reconstruct the original input. For details, please refer to... Figure 7 The diagram shows the training process of a pre-trained masked autoencoder with labeled samples.
[0118] S205. Perform masking processing on the second image training samples to obtain the second processed samples; wherein, the second processed samples include weakly masked samples and strongly masked samples.
[0119] In this embodiment, the method first adjusts the Pa value of the mask matrix to perform weak and strong masking on the unlabeled traffic image. The weak mask has Pa = 75%, preserving more features of the original image; the strong mask has Pa = 35%, preserving fewer features, with most of the image being obscured. Compared to the strongly masked image, the model is more likely to recognize the weakly masked image.
[0120] S206. Input the weakly masked samples and the strongly masked samples into the first encoder for unsupervised training to obtain the second encoder.
[0121] In this embodiment, the method inputs an unlabeled weakly masked image into the pre-trained ViT encoder from the previous step, uses the decoder to reconstruct and predict the image, and takes the class with the highest predicted probability as the pseudo-label for that sample. Simultaneously, it is also necessary to ensure that the pseudo-label is statistically significant. The hyperparameter τ is the confidence level, set at 95%. If the confidence level condition is met, the model's prediction of the weakly masked image is considered reliable.
[0122] In this embodiment, the autoencoder also predicts the strongly masked image, using pseudo-labels that meet the confidence criteria as a reference standard. The error between the predicted label and the pseudo-label is denoted as the unsupervised loss Lunlabel, and the cross-entropy loss function (CE) is chosen as the loss function.
[0123]
[0124] The I(·) function is 1 if it meets the threshold condition, and 0 otherwise.
[0125] Please refer to Figure 8 , Figure 8 A schematic diagram of the training process in the unsupervised training phase is shown.
[0126] S207. Obtain the supervised loss function for supervised training of the original encoder model and the unsupervised loss function for unsupervised training of the original encoder model.
[0127] In this embodiment, the method can use unlabeled traffic samples to train the encoder in an unsupervised manner, and can be referred to the advanced semi-supervised learning framework FixMatch.
[0128] S208. Based on supervised and unsupervised loss functions, construct the overall loss function for semi-supervised learning.
[0129] In this embodiment, the method can be combined with supervised loss L label And unsupervised loss L unlabel Construct the overall loss function L = L for semi-supervised learning. label +ɑL unlabel Here, the hyperparameter α represents the loss weight, adjusting the proportion of the two loss values in the overall loss function. Based on the overall loss function L, the model is optimized, enabling the ViT encoder to learn higher-quality features.
[0130] S209. Optimize the second encoder based on the overall loss function to obtain the mask autoencoder.
[0131] S210, Obtain the raw traffic to be detected.
[0132] In this embodiment, the original traffic can be a pcap traffic file.
[0133] S211. Perform group flow processing on the raw traffic to obtain the first processed traffic file.
[0134] S212. Redundant data packets are deleted from the first processed traffic file to obtain the second processed traffic file.
[0135] S213. Perform flow truncation processing on the second processing flow file according to the preset flow length to obtain multiple truncated flow files.
[0136] S214. Normalize the truncated traffic file and the Xining byte sequence to obtain the preprocessed traffic; wherein, the preprocessed traffic is a grayscale image of a preset size.
[0137] In this embodiment, the method can first preprocess the original pcap traffic file, including stream grouping and deleting redundant data packets. Then, stream truncation is performed, unifying each stream to a fixed byte length, padding with zeros at the end if necessary. Finally, the traffic with uniform length is normalized according to the byte sequence and converted into a grayscale image of size M×M.
[0138] S215. Perform masking on the preprocessed traffic to obtain masked traffic data to be processed.
[0139] S216. Extract features from the masked traffic data using a mask autoencoder to obtain the target traffic features.
[0140] S217. Input the target traffic features into a pre-built classifier to identify the traffic and obtain the traffic identification results.
[0141] In this embodiment, the method can output the hidden layer features of the ViT encoder to a classifier for traffic classification.
[0142] In this embodiment, the method uses an MLP network. After the features are input into the fully connected layer, the predicted value can be transformed into a probability distribution by combining the softmax function, and the label with the highest probability is taken as the category of the traffic sample.
[0143] S218. Generate a traffic detection report based on the traffic identification results.
[0144] For example, this method can be applied to various traffic detection software products (software, plugins, etc.). Specifically, this example illustrates how to dynamically detect traffic to improve detection accuracy and performance. The specific process includes:
[0145] (1) Collect labeled traffic samples and perform preprocessing;
[0146] (2) Use labeled samples to pretrain a masked autoencoder (MAE) model and save the model parameters and weights;
[0147] (3) The computer instructions developed based on the present invention are stored in a computer-readable storage medium, and the pre-trained MAE model is integrated into the traffic detection software;
[0148] (4) Use tools such as Wireshark to collect the traffic to be detected and perform preprocessing;
[0149] (5) Strong masking and weak masking are applied to the traffic to be detected. The traffic samples after masking are input into the MAE encoder to extract traffic features.
[0150] (6) Input the hidden layer features of the MAE encoder into the classifier for traffic identification;
[0151] (7) Output a detection report, including traffic type, traffic volume, detection time, etc.
[0152] In this embodiment, the subject executing the method can be a computing device such as a computer or server, and no limitation is made in this embodiment.
[0153] In this embodiment, the subject executing the method can also be a smart device such as a smartphone or tablet, and no limitation is made in this embodiment.
[0154] As can be seen, the traffic detection method described in this embodiment can efficiently extract key features of traffic. Faced with misleading or deceptive traffic features, this method can identify traffic categories based on limited key features, thus exhibiting good robustness and resistance to interference. Simultaneously, this method can be trained using a small amount of labeled data, reducing manpower consumption. Specifically, this method can employ a semi-supervised learning framework for traffic classification, using labeled data for supervised training and unlabeled data for unsupervised training to further optimize the model, thereby reducing the cost of manually labeling data. Furthermore, this method can also utilize the advanced FixMatch semi-supervised learning framework, allowing the autoencoder model to be adjusted and optimized by predicting unlabeled traffic samples after changes in strength, enabling the model to identify higher-quality, more representative, and more discriminative features.
[0155] Example 3
[0156] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of a flow detection device provided in this embodiment. Figure 3 As shown, the flow detection device includes:
[0157] The first acquisition unit 310 is used to acquire the original encoder model, training samples and a preset classifier; wherein, the training samples include labeled first image training samples and unlabeled second image training samples;
[0158] The first training unit 320 is used to perform supervised training on the original encoder model using the first image training samples to obtain the first encoder.
[0159] The second training unit 330 is used to perform unsupervised training on the first encoder using the second image training samples to obtain the second encoder.
[0160] The optimization unit 340 is used to optimize the second encoder to obtain a mask autoencoder;
[0161] The second acquisition unit 350 is used to acquire the raw traffic to be detected;
[0162] The preprocessing unit 360 is used to preprocess the raw traffic to obtain preprocessed traffic;
[0163] The traffic identification unit 370 is used to identify the traffic of the preprocessed traffic through a mask autoencoder and a classifier to obtain the traffic identification result.
[0164] The generation unit 380 is used to generate a traffic detection report based on the traffic identification results.
[0165] In this embodiment, the explanation of the flow detection device can be referred to the description in Embodiment 1 or Embodiment 2, and will not be repeated here.
[0166] As can be seen, the traffic detection device described in this embodiment can efficiently extract key features of traffic. Faced with misleading or deceptive traffic features, the device can identify traffic categories based on limited key features, thus exhibiting good robustness and anti-interference capabilities. Simultaneously, the device can be trained using a small amount of labeled data, reducing manpower consumption. Specifically, the device can employ a semi-supervised learning framework for traffic classification, using labeled data for supervised training and unlabeled data for unsupervised training to further optimize the model, thereby reducing the cost of manually labeling data. Furthermore, the device can also utilize the advanced FixMatch semi-supervised learning framework, allowing the autoencoder model to be adjusted and optimized by predicting unlabeled traffic samples after changes in strength, enabling the model to identify higher-quality, more representative, and more discriminative features.
[0167] Example 4
[0168] Please refer to Figure 4 , Figure 4This is a schematic diagram of the structure of a flow detection device provided in this embodiment. Figure 4 As shown, the flow detection device includes:
[0169] The first acquisition unit 310 is used to acquire the original encoder model, training samples and a preset classifier; wherein, the training samples include labeled first image training samples and unlabeled second image training samples;
[0170] The first training unit 320 is used to perform supervised training on the original encoder model using the first image training samples to obtain the first encoder.
[0171] The second training unit 330 is used to perform unsupervised training on the first encoder using the second image training samples to obtain the second encoder.
[0172] The optimization unit 340 is used to optimize the second encoder to obtain a mask autoencoder;
[0173] The second acquisition unit 350 is used to acquire the raw traffic to be detected;
[0174] The preprocessing unit 360 is used to preprocess the raw traffic to obtain preprocessed traffic;
[0175] The traffic identification unit 370 is used to identify the traffic of the preprocessed traffic through a mask autoencoder and a classifier to obtain the traffic identification result.
[0176] The generation unit 380 is used to generate a traffic detection report based on the traffic identification results.
[0177] As an optional implementation, the first training unit 320 includes:
[0178] The first processing subunit 321 is used to perform partial random masking processing on the first image training sample through a preset mask matrix to obtain the first processed sample.
[0179] The first acquisition subunit 322 is used to acquire the image portion sample that is not covered in the first processing sample;
[0180] The first training subunit 323 is used to train the original encoder model using partial image samples to obtain the first encoder.
[0181] As an optional implementation, the second training unit 330 includes:
[0182] The second processing subunit 331 is used to perform masking processing on the second image training samples to obtain the second processed samples; wherein, the second processed samples include weak masking samples and strong masking samples.
[0183] The second training subunit 332 is used to input weakly masked samples and strongly masked samples into the first encoder for unsupervised training to obtain the second encoder.
[0184] As an optional implementation, the optimization unit 340 includes:
[0185] The second acquisition subunit 341 is used to acquire the supervised loss function when performing supervised training on the original encoder model and the unsupervised loss function when performing unsupervised training on the original encoder model.
[0186] Subunit 342 is constructed to build the overall loss function for semi-supervised learning based on supervised and unsupervised loss functions;
[0187] The optimization subunit 343 is used to optimize the second encoder based on the overall loss function to obtain the mask autoencoder.
[0188] As an optional implementation, the preprocessing unit 360 is specifically used to perform grouping processing on the original traffic to obtain a first processed traffic file;
[0189] The preprocessing unit 360 is specifically used to perform redundant data packet deletion on the first processed traffic file to obtain the second processed traffic file;
[0190] The preprocessing unit 360 is further configured to perform stream truncation processing on the second processed stream file according to the preset stream length, so as to obtain multiple truncated stream files;
[0191] The preprocessing unit 360 is specifically used to normalize the truncated traffic file and the Xining byte sequence to obtain preprocessed traffic; wherein, the preprocessed traffic is a grayscale image of a preset size.
[0192] As an optional implementation, the traffic identification unit 370 includes:
[0193] The third processing subunit 371 is used to perform masking on the preprocessed traffic to obtain masked traffic data to be processed.
[0194] The feature extraction subunit 372 is used to extract features from the masked traffic data through the mask autoencoder to obtain the target traffic features;
[0195] The traffic identification subunit 373 is used to input the target traffic features into a pre-built classifier for traffic identification and obtain the traffic identification result.
[0196] In this embodiment, the explanation of the flow detection device can be referred to the description in Embodiment 1 or Embodiment 2, and will not be repeated here.
[0197] As can be seen, the traffic detection device described in this embodiment can efficiently extract key features of traffic. Faced with misleading or deceptive traffic features, the device can identify traffic categories based on limited key features, thus exhibiting good robustness and anti-interference capabilities. Simultaneously, the device can be trained using a small amount of labeled data, reducing manpower consumption. Specifically, the device can employ a semi-supervised learning framework for traffic classification, using labeled data for supervised training and unlabeled data for unsupervised training to further optimize the model, thereby reducing the cost of manually labeling data. Furthermore, the device can also utilize the advanced FixMatch semi-supervised learning framework, allowing the autoencoder model to be adjusted and optimized by predicting unlabeled traffic samples after changes in strength, enabling the model to identify higher-quality, more representative, and more discriminative features.
[0198] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the traffic detection method in embodiment 1 or embodiment 2 of this application.
[0199] This application provides a computer-readable storage medium storing computer program instructions, which are read and executed by a processor to perform the traffic detection method in embodiment 1 or embodiment 2 of this application.
[0200] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0201] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0202] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0203] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0204] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0205] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
Claims
1. A flow rate detection method, characterized in that, include: Obtain the original encoder model, training samples, and a preset classifier; wherein the training samples include labeled first image training samples and unlabeled second image training samples; The first encoder is obtained by supervising the original encoder model using the first image training samples. The second encoder is obtained by unsupervised training using the second image training samples. The second encoder is optimized to obtain a mask autoencoder; Obtain the raw traffic to be detected; The original flow rate is preprocessed to obtain the preprocessed flow rate; The preprocessed traffic is identified by the mask autoencoder and the classifier to obtain the traffic identification result. A traffic detection report is generated based on the traffic identification results; The step of using the second image training samples to perform unsupervised training on the first encoder to obtain the second encoder includes: The second image training samples are masked using the semi-supervised learning framework FixMatch to obtain the second processed samples; wherein, the second processed samples include weakly masked samples and strongly masked samples. Using the semi-supervised learning framework FixMatch, the weakly masked samples and the strongly masked samples are input into the first encoder for unsupervised training to obtain the second encoder.
2. The flow detection method according to claim 1, characterized in that, The step of performing supervised training on the original encoder model using the first image training samples to obtain the first encoder includes: The first image training sample is partially randomly masked using a preset mask matrix to obtain the first processed sample. Obtain the image portion sample that is not covered in the first processed sample; The original encoder model is trained using partial samples of the image to obtain the first encoder.
3. The flow detection method according to claim 1, characterized in that, The optimization of the second encoder to obtain a mask autoencoder includes: Obtain the supervised loss function for supervised training of the original encoder model and the unsupervised loss function for unsupervised training of the original encoder model; Based on the supervised loss function and the unsupervised loss function, construct the overall loss function for semi-supervised learning; The second encoder is optimized based on the overall loss function to obtain a masked autoencoder.
4. The flow detection method according to claim 1, characterized in that, The preprocessing of the original traffic to obtain preprocessed traffic includes: The original traffic is subjected to grouping processing to obtain a first processed traffic file; The first processed traffic file is processed by deleting redundant data packets to obtain the second processed traffic file; The second processed traffic file is truncated according to the preset traffic length to obtain multiple truncated traffic files; The truncated traffic file and the Xining byte sequence are normalized to obtain preprocessed traffic; wherein, the preprocessed traffic is a grayscale image of a preset size.
5. The flow detection method according to claim 1, characterized in that, The step of performing traffic identification on the preprocessed traffic using the mask autoencoder and the classifier to obtain traffic identification results includes: The preprocessed traffic is masked to obtain masked traffic data to be processed; The target traffic features are obtained by extracting features from the masked traffic data using the masked autoencoder. The target traffic features are input into a pre-built classifier for traffic identification, and the traffic identification results are obtained.
6. A flow detection device, characterized in that, The flow detection device includes: The first acquisition unit is used to acquire the original encoder model, training samples, and a preset classifier; wherein, the training samples include labeled first image training samples and unlabeled second image training samples; The first training unit is used to perform supervised training on the original encoder model using the first image training samples to obtain the first encoder. The second training unit is used to perform unsupervised training on the first encoder using the second image training samples to obtain the second encoder. An optimization unit is used to optimize the second encoder to obtain a mask autoencoder; The second acquisition unit is used to acquire the raw traffic to be detected; A preprocessing unit is used to preprocess the raw traffic to obtain preprocessed traffic; A traffic identification unit is used to identify the preprocessed traffic through the mask autoencoder and the classifier to obtain a traffic identification result. The generation unit is used to generate a traffic detection report based on the traffic identification results; The second training unit includes: The second processing subunit is used to perform masking processing on the second image training samples using the semi-supervised learning framework FixMatch to obtain the second processed samples; wherein, the second processed samples include weakly masked samples and strongly masked samples. The second training subunit is used to input weakly masked samples and strongly masked samples into the first encoder for unsupervised training using the semi-supervised learning framework FixMatch, thus obtaining the second encoder.
7. The flow detection device according to claim 6, characterized in that, The first training unit includes: The first processing subunit is used to perform partial random masking on the first image training sample using a preset mask matrix to obtain the first processed sample. The first acquisition subunit is used to acquire the image portion sample that is not covered in the first processed sample; The first training subunit is used to train the original encoder model using partial samples of the image to obtain the first encoder.
8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor running the computer program to cause the electronic device to perform the flow detection method according to any one of claims 1 to 5.
9. A readable storage medium, characterized in that, The readable storage medium stores computer program instructions, which, when read and executed by a processor, perform the flow detection method according to any one of claims 1 to 5.