Traffic identification method, device, electronic device and storage medium

Abstract

The embodiments of the present invention relate to the field of communications, and disclose a traffic identification method, device, electronic device, and storage medium. In the present invention, the method includes: determining the probability that the data stream to be identified belongs to each preset traffic type; wherein the probability of belonging to each preset traffic type is recorded as the belonging probability of each preset traffic type; comparing the relationship between the belonging probability of each preset traffic type and the preset probability threshold of each preset traffic type; if there is no preset traffic type with a belonging probability greater than or equal to the preset probability threshold in each preset traffic type, then the traffic type of the data stream to be identified is identified as an unknown type. A probability threshold is set for each traffic type in advance, and according to the relationship between the belonging probability and the probability threshold, unknown traffic that does not belong to the preset traffic type is identified, thereby avoiding misidentification of the traffic type of the data stream to be identified in the open data set, and improving the accuracy and practicality of traffic identification.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of communications, and in particular to a traffic identification method, device, electronic device and storage medium. Background Art

[0002] The development of traffic identification has gone through several important stages. In the earliest stage, it was mainly based on the identification method of the five-tuple of source IP, source port, destination IP, destination port and protocol name. However, its identification ability is very sensitive to the dynamic IP and port changes in the network, and it can no longer meet the current traffic identification needs. The most widely used traffic method is the deep packet inspection method (DPI). DPI obtains traffic matching patterns, keywords and other message features by parsing messages to achieve traffic identification. However, it relies on the acquisition of plaintext information of messages, which makes its ability to identify encrypted traffic weak. When identifying encrypted traffic in related technologies, the optimal classification features are obtained by artificially constructing data flow features to achieve the identification of part of the encrypted traffic, or a fixed category of encrypted traffic is identified by constructing a one-dimensional convolutional neural network model. In the process of identifying encrypted traffic, the practicality and accuracy of traffic identification methods are poor. Summary of the invention

[0003] The purpose of the embodiments of the present invention is to provide a traffic identification method, device, electronic device and storage medium to achieve accurate and efficient identification of encrypted traffic in open set data and improve the practicality of traffic identification.

[0004] To solve the above technical problems, an embodiment of the present invention provides a traffic identification method, including: determining the probability that a data flow to be identified belongs to each preset traffic type; wherein the probability of belonging to each preset traffic type is recorded as the belonging probability of each preset traffic type; comparing the belonging probability of each preset traffic type with the preset probability threshold of each preset traffic type; if there is no preset traffic type in each preset traffic type whose belonging probability is greater than or equal to the preset probability threshold, then the traffic type of the data flow to be identified is identified as an unknown type.

[0005] An embodiment of the present invention also provides a traffic identification device, including: a determination module, used to determine the probability that the data flow to be identified belongs to each preset traffic type; wherein the probability of belonging to each preset traffic type is recorded as the belonging probability of each preset traffic type; a comparison module, used to compare the relationship between the belonging probability of each preset traffic type and the preset probability threshold of each preset traffic type; an identification module, used to determine the traffic type of the data flow to be identified, if there is no preset traffic type among the preset traffic types whose belonging probability is greater than or equal to the preset probability threshold, then the traffic type of the data flow to be identified is identified as an unknown type.

[0006] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned traffic identification method.

[0007] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which implements the above-mentioned traffic identification method when executed by a processor.

[0008] Compared with the prior art, the embodiment of the present invention obtains the probability that the data flow to be identified belongs to each preset traffic type, and records the probability of belonging to each preset traffic type as the belonging probability of each preset traffic type, compares the belonging probability of each preset traffic type and the preset probability threshold of each preset traffic type, and if there is no preset traffic type whose belonging probability is greater than or equal to the preset probability threshold, the traffic type of the data flow to be identified is identified as an unknown type. In the identification of traffic types in an open data set, by setting a preset probability threshold for each preset traffic type in advance, after obtaining the probability that the data flow to be identified belongs to each preset traffic type, according to the relationship between the belonging probability and the preset probability threshold, the unknown type of traffic that does not belong to any preset traffic type is identified, thereby avoiding misidentification of the traffic type of the data flow to be identified that does not belong to the preset traffic type as a known preset traffic type, thereby improving the accuracy and practicality of traffic identification.

[0009] In addition, after comparing the relationship between the attribution probability of each preset traffic type and the preset probability threshold of each preset traffic type, the method further includes: if among the preset traffic types, there is a preset traffic type whose attribution probability is greater than or equal to the preset probability threshold, then taking the preset traffic type with the largest attribution probability as the traffic type of the data flow to be identified. By taking the preset traffic type with the largest attribution probability as the traffic type of the data flow to be identified, the traffic type of the data flow to be identified can be identified as accurately as possible.

[0010] In addition, the preset probability threshold of each preset traffic type is calculated in the following way: create a one-to-many classification network of the preset traffic type; obtain the output probability of each training sample through the one-to-many classification network; wherein the training sample is a data stream belonging to the preset traffic type; fit the output probability of each training sample into a Gaussian distribution; and calculate the preset probability threshold of the preset traffic type according to the mean and standard deviation of the Gaussian distribution. The output probability of multiple training samples of the preset traffic type is obtained through the created one-to-many classification network, and the output probability is fitted into a Gaussian distribution, and the preset probability threshold is determined according to the mean and standard deviation of the Gaussian distribution, thereby improving the accuracy of the obtained preset probability threshold.

[0011] In addition, fitting the output probability of each training sample into a Gaussian distribution includes: fitting the output probability of each training sample into a Gaussian distribution with a mean of 1. By fitting the output probability into a Gaussian distribution with a mean of 1, the calculation of the preset probability threshold is simplified, the calculation efficiency of the preset probability threshold is improved, and the processing burden of the system is reduced.

[0012] In addition, determining the probability that the data stream to be identified belongs to each preset traffic type includes: obtaining the effective load data of the application layer from the unencrypted data packet of the data stream to be identified; segmenting the payload data to generate a natural language processing NLP word list of the data stream to be identified; and determining the probability that the data stream to be identified belongs to each preset traffic type through a pre-trained recognition model based on the NLP word list. By segmenting the acquired payload data to obtain the NLP word list of the data stream to be identified, and calculating the probability that the data stream to be identified belongs to each preset traffic type through a pre-trained recognition model, the traffic information of the data stream to be identified is fully utilized to improve the accuracy of the acquired flow characteristics, thereby improving the accuracy of the obtained attribution probability.

[0013] In addition, the recognition model is trained in the following manner: providing a dynamic word vector model for pre-training; pre-training the dynamic word vector model according to the first text data; wherein the pre-training includes unsupervised training; inputting the encoding network parameters of the pre-trained dynamic word vector model into the recognition model, and performing supervised training on the recognition model according to the second text data. By adopting a training method based on the pre-training and fine-tuning mechanism of the dynamic word vector model to obtain the recognition model, the recognition model's ability to identify the data flow characteristics to be identified and the type of traffic to which it belongs is improved, thereby improving the accuracy of traffic recognition.

[0014] In addition, the payload data is segmented, including segmenting the payload data using a binary grammar. By using a binary grammar to segment the payload data, the size of the word library is expanded, making the number of words contained in the word library closer to natural language, and further improving the accuracy of subsequent traffic type recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] One or more embodiments are exemplarily described by the pictures in the corresponding drawings, and these exemplary descriptions do not constitute limitations on the embodiments.

[0016] Figure 1 is a flow chart of a flow identification method according to a first embodiment of the present invention;

[0017] Figure 2is a flow chart of a method for calculating the attribution probability according to a first embodiment of the present invention;

[0018] Figure 3 is a schematic diagram of bigram word segmentation according to the first embodiment of the present invention;

[0019] Figure 4 is a schematic diagram of a recognition model training method according to a first embodiment of the present invention;

[0020] Figure 5 is a flow chart of a method for calculating a preset probability threshold according to a second embodiment of the present invention;

[0021] Figure 6 is a schematic structural diagram of a flow identification device according to a third embodiment of the present invention;

[0022] Figure 7 FIG. 1 is a schematic diagram of the structure of an electronic device according to a first embodiment of the present invention. DETAILED DESCRIPTION

[0023] To make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0024] The first embodiment of the present invention relates to a traffic identification method, which determines the probability that a data stream to be identified belongs to each preset traffic type; wherein the probability of belonging to each preset traffic type is recorded as the belonging probability of each preset traffic type; the belonging probability of each preset traffic type is compared with the size relationship of the preset probability threshold of each preset traffic type; if there is no preset traffic type whose belonging probability is greater than or equal to the preset probability threshold among the preset traffic types, then the traffic type of the data stream to be identified is identified as an unknown type. By setting a preset probability threshold for each preset traffic type in advance, after obtaining the probability that the data stream to be identified belongs to each preset traffic type, according to the relationship between the belonging probability and the preset probability threshold, the unknown type of traffic that does not belong to any preset traffic type is identified, thereby avoiding misidentification of the traffic type of the data stream to be identified, improving the accuracy of traffic identification, and enabling the traffic identification method to be applied in the identification of traffic types in open data sets, thereby improving the practicality of the traffic identification method.

[0025] The implementation details of the traffic identification method of this embodiment are described in detail below. The following content is only provided for easy understanding of the implementation details and is not necessary for implementing this solution.

[0026] The specific process of the flow identification method in the first embodiment of the present invention is as follows: Figure 1 As shown, the specific steps include:

[0027] Step 101, determining the probability that the data flow to be identified belongs to each preset traffic type.

[0028] Specifically, after receiving the traffic identification task, the identification system will receive a whole flow transmitted to the identification system, and then capture the input traffic through the data packet capture function, and store the captured data packets as a file in pcap format, and divide the input traffic into data streams to generate multiple data streams to be identified, so that the identification system can identify each data stream one by one or synchronously. During the identification process, the identification system determines the probability that the data stream to be identified belongs to each preset traffic type based on the traffic information and flow characteristics of the data stream to be identified, combined with the preset traffic type; wherein the probability of belonging to each preset traffic type is recorded as the belonging probability of each preset traffic type.

[0029] In one example, when the recognition system determines that the data stream to be identified belongs to each preset traffic type, the application layer payload data is obtained from the unencrypted data packet of the data stream to be identified; the payload data is segmented to generate a natural language processing (NLP) word list of the data stream to be identified; according to the NLP word list, the probability that the data stream to be identified belongs to each preset traffic type is determined by a pre-trained recognition model. The specific flow chart of the method for the recognition system to obtain the probability that the data stream to be identified belongs to each preset traffic type is as follows: Figure 2 As shown, the specific steps include:

[0030] Sub-step 1011, segment the traffic to obtain the data flow to be identified.

[0031] Specifically, after receiving a complete flow, the identification system divides the pcap data of the flow input according to the quintuple consisting of the five parameters of source IP, source port, destination IP, destination port, and protocol name, and arranges the messages in each data stream in chronological order to generate at least one data stream to be identified, and transmits the acquired data stream to be identified in an orderly manner through an input port to the data input interface of the classification system in the identification system, or transmits multiple data streams to be identified in parallel to the data input interface of the classification system in the identification system through multiple ports, so that the identification system obtains each data stream to be identified.

[0032] Sub-step 1012, pre-processing the data stream to be identified.

[0033] Specifically, after obtaining the data stream to be identified, the recognition system obtains the payload data of the application layer from the unencrypted data packet of the data stream to be identified, performs word segmentation on the payload data, and generates a natural language processing (NLP) word list of the data stream to be identified. Thus, the abstract features of the data stream to be identified are extracted into the NLP word list for use in subsequent recognition steps.

[0034] In one example, when the recognition system segments the payload data, it segments the payload data using binary grammar. Since the message byte range is "0x00" to "0xff", if a single byte is used as an independent word in the NLP word list, the vocabulary can only contain 256 words, which is a large gap from the number of words in natural language. When the payload data is segmented using binary grammar, the byte range is changed to "0x0000" to "0xffff", and the number of words that the vocabulary can contain becomes 65536, which basically meets the requirements of the natural language vocabulary, thereby facilitating the fitting of the obtained phrases with the natural language.

[0035] For example, the segmentation diagram of NLP word flow obtained by segmenting the payload data according to the bigram is as follows: Figure 3 As shown in the figure, the original payload data obtained is: 1601031122334455aa, and the NLP word list obtained after word segmentation according to Bigram is: 1601, 0103, 0311, 1122, 2233, 3344, 4455, 55aa.

[0036] Sub-step 1013, determining the probability that the data flow to be identified belongs to each preset traffic type.

[0037] Specifically, after preprocessing the data stream to be identified and obtaining the NLP word list of the data stream to be identified, the NLP word list is input into the recognition model of the recognition system. The recognition model determines the feature vector of the data stream to be identified based on the input NLP word list, and determines the probability that the data stream to be identified belongs to each preset traffic type based on the feature vector.

[0038] In one example, when determining the feature vector of the data stream to be identified based on the NLP word list, the recognition model obtains the feature vector of the data stream to be identified through the dynamic word vector algorithm based on the obtained NLP word list, and then calculates the probability that the data stream to be identified belongs to each preset traffic type based on the feature vector through the classification model in the recognition model, such as the softmax multi-classifier. By adopting the dynamic word vector algorithm to obtain the feature vector of the data stream to be identified, and utilizing the characteristic of the dynamic word vector algorithm to adjust itself according to the context of the input NLP word list, the problem that the static word vector can only achieve 1-to-1 mapping, resulting in inaccurate abstract feature extraction, is solved, and the directionality of the abstract features corresponding to the obtained word vector is improved.

[0039] The recognition model is trained in the following manner: providing a dynamic word vector model for pre-training; pre-training the dynamic word vector model according to the first text data; wherein the pre-training includes unsupervised training; inputting the encoding network parameters of the pre-trained dynamic word vector model into the recognition model, and performing supervised training on the recognition model according to the second text data.

[0040] For example, the training diagram of the recognition model is as follows Figure 4As shown, the first text data is Bigram text data without traffic type labels, and the second text data is Bigram text data containing traffic type labels. The training of the recognition model includes the representation learning of the dynamic word vector encoding network and the classification learning of the upper classification network of the classification model. When the dynamic word vector encoding network is trained for representation, Bigram text data can be directly extracted from the pre-obtained unsupervised data set, and the representation ability of the provided dynamic word vector model is trained in an unsupervised learning manner in the upper pre-trained network. The classification model in the classification system is divided into an upper classifier network and a lower encoding network. When training the classification model, the parameters in the lower encoding network can be directly copied from the pre-trained dynamic word vector encoding network, and the pre-trained dynamic word vector encoding network can be reused. Then, the Bigram text data extracted in the supervised learning set is directly called for training, and the parameters of the dynamic word vector encoding network in the model are fine-tuned in combination with the classification training results of the upper classification network until the training is completed. The recognition model is trained through a pre-training and fine-tuning mechanism. A large amount of unlabeled data is used to pre-train the dynamic word vector encoding network. When performing classification training, the parameters of the dynamic vector encoding network are reused, thereby completing the initialization of the encoding network before classification training, improving the fitting speed of classification training and the accuracy of classification. By separating the representation training and classification training in the classification system training process, more data is used to learn data stream features, thereby improving the recognition model's ability to identify the data stream characteristics and the traffic type to which it belongs, thereby improving the recognition model's accuracy in extracting feature vectors and calculating the probability that the data stream to be identified belongs to each preset traffic type, thereby improving the accuracy of traffic recognition.

[0041] In addition, the dynamic word vector model used in this embodiment includes but is not limited to any of the following dynamic word vector models: Bert (Bidirectional Encoder Representations from Transformers), Albert (ALite Bert), RoBERTa (A Robustly Optimized Bert).

[0042] Step 102, detect whether there is a preset traffic type with an belonging probability greater than a preset probability threshold. If there is no preset traffic type with an belonging probability greater than the preset probability threshold, go to step 103; if there is a preset traffic type with an belonging probability greater than the preset probability threshold, go to step 104.

[0043] Specifically, after obtaining the attribution probability of each preset traffic type based on the probability that the data flow to be identified after segmentation belongs to each preset traffic, the identification system reads the preset probability threshold of each preset traffic type pre-stored in itself, and then compares the attribution probability of each preset traffic type with the preset probability threshold of each preset traffic type, and detects whether there is a preset traffic type with an attribution probability greater than the preset probability threshold. If there is no preset traffic type with an attribution probability greater than the preset probability threshold, go to step 103; if there is a preset traffic type with an attribution probability greater than the preset probability threshold, go to step 104.

[0044] Step 103: determine that the traffic type of the data flow to be identified is an unknown type.

[0045] Specifically, after comparing the relationship between the attribution probability of each preset traffic type and the preset probability threshold of each preset traffic type, according to the comparison result, if there is no preset traffic type among the preset traffic types whose attribution probability is greater than or equal to the preset probability threshold, then the traffic type of the data flow to be identified is identified as an unknown type.

[0046] For example, there are three preset traffic types. The probability that the data flow to be identified belongs to type 1 is 0.3, the probability of belonging to type 2 is 0.35, and the probability of belonging to type 3 is 0.28. The preset probability threshold of type 1 is 0.45, the preset probability threshold of type 2 is 0.42, and the preset probability threshold of type is 0.55. There is no preset traffic type with a probability greater than the preset probability threshold, and the traffic type of the data flow to be identified is determined to be an unknown type.

[0047] Step 104: taking the preset traffic type with the highest belonging probability as the traffic type of the data flow to be identified.

[0048] Specifically, after completing the comparison of the attribution probability of each preset traffic type with the size of the preset probability threshold, if there is a preset traffic type among the preset traffic types whose attribution probability is greater than or equal to the preset probability threshold, the preset traffic type with the largest attribution probability will be used as the traffic type of the data flow to be identified.

[0049] In addition, when determining the traffic type of the data flow to be identified, a traffic type can be randomly selected from the preset traffic types whose belonging probability is greater than the preset probability threshold as the traffic type of the data flow to be identified. This embodiment does not limit the specific selection method of the traffic type of the data flow to be identified.

[0050] Therefore, this embodiment provides a traffic identification method, which divides the received traffic, obtains the NLP word list of the traffic to be identified according to the payload data of the unencrypted data packet in the data stream to be identified, and then extracts the feature vector of the data stream to be identified according to the dynamic word vector algorithm, and then calculates the probability that the data stream to be identified belongs to each preset traffic type through a convolutional neural network as the probability of belonging to the preset traffic type, and identifies the unknown type of traffic that does not belong to the preset traffic type according to the relationship between the probability of belonging to each preset traffic type and the preset probability threshold, or determines the traffic type of the data stream to be identified in the traffic type whose probability of belonging is greater than the preset threshold, and makes full use of the traffic information and flow characteristics to identify the traffic type, so as to realize accurate identification of the traffic type in the open set data traffic, avoid misidentification of traffic belonging to an unknown type as traffic of a known type, and improve the accuracy of traffic identification.

[0051] The second embodiment of the present invention relates to a traffic identification method. The second embodiment is substantially the same as the first embodiment. In this embodiment, when obtaining the preset probability threshold of each preset traffic type, the output probability obtained by the training sample through the created one-to-many classification network is fitted into a Gaussian distribution, and the preset probability threshold is calculated according to the mean and standard deviation of the Gaussian distribution to ensure the accuracy of the set probability threshold; after obtaining the unknown type of traffic, the model in the identification system can be expanded with the help of manual identification to improve the practicality of the identification system.

[0052] The process of the traffic identification method in this embodiment is basically the same as the process in the first embodiment. In this embodiment, before obtaining the pre-stored preset probability thresholds of each preset traffic type, the preset probability thresholds of each preset traffic type can be obtained according to the following method to create a one-to-many classification network for the preset traffic type; through the one-to-many classification network, the output probability of each training sample is obtained; wherein the training sample is a data stream belonging to the preset traffic type; the output probability of each training sample is fitted into a Gaussian distribution; and the preset probability threshold of the preset traffic type is calculated according to the mean and standard deviation of the Gaussian distribution.

[0053] In one example, when obtaining the preset probability threshold of each preset traffic type, the preset probability threshold is characterized by the open data set identification OSR rejection value of each preset traffic type calculated by the Deep Open Classification Algorithm (DOC). The schematic diagram of the preset probability threshold acquisition process is as follows: Figure 5 As shown, the following steps are included:

[0054] Step 501, create a binary classification network.

[0055] Specifically, the classification model includes N preset traffic types. The pre-trained classification model, such as the softmax classifier, is modified into N binary classification networks, and each training sample is input into each binary classification network. According to the output probability of each training sample, the cross entropy loss of each binary classification network is obtained, and then the average value of the N cross entropy losses is obtained. Then, the optimizer is used for back propagation training based on the average value of the cross entropy loss, and the final loss parameters of each binary classification network are adjusted to obtain a binary classification network.

[0056] Step 502: Obtain the output probability of each training sample.

[0057] Specifically, after the binary classification neural network training is completed, multiple training samples of each preset traffic type are input into the trained binary classification neural network, the output probability of each training sample is obtained through the binary classification neural network, and the output probability of each training sample is stored according to the preset traffic type to which the training sample belongs.

[0058] Step 503: Obtain a preset probability threshold for each preset traffic type.

[0059] Specifically, after obtaining the output probability of each training sample after the binary classification neural network, the output probability of multiple training samples of each preset type is respectively fitted into a Gaussian distribution, the standard deviation σ of each Gaussian distribution is calculated, and the mean μ of each Gaussian distribution is obtained. According to the rejection condition of the DOC algorithm, the difference between the output probability p and the mean μ is greater than a times the standard deviation σ, that is, "p-μ>>a×σ". Combined with the mean μ, standard deviation σ of each Gaussian distribution and the pre-set hyperparameter multiple a, the OSR rejection value M=μ-a×σ of each preset traffic type is obtained, and then the OSR rejection value of each preset traffic type is stored in the storage interval as the preset probability threshold. Therefore, the OSR rejection value of each preset traffic type calculated according to the rejection condition of the DOC algorithm is used as the preset probability threshold of each preset traffic type, and the accuracy of the preset probability threshold of each preset traffic type is guaranteed as much as possible.

[0060] Among them, when fitting the output probability of each training sample into a Gaussian distribution, the output probability of each training sample can be fitted into a Gaussian distribution with a mean of 1. By setting the mean to 1 during fitting, the calculation formula of the OSR rejection value is changed to M=1-a×σ, which simplifies the calculation of the preset probability threshold of each preset traffic type, thereby improving the calculation efficiency of each preset probability threshold and reducing the burden on the system.

[0061] When calculating the OSR rejection value, you can choose to use the DOC algorithm, the Open Max algorithm, or other open data set recognition algorithms in the classification field. This embodiment does not limit the algorithm selected for calculating the OSR rejection value.

[0062] In addition, in the process of obtaining the OSR rejection value of each preset traffic type in this embodiment, it can be updated synchronously with the training of the recognition model of the entire recognition system. In the process of training the recognition model, after the training time of the recognition model reaches the preset time interval, the one-to-many classification network is recreated according to the latest classification model parameters, and then the current OSR rejection value of each preset traffic type is obtained according to the latest output probability of each training sample in the newly created one-to-many classification network. Then, the preset probability threshold of each preset traffic type is updated according to the obtained new OSR rejection value. In the process of training the recognition model, the OSR rejection value is updated at every preset time interval until the recognition model training is completed, and the last obtained OSR rejection value is used to characterize the preset probability threshold of each preset traffic type.

[0063] By updating the OSR rejection value of each preset traffic type according to the preset update interval, it is ensured that the final OSR rejection value is the data closest to the actual situation of the trained classification recognition model, thereby improving the accuracy of the obtained OSR rejection value and thus improving the accuracy of each preset probability threshold.

[0064] Furthermore, after identifying data flows of unknown types, the unknown types of traffic are stored and summarized separately, and then passed to the offline analysis process. In the offline analysis stage, with the help of technical personnel's statistical analysis of traffic and the experience of experts, the traffic can be identified and labeled with specific labels. If the labeled traffic belongs to an existing preset traffic type, the labeled data flow is added to the supervised training set of the preset traffic type corresponding to the label; if the labeled data flow does not belong to any preset traffic type, the labeled data flow is treated as an independent traffic type and imported into the supervised learning set, and the parameters of the classification network are adjusted to expand new preset traffic types. After completing the addition and import of data, the classification model is retrained to expand the recognition ability and accuracy of the recognition model.

[0065] In addition, the method of this embodiment can also be combined with the DPI identification technology in the prior art. First, the DPI device identifies the data flow to be identified after the diversion. If the DPI device can complete the identification of the traffic type to which all the data flows to be identified belong, the identification result is directly output. If the DPI device cannot complete the identification of the traffic type to which all the data flows to be identified belong, the data flow to be identified that the DPI device cannot identify is transmitted to the trained identification model for identification, thereby ensuring the efficiency of traffic identification while improving the accuracy and practicality of traffic identification as much as possible.

[0066] In addition, those skilled in the art can understand that the step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the protection scope of this patent.

[0067] A third embodiment of the present invention relates to a flow identification device, such as Figure 6 As shown, including:

[0068] The determination module 601 is used to determine the probability that the data flow to be identified belongs to each preset traffic type; wherein the probability of belonging to each preset traffic type is recorded as the belonging probability of each preset traffic type;

[0069] A comparison module 602 is used to compare the relationship between the belonging probability of each preset traffic type and the preset probability threshold of each preset traffic type;

[0070] The identification module 503 is used to determine the traffic type of the data flow to be identified. If there is no preset traffic type with an attribution probability greater than or equal to a preset probability threshold among the preset traffic types, the traffic type of the data flow to be identified is identified as an unknown type.

[0071] In one example, the identification module 603 is further used to, when there is a preset traffic type with an belonging probability greater than or equal to a preset probability threshold among the preset traffic types, use the preset traffic type with the highest belonging probability as the traffic type of the data flow to be identified.

[0072] In another example, the preset probability threshold of each preset traffic type is calculated in the following manner: create a one-to-many classification network for the preset traffic type; obtain the output probability of each training sample through the one-to-many classification network; wherein the training sample is a data stream belonging to the preset traffic type; fit the output probability of each training sample into a Gaussian distribution; and calculate the preset probability threshold of the preset traffic type based on the mean and standard deviation of the Gaussian distribution.

[0073] In another example, fitting the output probability of each training sample into a Gaussian distribution includes: fitting the output probability of each training sample into a Gaussian distribution with a mean of 1.

[0074] In another example, the determination module 601 is specifically used to obtain application layer payload data from unencrypted data packets of the data stream to be identified; segment the payload data to generate a natural language processing (NLP) word list of the data stream to be identified; and determine, based on the NLP word list, the probability that the data stream to be identified belongs to each preset traffic type through a pre-trained recognition model.

[0075] In another example, the recognition model is trained in the following manner: providing a dynamic word vector model for pre-training; pre-training the dynamic word vector model according to first text data; wherein the pre-training includes unsupervised training; inputting the encoding network parameters of the pre-trained dynamic word vector model into the recognition model, and performing supervised training on the recognition model according to second text data.

[0076] In another example, the determination module 601 is specifically configured to segment the payload data using a bigram.

[0077] It is not difficult to find that this embodiment is a device embodiment corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the first embodiment.

[0078] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other units in this embodiment.

[0079] A fourth embodiment of the present invention relates to an electronic device, such as Figure 7 As shown, it includes at least one processor 701; and a memory 702 that is communicatively connected to the at least one processor 701; wherein the memory 702 stores instructions that can be executed by the at least one processor 701, and the instructions are executed by the at least one processor 701 so that the at least one processor 701 can execute the above-mentioned traffic identification method.

[0080] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0081] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0082] A fifth embodiment of the present invention relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0083] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0084] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present invention, and that in actual applications, various changes may be made in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A flow identification method, characterized in that: include: Obtaining application layer payload data from unencrypted data packets of the data stream to be identified; By using binary grammar, the payload data is segmented to generate a natural language processing (NLP) word list of the data stream to be identified; According to the NLP word list, the probability that the data flow to be identified belongs to each preset traffic type is determined by a pre-trained recognition model; wherein the probability of belonging to each preset traffic type is recorded as the belonging probability of each preset traffic type; Compare the relationship between the attribution probability of each preset traffic type and the preset probability threshold of each preset traffic type; If there is no preset traffic type whose belonging probability is greater than or equal to the preset probability threshold among the preset traffic types, then the traffic type of the data flow to be identified is identified as an unknown type; The recognition model is trained in the following way: Provides a dynamic word vector model for pre-training; Pre-training the dynamic word vector model according to the first text data; wherein the pre-training includes unsupervised training; The encoding network parameters of the pre-trained dynamic word vector model are input into the recognition model, and the recognition model is supervised and trained according to the second text data.

2. The flow identification method according to claim 1, characterized in that: After comparing the relationship between the belonging probabilities of the preset traffic types and the preset probability thresholds of the preset traffic types, the method further includes: If there is a preset traffic type whose belonging probability is greater than or equal to the preset probability threshold among the preset traffic types, the preset traffic type with the largest belonging probability is used as the traffic type of the data flow to be identified.

3. The flow identification method according to claim 1, characterized in that: The preset probability threshold of each preset traffic type is calculated in the following way: Creating a one-to-many classification network of the preset traffic type; Obtaining the output probability of each training sample through the one-to-many classification network; wherein the training sample is a data flow belonging to the preset traffic type; Fitting the output probability of each training sample into a Gaussian distribution; The preset probability threshold of the preset traffic type is calculated according to the mean and standard deviation of the Gaussian distribution.

4. The flow identification method according to claim 3, characterized in that: The step of fitting the output probability of each training sample into a Gaussian distribution comprises: The output probability of each training sample is fitted into a Gaussian distribution with a mean of 1.

5. A flow identification device, characterized in that: include: A determination module, used for obtaining the payload data of the application layer from the unencrypted data packet of the data stream to be identified; The payload data is segmented by binary grammar to generate a natural language processing (NLP) word list of the data stream to be identified; based on the NLP word list, the probability that the data stream to be identified belongs to each preset traffic type is determined by a pre-trained recognition model; wherein the probability of belonging to each preset traffic type is recorded as the belonging probability of each preset traffic type; A comparison module, used for comparing the relationship between the belonging probability of each of the preset traffic types and the preset probability threshold of each of the preset traffic types; an identification module, configured to determine the traffic type of the data flow to be identified, and if there is no preset traffic type whose belonging probability is greater than or equal to a preset probability threshold among the preset traffic types, then identifying that the traffic type of the data flow to be identified is an unknown type; The determination module is also used to provide a dynamic word vector model for pre-training; pre-train the dynamic word vector model according to the first text data; wherein the pre-training includes unsupervised training; input the encoding network parameters of the pre-trained dynamic word vector model into the recognition model, and perform supervised training on the recognition model according to the second text data.

6. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the traffic identification method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the traffic identification method according to any one of claims 1 to 4 is implemented.

Images