Network traffic data analysis method based on traffic large model

Abstract

The invention relates to the technical field of network traffic data analysis, in particular to a network traffic data analysis method based on a traffic large model. The method comprises the following steps: collecting network flow data from network flow management equipment, analyzing messages in a flow data packet based on a network protocol or application features and the like, and extracting key feature information; and recording corresponding specific behavior operation of the traffic data in the network platform by using the data label, collecting and forming a behavior data set, constructing a traffic large model based on the general large model, and training and verifying the model by inputting the behavior data set. The large model is applied to network management equipment, behavior data of network users are recorded, then the behavior data are counted to form a visual chart, possible behaviors of the next network users can be predicted through the current behavior data, and network managers are helped to supervise behavior specifications of the users and guarantee safety of a network platform.

Description

Technical Field

[0001] The present invention relates to the technical field of network flow data analysis, and in particular to a network flow data analysis method based on a large flow model. Background Art

[0002] Network traffic data analysis is a method of collecting, analyzing and processing network data packets, mainly used to monitor, optimize and protect network performance. With the development of network technology, the behavior data of users on the network platform has gradually diversified, and the corresponding network traffic data analysis technology has become increasingly important. There are a large number of user behavior activities on the network service side, which will have a certain impact on the maintenance and supervision efficiency of the network traffic management party. For example, if there are certain intrusion behavior data in these network traffic data, these behaviors combined may directly affect the service platform. If the current user's behavior data is counted through network traffic packets, visual information can be provided to network traffic management personnel, and intrusion behaviors that may flow through the network platform can also be eliminated in advance. With the growth of network traffic data, whether it is possible to efficiently identify the specific behaviors generated by data on the service platform from numerous network traffic packets is worth further study in network traffic management and data analysis.

[0003] In recent years, with the vigorous development of artificial intelligence, large language models have emerged. ChatGPT, the most well-known model at this stage, is one of its representatives. Such models can not only generate fluent and natural text but also understand the context of input, providing support for a variety of application scenarios. If the large models used in artificial intelligence are applied to network traffic data analysis and the specific behavior visualization data of users on the service platform is obtained, it will be beneficial for network managers to supervise and make timely adjustments.

[0004] Traditional methods of identifying network traffic data are slow, and if they are maintained at a high rate, the accuracy may be low. In order to cope with the needs of massive network traffic data analysis, new means need to be adopted to deal with it. In view of the large language model of artificial intelligence, if a rich large network traffic model is built in advance, when facing massive data recognition, the accuracy and rate of recognition data can be improved and the subsequent behavior can be predicted. To this end, the present invention identifies and analyzes network traffic behavior data based on the large traffic model. Summary of the invention

[0005] The purpose of the present invention is to provide a network traffic data analysis method based on a large traffic model to solve the above technical problems.

[0006] A network traffic data analysis method based on a large traffic model, the method comprising the following steps:

[0007] S1. Obtain network traffic data from a network traffic management device, or collect network traffic data packets from a network traffic database;

[0008] S2. Analyze the message based on network protocol or application characteristics to extract key data information;

[0009] S3, by inputting network traffic data into the network platform to observe its specific behavior in the platform, supplement the characteristic information and annotate the corresponding data;

[0010] S4, build a large traffic model, input the labeled network traffic data into the model for training and verification;

[0011] S5. Obtain the network traffic data flowing through the device from the network traffic management device, input it into the model in real time, perform network traffic data analysis, generate visual behavior data charts and network traffic behavior prediction possibilities, and help managers perform data analysis.

[0012] As a further description of the solution of the present invention, the method for collecting network traffic data from the network traffic management device in step S1 uses a packet capture tool or a built-in function of the network device to collect traffic data packets.

[0013] As a further description of the solution of the present invention, the specific process of step S2 includes:

[0014] S21, pre-process the network traffic data packets, delete the redundant protocol data and retain the key behavior data;

[0015] S23, after removing redundant protocol data and leaving key behavior information, extract the current network protocol and IP address feature data.

[0016] As a further description of the solution of the present invention, the specific process of step S3 includes:

[0017] S31, observe the behavior reaction on the network platform to obtain the specific behavior affected by the current data, and after knowing the behavior, label the network traffic data with the behavior, and collect it into the data set;

[0018] S32, annotate the behaviors before and after the behavior and supplement the characteristic data with behavioral characteristics in order to identify and predict the behavior.

[0019] As a further description of the solution of the present invention, the specific process of step S3 also includes: adding the behavior data as a feature into the preprocessing feature to form a set X = {x1, x2, ..., x n , x n+1 , x n+2 , x n+3}, where xn+1 、x n+2 、x n+3 They are respectively represented as the specific platform behavior corresponding to the data, the behavior data before the behavior, and the behavior data after the behavior;

[0020] Standardize the features by normalizing:

[0021]

[0022] Among them, F norm (x i ) is the normalized standard feature, F(x i ) is the normalized feature, μ is the feature mean, and σ is the feature standard deviation.

[0023] As a further description of the solution of the present invention, the specific process of step S4 includes:

[0024] S41, inputting the labeled network traffic data into the traffic model, training the model parameters, classifying the data into training set and validation set before inputting the model, and adjusting the network model parameters according to the output data results;

[0025] S42. Set a threshold parameter. If the output result is greater than the threshold, it is considered that the predicted behavior may occur. If it is less than the threshold, it indicates that the predicted behavior is unlikely to occur.

[0026] As a further description of the scheme of the present invention, the method of model construction includes:

[0027] Based on the general large model, we make adjustments to build a large traffic model. We select the pre-trained large model M based on the Transformer architecture. The input of the model is feature F. norm (X), where X is the feature vector extracted from the data;

[0028] The labeled data set D = {(F norm (x i ), y i )} is input into the large model to train its parameters, where y i For the corresponding label, the model parameters are optimized iteratively, and the loss function used is the cross entropy loss. The specific formula is:

[0029]

[0030] Among them, N is the number of samples, i belongs to n, C is the number of categories, k belongs to C, y ik is the true label of sample i, is the predicted probability of the model. The model is trained through the training set, and then the validation set is input to verify the accuracy of the model;

[0031] For the output results, the model's accuracy (A), precision (P), recall (R), and F1 value are calculated through the confusion matrix. The calculation formula is:

[0032]

[0033] Among them, TP represents the number of correctly predicted positive samples, TN represents the number of correctly predicted negative samples; FP represents the number of negative samples incorrectly predicted as positive samples, and FN represents the number of positive samples incorrectly predicted as negative samples.

[0034] As a further description of the solution of the present invention, the specific process of step S5 includes:

[0035] S51, obtaining the current network user behavior data through the model output, counting all the current user behavior information, and generating a visual statistical chart in the post-processing;

[0036] S52. During real-time data analysis, if it is detected that the network traffic contains unlabeled behaviors, the unknown behavior data set is collected and formed.

[0037] The beneficial effects of the present invention are as follows: 1. A network traffic data analysis method based on a large traffic model, which constructs a large network traffic model to annotate and collect network traffic data corresponding to the behavior operations of network users to form a large model. When identifying massive network traffic data, user behavior statistical data can be obtained efficiently, which is convenient for supervision by network managers.

[0038] 2. The network traffic data analysis method based on the traffic big model obtains the specific behavior of users from the operation data in the network traffic data, which is the process before the user accesses the platform. If the big model can detect the threat behavior before certain behaviors damage the service platform, it can eliminate this part of the network threat behavior in the traffic propagation path to ensure the security of the network platform.

[0039] 3. A network traffic data analysis method based on a large traffic model. The specific research content of the present invention is the behavior data analysis in network traffic data. In addition to user behavior data, large models of other different focus points can also be constructed to meet the needs of massive network data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The present invention will be further described below in conjunction with the accompanying drawings.

[0041] Figure 1 The present invention is a flowchart of the steps of the network traffic data analysis method based on the large traffic model.

[0042] Figure 2The present invention is a flow chart of the data set composition of the network traffic data analysis method based on the large traffic model. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] See also Figure 1 , Figure 2 As shown, the present invention is a network traffic data analysis method based on a large traffic model, characterized in that the method comprises the following steps:

[0045] S1. Obtain network traffic data from a network traffic management device, or collect network traffic data packets from a network traffic database;

[0046] The method for collecting network traffic data from the network traffic management device in step S1 uses a packet capture tool or a built-in function of the network device to collect traffic data packets.

[0047] Building a large traffic model that focuses on network user behavior requires collecting a large amount of network traffic data. Because different platforms may have different behavioral characteristics, the data sets for different platforms will also be different. Collect as much data as possible from network traffic management devices or network traffic databases. For some low-probability behaviors that may not be collected in the initial data set, you can retain unlabeled data when running the large model later and further supplement the data set.

[0048] S2. Analyze the message based on network protocol or application characteristics to extract key data information;

[0049] The specific process of step S2 includes:

[0050] S21, pre-process the network traffic data packets, delete the redundant protocol data and retain the key behavior data;

[0051] S23, after removing redundant protocol data and leaving key behavior information, extract the current network protocol and IP address feature data.

[0052] After obtaining network traffic, the data packets are parsed. In order to match network traffic data, it is necessary to parse the packets using methods based on network protocols or application features. The matching method can be to use quintuples, protocol fields based on network protocols, or regular expressions, URLs, specific fields, etc. based on application features. Preprocessing is used to extract feature information about user behavior from network traffic packets, and to remove redundant protocol data. Preprocessing can also reduce the amount of data to be processed and improve data processing efficiency.

[0053] S3, by inputting network traffic data into the network platform to observe its specific behavior in the platform, supplement the characteristic information and annotate the corresponding data;

[0054] 4. According to the network traffic data analysis method based on the traffic big model according to claim 1, it is characterized in that the specific process of step S3 includes:

[0055] S31, observe the behavior reaction on the network platform to obtain the specific behavior affected by the current data, and after knowing the behavior, label the network traffic data with the behavior, and collect it into the data set;

[0056] S32, annotate the behaviors before and after the behavior and supplement the characteristic data with behavioral characteristics in order to identify and predict the behavior.

[0057] The specific process of step S3 also includes: adding the behavior data as a feature into the preprocessing feature to form a set X = {x1, x2, ..., x n , x n+1 , x n+2 , x n+3}, where x n+1 、x n+2 、x n+3 They are respectively represented as the specific platform behavior corresponding to the data, the behavior data before the behavior, and the behavior data after the behavior;

[0058] Standardize the features by normalizing:

[0059]

[0060] Among them, F norm (x i ) is the normalized standard feature, F(x i ) is the normalized feature, μ is the feature mean, and σ is the feature standard deviation.

[0061] The network traffic data is associated with the specific behavior generated by the data on the network platform, and the data corresponding to the specific behavior is labeled. This process requires a large amount of network traffic data and behavior association as a data set. The feature information extracted in the previous process is added with the current behavior and the previous and next behavior feature information. This can accurately identify the current behavior when identifying the data, and predict the possibility of subsequent behavior through the previous and next relationships. Since the user's behavior operations are often a series of behaviors, the subsequent behavior can be predicted through partial behaviors. For behaviors that may threaten the platform, the subsequent behavior can be judged and responded to in time. Performing large model recognition operations before the traffic data flows through the service platform can predict risky behaviors in advance.

[0062] S4, build a large traffic model, input the labeled network traffic data into the model for training and verification;

[0063] The specific process of step S4 includes:

[0064] S41, inputting the labeled network traffic data into the traffic model, training the model parameters, classifying the data into training set and validation set before inputting the model, and adjusting the network model parameters according to the output data results;

[0065] S42. Set a threshold parameter. If the output result is greater than the threshold, it is considered that the predicted behavior may occur. If it is less than the threshold, it means that the predicted behavior is unlikely to occur.

[0066] The method of constructing the model comprises:

[0067] Based on the general large model, we make adjustments to build a large traffic model. We select the pre-trained large model M based on the Transformer architecture. The input of the model is feature F. norm (X), where X is the feature vector extracted from the data;

[0068] The labeled data set D = {(F norm (x i ), y i )} is input into the large model to train its parameters, where y i For the corresponding label, the model parameters are optimized iteratively, and the loss function used is the cross entropy loss. The specific formula is:

[0069]

[0070] Among them, N is the number of samples, i belongs to n, C is the number of categories, k belongs to C, y ik is the true label of sample i, is the predicted probability of the model. The model is trained through the training set, and then the validation set is input to verify the accuracy of the model;

[0071] For the output results, the model's accuracy (A), precision (P), recall (R), and F1 value are calculated through the confusion matrix. The calculation formula is:

[0072]

[0073] Among them, TP represents the number of correctly predicted positive samples, TN represents the number of correctly predicted negative samples; FP represents the number of negative samples incorrectly predicted as positive samples, and FN represents the number of positive samples incorrectly predicted as negative samples.

[0074] S5. Obtain the network traffic data flowing through the device from the network traffic management device, input it into the model in real time, perform network traffic data analysis, generate visual behavior data charts and network traffic behavior prediction possibilities, and help managers perform data analysis.

[0075] The specific process of step S5 includes:

[0076] S51. Obtain the current network user behavior data through model output, count all current user behavior information, and generate visual statistical charts during post-processing, so that management personnel can focus on observing the frequently occurring behavior operations on the platform and strengthen this part of technical maintenance;

[0077] The output of the model also includes predictions of network user behavior, helping network administrators predict in advance what behavior will occur immediately after the current behavior. If the user's subsequent behavior will pose a certain threat to the platform, the large model can allow the platform to take precautions in advance.

[0078] S52. During real-time data analysis, if it is detected that the network traffic contains unlabeled behaviors, the unknown behavior data set is collected. On the one hand, it can be left for later data set supplementation, and on the other hand, if the behavior involves endangering network security, it can be placed in the threat data set.

[0079] The large model is placed in the network traffic management device, and the network traffic data packets flowing through the network traffic management device are preprocessed and input into the large model. If some traffic data is not labeled, the traffic data can be retained and collected to be added to the unlabeled data set, waiting for later supplementary labeling. After identification and prediction by the large model, on the one hand, chart data of user behavior statistics will be obtained, and on the other hand, predictive data on the future behavior of certain users will be obtained. Chart data can help managers monitor the behavior data of the platform, and predictive data can determine possible behaviors after the behavior. For the predicted data, a threshold needs to be set in advance for judgment:

[0080]

[0081] The behavior category is determined based on the predicted threshold θ. If the predicted value is greater than the threshold, it is considered that the subsequent behavior is likely to be the predicted behavior. Otherwise, it is highly unlikely to be the predicted behavior. The user's series of behaviors are compared with a series of behavior data sets of the threat platform. If it is a threatening behavior, the administrator needs to handle it in a timely manner.

[0082] Output prediction results through the trained model

[0083]

[0084] Among them, W is the weight matrix and b is the bias term. For a behavior, there are usually multiple possibilities for its subsequent behavior. The model will bring out the most likely behavior and regard it as the behavior that is most likely to occur. Once the behavior may have a certain impact on the network platform, the administrator can have a certain reaction time and can be warned in advance. For the accuracy of the prediction, it is compared with the previously set threshold. If the prediction result is greater than the threshold, the output result is 1, indicating that the behavior is likely to occur; if it is less than the threshold, the output result is 0, indicating that the behavior is unlikely to occur.

[0085] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A network traffic data analysis method based on a large traffic model, characterized in that: The method comprises the following steps: S1. Obtain network traffic data from a network traffic management device, or collect network traffic data packets from a network traffic database; S2. Analyze the message based on network protocol or application characteristics to extract key data information; S3, by inputting network traffic data into the network platform to observe its specific behavior in the platform, supplement feature information and annotate the corresponding data; S4, build a large traffic model, input the labeled network traffic data into the model for training and verification; S5. Obtain the network traffic data flowing through the device from the network traffic management device, input it into the model in real time, perform network traffic data analysis, generate visual behavior data charts and network traffic behavior prediction possibilities, and help managers perform data analysis.

2. The network traffic data analysis method based on the traffic big model according to claim 1 is characterized in that: The method for collecting network traffic data from the network traffic management device in step S1 uses a packet capture tool or a built-in function of the network device to collect traffic data packets.

3. The network traffic data analysis method based on the traffic big model according to claim 2 is characterized in that: The specific process of step S2 includes: S21, pre-process the network traffic data packets, delete the redundant protocol data and retain the key behavior data; S23, after removing redundant protocol data and leaving key behavior information, extract the current network protocol and IP address feature data.

4. The network traffic data analysis method based on the traffic big model according to claim 1 is characterized in that: The specific process of step S3 includes: S31, observe the behavior reaction on the network platform to obtain the specific behavior affected by the current data, and after knowing the behavior, label the network traffic data with the behavior, and collect it into the data set; S32, annotate the behaviors before and after the behavior and supplement the characteristic data with behavioral characteristics in order to identify and predict the behavior.

5. The network traffic data analysis method based on the traffic big model according to claim 4 is characterized in that: The specific process of step S3 also includes: adding the behavior data as a feature into the preprocessing feature to form a set X = {x1, x2, ..., x n , x n+1 , x n+2 , x n+3 }, where x n+1 、x n+2 、x n+3 They are respectively represented as the specific platform behavior corresponding to the data, the behavior data before the behavior, and the behavior data after the behavior; Standardize the features by normalizing: Among them, F norm (x i ) is the normalized standard feature, F(x i ) is the normalized feature, μ is the feature mean, and σ is the feature standard deviation.

6. The network traffic data analysis method based on the traffic big model according to claim 1 is characterized in that: The specific process of step S4 includes: S41, inputting the labeled network traffic data into the traffic model, training the model parameters, classifying the data into training set and validation set before inputting the model, and adjusting the network model parameters according to the output data results; S42. Set a threshold parameter. If the output result is greater than the threshold, it is considered that the predicted behavior may occur. If it is less than the threshold, it indicates that the predicted behavior is unlikely to occur.

7. The network traffic data analysis method based on the traffic big model according to claim 6 is characterized in that: The method of constructing the model comprises: Based on the general large model, we make adjustments to build a large traffic model. We select the pre-trained large model M based on the Transformer architecture. The input of the model is feature F. norm (X), where X is the feature vector extracted from the data; The labeled data set D = {(F norm (x i ), y i )} is input into the large model to train its parameters, where y i For the corresponding label, the model parameters are optimized iteratively, and the loss function used is the cross entropy loss. The specific formula is: Among them, N is the number of samples, i belongs to n, C is the number of categories, k belongs to C, y ik is the true label of sample i, is the predicted probability of the model. The model is trained through the training set, and then the validation set is input to verify the accuracy of the model; For the output results, the model's accuracy (A), precision (P), recall (R), and F1 value are calculated through the confusion matrix. The calculation formula is: Among them, TP represents the number of correctly predicted positive samples, TN represents the number of correctly predicted negative samples; FP represents the number of negative samples incorrectly predicted as positive samples, and FN represents the number of positive samples incorrectly predicted as negative samples.

8. The network traffic data analysis method based on the traffic big model according to claim 1 is characterized in that: The specific process of step S5 includes: S51, obtaining the current network user behavior data through the model output, counting all the current user behavior information, and generating a visual statistical chart in the post-processing; S52. During real-time data analysis, if it is detected that the network traffic contains unlabeled behaviors, the unknown behavior data set is collected and formed.

Images