A highly efficient abnormal traffic detection method and system based on data reduction
By combining the data streamlining and construction process of abnormal traffic detection models, and using optimization technology to obtain simulated data, the pressure on storage and calculation costs of traditional abnormal traffic detection methods is solved, and efficient abnormal traffic detection model training and performance maintenance is achieved.
Patent Information
- Application Number
- CN202210149568.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-18
- Publication Date
- 2025-04-25
- Estimated Expiration
- 2042-02-18
AI Technical Summary
Traditional abnormal traffic detection methods have pressure on storage and computing costs, and are difficult to meet the security needs of modern communications.
By combining the data streamlining process and the construction process of the abnormal traffic detection model, we use optimization technology to obtain simulated data, train the abnormal traffic detection model, ensure that the model performance is not damaged, and at the same time reduce data storage requirements.
It realizes the performance of the abnormal traffic detection model while reducing data storage and computing costs, and alleviates the pressure of base station data storage.
Smart Images
Figure CN114580506B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a highly efficient abnormal traffic detection method and system based on data simplification, which mainly utilizes optimization technology to obtain a batch of simulation data to alleviate the pressure of data storage on a base station. Background Art
[0002] With the development of communication technology and the emergence of various terminal applications, the potential harm caused by network attacks has become increasingly obvious. Abnormal traffic detection is a technology that automatically discovers abnormal communication requests based on traffic data. However, traditional abnormal traffic detection methods can no longer meet the needs of today's communication security, and anomaly detection technology based on machine learning algorithms has gradually become mainstream. Machine learning technology can learn a model from historical data, and use this model to predict whether new data is abnormal traffic. However, the training of machine learning models often relies on a large amount of data, but the storage capacity on the base station is often limited and cannot store a large amount of historical communication data. If the data is stored in a remote data server, it will bring additional communication costs. Therefore, it is very important to streamline the data stored on the base station.
[0003] The purpose of data reduction is to reduce the storage cost and the computational cost of model training. In the field of machine learning, there are many methods of data reduction, such as feature selection, feature extraction, key sample selection, etc. Feature selection refers to selecting a part of the features that most affect the performance of the model from all the features of the sample and ignoring other features. In this way, only low-dimensional samples can be used when training the model, thereby converging faster. The purpose of feature extraction is similar to that of feature selection, but feature extraction constructs a set of new features based on the original features of the sample, and this set of features is not selected from the original features. Key sample selection does not change the dimension of the sample, but selects the most important samples from the entire data set, thereby reducing the sample size. However, the current data reduction methods are difficult to combine with the model training process, and there is no guarantee that the model trained with the reduced data can still maintain good performance. Summary of the invention
[0004] Purpose of the invention: In view of the problems and shortcomings in the prior art, the present invention provides an efficient abnormal traffic detection method and system based on data reduction, which combines the process of data reduction with the process of building an abnormal traffic detection model, thereby reducing the demand for data storage while maintaining the performance of the model. The key to the present invention is to use optimization technology to obtain a batch of simulated data. The scale of this batch of simulated data is much smaller than the real data, but when this batch of simulated data is used to train the abnormal traffic detection model, the update direction of the model parameters is almost the same as the update direction when using real data for training, so that the trained model has similar performance to the model trained using real data. With the help of this technology, the pressure on data storage on the base station can be greatly alleviated.
[0005] Technical solution: An efficient abnormal traffic detection method based on data reduction, which includes two major processes, namely the data reduction process and the simplified abnormal traffic detection model training process.
[0006] During the data reduction process, obtain real data from the base station database; randomly initialize simulated data; randomly initialize an abnormal traffic detection model; use real data and simulated data to train the model respectively, record the loss function of the model on the two sets of data and the corresponding parameter gradient; calculate the Euclidean distance between the two sets of gradients; minimize the above Euclidean distance to update the simulated data; repeat the above process until the simulated data converges, record the simulated data, and clear the real data in the base station database. During the simplified abnormal traffic detection model training process, the user selects an abnormal traffic detection model and initializes the model; obtains previously saved simulated data; and trains the model using simulated data. The present invention obtains a small amount of simulated data based on gradient matching, which greatly reduces the storage pressure of the base station database and ensures that the performance of the model does not deteriorate.
[0007] The data reduction process is as follows:
[0008] Step 100, obtaining an existing real data set in the base station database, which is denoted as T;
[0009] Step 101, randomly initialize a simulation data set S;
[0010] Step 102, the user specifies the maximum number of iterations K;
[0011] Step 103, randomly initialize an abnormal traffic detection model M, the specific type and structure of the model can be specified by the user; the user specifies a loss function, generally a cross entropy loss;
[0012] Step 104, randomly sample a small batch of samples B_T from the real data set T, and randomly sample a small batch of samples B_S from the simulated data set S;
[0013] Step 105, according to the model M, respectively calculate its loss function value l_T on B_T and its loss function value l_S on B_S;
[0014] Step 106, respectively calculating the gradients g_T and g_S on the parameters of the model M according to l_T and l_S, and the specific calculation method is common knowledge in this field;
[0015] Step 107, calculate the Euclidean distance between g_T and g_S, calculate the gradient on the small batch B_S of the simulated data set based on the Euclidean distance, and update the small batch of simulated data sets based on relevant optimization techniques. The calculation method of the gradient is similar to that in step 106, and both belong to the general knowledge in this field;
[0016] Step 108, based on relevant optimization techniques, using the gradient g_T to update the model M;
[0017] Step 109, check whether the simulation data set S converges, if not converged and the current number of iterations does not reach K, return to step 104, otherwise go to step 110;
[0018] Step 110, record the updated simulated data set S, and clear the real data set T in the database.
[0019] The simplified abnormal traffic detection model training process is as follows:
[0020] Step 200, obtaining the updated simulation data set S in the previous process;
[0021] Step 201, the user specifies the maximum number of iterations K;
[0022] Step 202: randomly initialize an abnormal traffic detection model M. The specific type and structure of the model can be specified by the user. The user specifies a loss function, which is generally a cross entropy loss.
[0023] Step 203, randomly sampling a small batch of samples B_S from the simulated data set S;
[0024] Step 204, based on the model M, calculate its loss function l_S on B_S;
[0025] Step 205, calculating the gradient g_S of the parameters of the model M based on l_S, the specific calculation method is common knowledge in this field;
[0026] Step 206, based on relevant optimization techniques, using the gradient g_S to update the model M;
[0027] Step 207, check whether the model M has converged, if not and the current number of iterations has not reached K, return to step 203, otherwise go to step 208;
[0028] Step 208: output the trained model M.
[0029] The optimization technique refers to the gradient descent method in optimization theory. By using this method, we can find a local optimal solution to the loss function of the problem and update the target parameters;
[0030] The abnormal traffic detection model M can generally be specified as a neural network, and the specific type and structure of the model can be specified by the user. How to design the model is irrelevant to this technology, that is, this technology can be used for neural networks of any type and structure.
[0031] Virtual data is constructed to replace real data, and the process of constructing the virtual data is related to the training process of the model.
[0032] The construction process of the virtual data is based on matching the gradient of the model parameters, so that the virtual data can have a training effect similar to that of the real data.
[0033] An efficient abnormal traffic detection system based on data reduction, comprising a data reduction module and a simplified abnormal traffic detection model training module;
[0034] The data reduction module obtains real data from the base station database; randomly initializes simulated data; randomly initializes an abnormal traffic detection model; uses real data and simulated data to train the model respectively, and records the loss function and corresponding parameter gradient of the model on the two sets of data; calculates the Euclidean distance between the two sets of gradients; minimizes the Euclidean distance to update the simulated data; repeats the above process until the simulated data converges, records the simulated data, and clears the real data in the base station database; in the simplified abnormal traffic detection model training module, the user selects an abnormal traffic detection model and initializes the model; obtains the updated simulated data saved by the data reduction module; and uses the simulated data to train the abnormal traffic detection model.
[0035] The implementation process and method of the efficient abnormal traffic detection system based on data reduction are the same.
[0036] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the efficient abnormal traffic detection method based on data reduction as described above is implemented.
[0037] A computer-readable storage medium stores a computer program for executing the above-mentioned efficient abnormal traffic detection method based on data reduction.
[0038] Beneficial effects: Compared with the prior art, the efficient abnormal traffic detection method and system based on data simplification provided by the present invention only needs to store a small amount of virtual training data to train an abnormal traffic detection model. Therefore, the present method reduces the storage cost of base station data and reduces the computational cost of model training. In addition, the generation process of simulated data in the present method is related to the model training process, so the abnormal traffic detection model finally obtained has better performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 Flow chart of the data reduction process in an embodiment of the present invention;
[0040] Figure 2The present invention is a simplified flowchart of the abnormal traffic detection model training process in an embodiment of the present invention. DETAILED DESCRIPTION
[0041] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.
[0042] The process of generating virtual data using real data stored in the base station database is as follows: Figure 1 As shown. First, data T is obtained from the real database stored in the base station. These data are historical records of real users (each historical record corresponds to a visit of the user to the base station. The historical record contains some relevant information of this visit, such as "visit time", "geographic location", "base station number", "base station status", "visit duration", "traffic size", etc. The specific content depends on what the base station records.), and is marked with abnormality or not (step 10); then, a set of simulated data sets S is randomly initialized. This set of data sets is represented in the form of a matrix, each row represents a sample, each column represents a feature, and the specific size of the matrix can be specified by the user. This set of data will be updated in the following process and eventually become available simulation data (step 11); initialize an abnormal traffic detection model M, generally a neural network (step 12); start iteration, and each step first samples a small batch of samples B_T and B_S from the real data set T and the simulated data set S (step 13); use the cross entropy loss function to calculate the loss values of the model M on B_T and B_S respectively (step 14), and calculate the gradients of these two loss values on the parameters of the model M respectively (step 15); calculate the Euclidean distance between the two gradients, and calculate the gradient of this distance to the simulated sample S_T (step 16); use the calculated gradient to update the simulated sample (step 17); repeat the above process until all simulated samples no longer produce significant changes (step 18, the degree of change here can be specified by the user, and the standard for measuring changes can be the F norm of the difference between the sample matrices); finally, clear the real historical records in the database to reduce the storage cost and save the smaller-scale simulation data (step 19).
[0043] The process of using streamlined simulated data to train the abnormal traffic detection model is as follows Figure 2 First, by Figure 1The process shown obtains a simulated data set S (step 20); then, randomly initializes an abnormal traffic detection model M specified by the user (step 21); then starts iteration, and at each step, first randomly samples a sample batch B_S from S (step 22), calculates the cross entropy loss function value of M on S_T (step 23), and uses it to calculate the gradient of the model M (step 24); then, uses the calculated gradient to update the parameters of the model M (step 25); checks whether the model has converged (step 26, where the convergence indicator can be specified by the user, such as the change in the model parameters does not exceed a given threshold); if the model has converged, save M and end (step 27).
[0044] An efficient abnormal traffic detection system based on data reduction, including a data reduction module and a simplified abnormal traffic detection model training module;
[0045] The data reduction module obtains the real data in the base station database; randomly initializes the simulated data; randomly initializes an abnormal traffic detection model; uses the real data and simulated data to train the model respectively, and records the loss function and corresponding parameter gradient of the model on the two sets of data; calculates the Euclidean distance between the two sets of gradients; minimizes the Euclidean distance to update the simulated data; repeats the above process until the simulated data converges, records the simulated data, and clears the real data in the base station database; simplifies the abnormal traffic detection model training module, the user selects an abnormal traffic detection model and initializes the model; obtains the updated simulated data saved by the data reduction module; and trains the abnormal traffic detection model using the simulated data.
[0046] Obviously, those skilled in the art should understand that the various steps of the above-mentioned efficient abnormal traffic detection method based on data reduction or the various modules of the efficient abnormal traffic detection system based on data reduction of the embodiment of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiment of the present invention is not limited to any specific combination of hardware and software.
Claims
1. An efficient abnormal traffic detection method based on data reduction, characterized in that: Includes data reduction process and simplified abnormal traffic detection model training process; In the data reduction process, real data in the base station database is obtained; simulation data is randomly initialized; an abnormal traffic detection model is randomly initialized; Use real data and simulated data to train the model respectively, record the loss function of the model on the two sets of data and the corresponding parameter gradient; calculate the Euclidean distance between the two sets of gradients; minimize the Euclidean distance to update the simulated data; repeat the above process until the simulated data converges, record the simulated data, and clear the real data in the base station database; During the simplified abnormal traffic flow detection model training process, the user selects an abnormal traffic flow detection model and initializes the model; obtains previously saved simulation data; and uses the simulation data to train the model.
2. The efficient abnormal traffic detection method based on data reduction according to claim 1 is characterized in that: The data reduction process is specifically as follows: Step 100, obtaining an existing real data set in the base station database, which is denoted as T; Step 101, randomly initialize a simulation data set S; Step 102, specifying the maximum number of iterations K; Step 103, randomly initialize an abnormal traffic detection model M; Step 104, randomly sample a small batch of samples B_T from the real data set T, and randomly sample a small batch of samples B_S from the simulated data set S; Step 105, according to the model M, respectively calculate its loss function value l_T on B_T and its loss function value l_S on B_S; Step 106, calculating the gradients g_T and g_S on the parameters of the model M according to l_T and l_S respectively; Step 107, calculate the Euclidean distance between g_T and g_S, calculate the gradient on the small batch B_S of the simulated data set based on the Euclidean distance, and update the small batch of simulated data sets based on the optimization technique; Step 108, based on the optimization technique, using the gradient g_T to update the model M; Step 109, check whether the simulation data set S converges, if not converged and the current number of iterations does not reach K, return to step 104, otherwise go to step 110; Step 110, record the updated simulated data set S, and clear the real data set T in the database.
3. The efficient abnormal traffic detection method based on data reduction according to claim 1 is characterized in that: The simplified abnormal traffic detection model training process is specifically as follows: Step 200, obtaining an updated simulation data set S; Step 201, specifying the maximum number of iterations K; Step 202, randomly initialize an abnormal traffic detection model M; Step 203, randomly sampling a small batch of samples B_S from the simulated data set S; Step 204, based on the model M, calculate its loss function l_S on B_S; Step 205, calculating the gradient g_S of the parameters of the model M based on l_S; Step 206, based on the optimization technique, update the model M using the gradient g_S; Step 207, check whether the model M has converged, if not and the current number of iterations has not reached K, return to step 203, otherwise go to step 208; Step 208: output the trained model M.
4. The efficient abnormal traffic detection method based on data reduction according to claim 1 is characterized in that: A real data set T is obtained from a real database stored in a base station. The data in the real data set T is the historical records of real users and is marked with abnormalities or not.
5. The efficient abnormal traffic detection method based on data reduction according to claim 1 is characterized in that: A set of simulated data sets S is randomly initialized. This set of data sets is represented in the form of a matrix, where each row represents a sample and each column represents a feature.
6. The efficient abnormal traffic detection method based on data reduction according to claim 2 is characterized in that: The optimization technique refers to the gradient descent method, which is used to find a local optimal solution to the loss function and update the target parameters.
7. An efficient abnormal traffic detection system based on data reduction, characterized in that: Contains data reduction module and simplified abnormal traffic detection model training module; The data reduction module obtains real data from the base station database; randomly initializes simulation data; and randomly initializes an abnormal traffic detection model; Use real data and simulated data to train the model respectively, record the loss function of the model on the two sets of data and the corresponding parameter gradient; calculate the Euclidean distance between the two sets of gradients; minimize the Euclidean distance to update the simulated data; repeat the above process until the simulated data converges, record the simulated data, and clear the real data in the base station database; In the simplified abnormal traffic detection model training module, the user selects an abnormal traffic detection model and initializes the model; obtains the updated simulation data saved by the data reduction module; and uses the simulation data to train the abnormal traffic detection model.
8. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the efficient abnormal traffic detection method based on data reduction as described in any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the efficient abnormal traffic detection method based on data reduction as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Online collected electric power data classifying method
CN104809473A
Risk identification method and system
CN108629680A