A network intrusion detection model generation method, detection method and electronic device

Abstract

The present invention discloses a network intrusion detection model generation method, a detection method and an electronic device. The generation method obtains a public data set and a private data set; the public data set and the private data set are standardized by the traffic data in the public data set to obtain a first training data set and a second training data set; a parent model is trained according to the first training data set, and several child models are trained according to the second training data set and a first latent variable to obtain a network intrusion detection model. The present invention trains a parent model and several child models respectively by using a public data set and a private data set, and guides the learning of several child models by using a parent model, while ensuring that several child models learn the distribution of private data sets, the training effect is improved, and the traffic data of unknown attack type can be detected without collecting abnormal traffic in the target network environment, reducing the demand for data volume collection and improving the accuracy and robustness of abnormal traffic detection.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a network intrusion detection model generation method, a detection method and an electronic device. Background Art

[0002] With the rapid development of information technology, Internet applications have penetrated into every aspect of our lives. At the same time, the security issues faced by the Internet environment are becoming increasingly prominent. Network intrusion methods such as denial of service (DoS) attacks, botnet attacks, and infiltration attacks are emerging in an endless stream. These attacks can cause system downtime and affect the stability of system operation, or even lead to the theft of key information such as user privacy data and commercial secrets.

[0003] In order to detect these intrusion behaviors, existing intrusion detection methods usually extract traffic data on the network and then build a machine learning classification model for detection. However, these methods have the following disadvantages: First, they cannot effectively detect new attacks. New attack methods emerge in an endless stream on the network. From the emergence of a new attack method to its use to attack the target system, the existing detection system cannot obtain information about this attack method, so it is also called a "zero-day attack." Since the training data set cannot contain samples of "zero-day attacks", classification models cannot be used for effective detection; second, data sets are difficult to obtain. Public data sets usually do not contain the latest attack types, or the data is anonymized, resulting in reduced usability. The production of a large-scale private data set with comprehensive attack types requires complex intrusion testing and annotation, which is a huge workload.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the present invention provides a network intrusion detection model generation method, a detection method and an electronic device, aiming to solve the problem that the existing network intrusion detection method has unsatisfactory detection results due to unknown attack means and difficulty in data acquisition.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] A network intrusion detection model generation method, comprising:

[0008] Obtain a public data set for network intrusion detection and a private data set of the target network; wherein the public data set includes a number of normal traffic data and a number of abnormal traffic data, and the private data set includes a number of normal traffic data;

[0009] The public data set and the private data set are standardized by using the traffic data in the public data set to obtain a first training data set and a second training data set;

[0010] A parent model of a preset neural network model is trained according to the first training data set, and several child models of the preset neural network model are trained according to the second training data set and the first latent variable of the parent model to obtain a network intrusion detection model; wherein the first latent variable is a feature vector output by the Mth layer of the parent model, and M is a positive integer.

[0011] The network intrusion detection model generation method, wherein the step of normalizing the public data set and the private data set by using the traffic data in the public data set to obtain the first training data set and the second training data set comprises:

[0012] Obtain the mean and variance of several traffic data in the public data set;

[0013] The public data set and the private data set are standardized according to the mean and the variance to obtain a first training data set and a second training data set.

[0014] The network intrusion detection model generation method, wherein the step of training a parent model of a preset neural network model according to the first training data set comprises:

[0015] Inputting the first training data set into a parent model of a preset neural network model, and obtaining a predicted traffic label of the first training data set output by the parent model;

[0016] The model parameters of the parent model are updated according to the predicted traffic label and the actual traffic label of the first training data set until the training condition of the parent model meets the preset conditions.

[0017] The network intrusion detection model generation method, wherein the step of training several sub-models of the preset neural network model according to the second training data set and the first implicit variable of the parent model comprises:

[0018] Inputting the second training data set into several sub-models of a preset neural network model, obtaining the predicted flow value of the second training data set output by the several sub-models and the second latent variables of the several sub-models; wherein the second latent variable is a feature vector output by the Mth layer of the several sub-models;

[0019] The model parameters of the plurality of sub-models are updated according to the predicted flow value and the actual flow value of the second training data set as well as the first latent variable and the second latent variable until the training conditions of the plurality of sub-models meet preset conditions.

[0020] A network intrusion detection method, wherein the network intrusion detection model generated by the network intrusion detection model generation method is applied, comprising:

[0021] Standardizing the first target flow data sample by using the flow data in the public data set to obtain a second target flow data sample;

[0022] Inputting the second target traffic data sample into the network intrusion detection model to determine a target anomaly index of the first target traffic data sample;

[0023] According to the target anomaly index and a preset index threshold, determine whether the first target traffic data sample is invaded.

[0024] The network intrusion detection method, wherein the step of inputting the second target flow data sample into the network intrusion detection model and determining the target anomaly index of the first target flow data sample comprises:

[0025] Inputting the second target traffic data sample into the parent model of the network intrusion detection model, obtaining the target classification result output by the parent model and the third latent variable of the parent model; wherein the third latent variable is the feature vector output by the Mth layer of the parent model;

[0026] Inputting the second target traffic data sample into several sub-models of the network intrusion detection model, obtaining the target traffic values ​​output by the several sub-models and several fourth implicit variables of the several sub-models; wherein the several fourth implicit variables are the characteristic vectors output by the Mth layer of the several sub-models;

[0027] A target abnormality index of the first target flow data sample is determined according to the target classification result, the target flow value, the third latent variable and the plurality of fourth latent variables.

[0028] The network intrusion detection method, wherein the step of determining the target anomaly index of the first target traffic data sample according to the target classification result, the target traffic value, the third implicit variable and the plurality of fourth implicit variables comprises:

[0029] Determining a first abnormality index according to the target classification result;

[0030] Determining a second abnormality index according to the plurality of fourth implicit variables and the third implicit variable;

[0031] Determining a third abnormality index according to the plurality of fourth implicit variables;

[0032] determining a fourth abnormality index according to the target flow value and the actual flow value of the first target flow data sample;

[0033] A target abnormality index of the first target traffic data sample is determined according to the first abnormality index, the second abnormality index, the third abnormality index and the fourth abnormality index.

[0034] The network intrusion detection method, wherein the step of obtaining the index threshold comprises:

[0035] Inputting the second training data set into the parent model of the network intrusion detection model, obtaining the private data classification result output by the parent model and the fifth latent variable of the parent model; wherein the fifth latent variable is the feature vector output by the Mth layer of the parent model;

[0036] Inputting the second training data set into several sub-models of the network intrusion detection model, obtaining the private data flow values ​​output by the several sub-models and several sixth latent variables output by the several sub-models; wherein the sixth latent variable is a feature vector output by the Mth layer of the several sub-models;

[0037] Determining a plurality of private data anomaly indexes of the private data set according to the private data classification result, the private data flow value, the fifth latent variable and the plurality of sixth latent variables;

[0038] A private data anomaly index is selected from the plurality of private data anomaly indexes as the index threshold.

[0039] A terminal, comprising: a processor, a storage medium communicatively connected to the processor, the storage medium being suitable for storing a plurality of instructions; the processor being suitable for calling the instructions in the storage medium to execute the steps in the network intrusion detection model generation method or the steps in the network intrusion detection method.

[0040] A storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor to implement the steps in the above-mentioned network intrusion detection model generation method or the steps in the above-mentioned network intrusion detection method.

[0041] Beneficial effects of the present invention: The present invention trains a parent model and several child models respectively through public data sets and private data sets, and uses the parent model to guide the learning of several child models, thereby improving the training effect while ensuring that several child models learn the distribution of private data sets. It is not necessary to collect abnormal traffic in the target network environment to detect traffic data of unknown attack types, which reduces the demand for data collection and improves the accuracy and robustness of abnormal traffic detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of an embodiment of a method for generating a network intrusion detection model provided in Embodiment 1 of the present invention;

[0043] Figure 2 It is a structural diagram of a network intrusion detection model generation method provided in the first embodiment of the present invention;

[0044] Figure 3 is a flow chart of an embodiment of a network intrusion detection method provided in Embodiment 2 of the present invention;

[0045] Figure 4 This is a functional principle diagram of a terminal provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The network intrusion detection model generation method and the network intrusion detection method provided by the present invention can be applied to a terminal. The terminal can be, but is not limited to, various personal computers, laptops, mobile phones, tablet computers, vehicle-mounted computers, and portable wearable devices. The terminal of the present invention adopts a multi-core processor. The processor of the terminal can be at least one of a central processing unit (CPU), a graphics processing unit (GPU), a video processing unit (VPU), etc.

[0049] Embodiment 1

[0050] The existing network intrusion detection method extracts traffic data on the network and builds a machine learning classification model for detection. However, due to the difficulty in obtaining training data sets, public data sets usually do not contain the latest attack types, or the data is anonymized, resulting in unsatisfactory detection results of the classification model. The production of a large-scale private data set with comprehensive attack types requires complex intrusion testing and annotation, which is a huge workload.

[0051] In order to solve the above problems, a network intrusion detection model generation method is provided in the first embodiment of the present invention. Figure 1 , Figure 1 The present invention provides a flow chart of an embodiment of a method for generating a network intrusion detection model.

[0052] In one embodiment of the present invention, the network intrusion detection model generation method has three steps:

[0053] S100, obtaining a public data set for network intrusion detection and a private data set of a target network; wherein the public data set includes some normal traffic data and some abnormal traffic data, and the private data set includes some normal traffic data.

[0054] Since public data sets usually do not contain the latest attack types, and the workload of making a large-scale private data set with comprehensive attack types is huge, before training the preset network model in this embodiment, the existing public data sets for network intrusion detection on the network are obtained, such as the CICIDS2017 data set, the CSE-CIC-IDS2018 data set, etc., wherein the public data set contains some normal data traffic and some abnormal data traffic. Since public data sets usually do not contain the latest attack types, in this embodiment, existing tools such as Wireshark are further used to collect and parse some normal traffic data as private data sets in the target network environment. In this embodiment, when training the model, public data sets and private data sets are used at the same time, and the method of transfer learning is adopted. With the use of public data sets, there is no need to collect abnormal traffic data in the target network environment, and traffic data of unknown attack types can be detected to resist "zero-day attacks". At the same time, the demand for data collection and annotation of private data sets is reduced, and the practicality of network intrusion detection models is improved.

[0055] S200 , standardize the public data set and the private data set by using the traffic data in the public data set to obtain a first training data set and a second training data set.

[0056] Since the public data set contains some normal traffic and some abnormal traffic, after obtaining the public data set in this embodiment, the category to which each traffic data in the public data set belongs is labeled, so that the public data set can be used to train the preset network model to classify the traffic data in the subsequent steps. After the labeling is completed, the public data set is feature extracted using tools such as CICFlowMeter to obtain the traffic data in the public data set, and the traffic data in the public data set is used to standardize the public data set. Then, the labels of the previously labeled public data set are converted, and the labels of normal traffic data are converted to 0, and the labels of abnormal traffic data are converted to 1.

[0057] After obtaining the private data set, the same processing method as the public data set is used, that is, using tools such as CICFlowMeter to extract features from the private data set, obtain the flow data in the private data set, and use the flow data in the public data set to standardize the private data set. Then, the labels of the private data set are converted. Since the private data set does not contain abnormal flow data, its flow data labels are all 0 after conversion.

[0058] In a specific implementation, the step S200 specifically includes:

[0059] S210, obtaining the mean and variance of several flow data in the public data set;

[0060] S220 , standardize the public data set and the private data set according to the mean and the variance to obtain a first training data set and a second training data set.

[0061] In order to accelerate the convergence of the preset neural network model when training the preset neural network model, in this embodiment, after obtaining the traffic data in the public data set, the mean and variance of several traffic data in the public data set are calculated. Assuming that the public data set contains N F flow data, the characteristic vector of the i-th flow data is v i , then the mean variance

[0062] After obtaining the mean and variance of several traffic data in the public data set, according to the mean μ and the variance δ 2 The public data set and the private data set are standardized to obtain a first training data set and a second training data set. The public data set standardization formula is: Among them, x raw1 is a public data set, μ is the mean of several traffic data in the public data set, σ 2 is the variance of some traffic data in the public dataset, x 1 is the first training data set. Similar to the public data set normalization formula, the private data set normalization formula is: Among them, x raw2 is a private dataset, μ is the mean of several traffic data in the public dataset, σ 2 is the variance of some traffic data in the public dataset, x 2 is the second training data set.

[0063] S300. Train a parent model of a preset neural network model according to the first training data set, and train several child models of the preset neural network model according to the second training data set and the first latent variable of the parent model to obtain a network intrusion detection model; wherein the first latent variable is a feature vector output by the Mth layer of the parent model, and M is a positive integer.

[0064] like Figure 2 As shown, in this embodiment, a neural network model is pre-built, and the neural network model includes a parent model and several child models. After obtaining the first training data set and the second training data set, the parent model is trained with the first training data set so that the parent model can classify normal traffic data and abnormal traffic data, and the child model is trained with the second training data set to remove noise in the normal traffic data so that the child model can learn low-dimensional features.

[0065] Since the workload of obtaining private data sets is relatively large, in order to reduce the data volume demand as much as possible while ensuring the learning quality of the sub-model, in this embodiment, when training the sub-model, not only the second training data set but also the first latent variable of the parent model is used to train the sub-model, and the transfer learning and knowledge distillation methods are used to enable the sub-model to learn the parameters of the parent model.

[0066] In a specific implementation, the step of training the parent model of the preset neural network model according to the first training data set in step S300 specifically includes:

[0067] S310: input the first training data set into a parent model of a preset neural network model, and obtain a predicted traffic label of the first training data set output by the parent model;

[0068] S320. Update the model parameters of the parent model according to the predicted traffic label and the actual traffic label of the first training data set until the training status of the parent model meets the preset conditions.

[0069] In this embodiment, the parent model adopts a fully connected neural network structure, as shown in Table 1 below, which includes an input layer and several fully connected layers. F is the number of samples input into the parent model, and f is the feature dimension.

[0070] Table 1 Parent model network structure

[0071]

[0072]

[0073] When the first training data set is used to train the parent model, the first training data set is input into the parent model, which is a traffic data classification model that outputs the predicted traffic label of the first training data set. The predicted traffic label output by the parent model is then compared with the actual traffic label of the first training data set, and the model parameters of the parent model are updated according to the predicted traffic label and the actual traffic label of the first training data set until the training of the parent model meets the preset conditions.

[0074] When judging whether the training of the parent model meets the preset conditions, the loss function is used to calculate the loss value between the predicted traffic label and the real traffic label. Generally, the smaller the loss value, the better the performance of the parent model. After obtaining the loss value, it is judged whether the loss value is less than the preset threshold; if so, it means that the training of the parent model meets the preset conditions; if not, it means that the training of the parent model does not meet the preset conditions, then the model parameters of the parent model are updated according to the preset parameter learning rate, and the first training data set is input into the parent model to obtain the predicted traffic label until the loss value is less than the preset threshold. Among them, the loss function can be selected according to actual needs. In a specific embodiment, assuming that the input sample is x 1 , the sample size is N F , the feature dimension is f, and the predicted traffic label is The true label is y, and the loss function used to train the parent model is

[0075] In a specific implementation, the step of training several sub-models of the preset neural network model according to the second training data set and the first latent variable of the parent model in step S300 specifically includes:

[0076] R310, inputting the second training data set into several sub-models of the preset neural network model, obtaining the predicted flow value of the second training data set output by the several sub-models and the second implicit variables of the several sub-models; wherein the second implicit variable is the feature vector output by the Mth layer of the several sub-models;

[0077] R320. Update the model parameters of the plurality of sub-models according to the predicted flow value and the actual flow value of the second training data set as well as the first latent variable and the second latent variable until the training conditions of the plurality of sub-models meet preset conditions.

[0078] In this embodiment, the structures of several sub-models are the same, and all adopt the structure of autoencoder, as shown in Table 2. The encoder and decoder are both fully connected neural networks, and the structure of the sub-model encoder is similar to that of the parent model. Assume that the input sample is x 2 , the sample size is N C , the feature dimension is f (the same as the parent model), then the child model outputs samples It can be expressed as Among them, E is the encoder and D is the decoder.

[0079] Table 2 Network structure of sub-model

[0080]

[0081]

[0082] When the second training data set and the first latent variable are used to train several sub-models, the first training data set is input into several sub-models respectively. The sub-model is to remove the noise in the flow data and learn low-dimensional features, and it will output the predicted flow value of the second training data set. Since the workload of obtaining private data sets is large, in order to reduce the demand for data volume as much as possible and ensure the learning quality of the sub-model at the same time, when training several sub-models in this embodiment, the second latent variables of several sub-models are also obtained, that is, the characteristic vectors of the Mth layer output of several sub-models. Then the predicted flow value is compared with the actual flow value of the second training data set, and the second latent variable is compared with the first latent variable. According to the predicted flow value and the actual flow value of the second training data set and the first latent variable and the second latent variable, the model parameters of several sub-models are updated until the training of several sub-models meets the preset conditions.

[0083] Considering that the feature vector output by the child model after encoding is similar to the feature vector output by the same layer of the parent model, the M value is the number of the last layer of the encoder of the child model. In a specific embodiment, when the network structure of the child model is as shown in Table 2, the M value is 5, that is, the first latent variable is the feature vector output by the 5th layer of the parent model, and the second latent variable is the feature vector output by the last layer of the encoder of several child models, that is, the 5th layer of the model.

[0084] When judging that the training of several sub-models meets the preset conditions, the loss function is used to calculate the loss value between the predicted flow value and the actual flow value and the first implicit variable and the second implicit variable. Generally, the smaller the loss value, the better the performance of several sub-models. After obtaining the loss value, it is judged whether the loss value is less than the preset threshold; if so, it indicates that the training of several sub-models meets the preset conditions; if not, it indicates that the training of several sub-models does not meet the preset conditions, then the model parameters of several sub-models are updated according to the preset parameter learning rate, and the second training data set is input into the parent model to obtain the predicted flow value and the second implicit variable until the loss value is less than the preset threshold. Among them, the loss function can be selected according to actual needs. In a specific embodiment, it is assumed that the predicted flow value is The actual flow value is x 2 , the first implicit variable is The second implicit variable is The loss function used in training the sub-model is Among them, λ 1 and λ 2 is the weight coefficient, λ 1 ,λ 2 ∈[0,1],λ 1 +λ 2=1, In this embodiment, the reconstruction error of the sub-model is minimized when training the sub-model, that is, By training the autoencoder for reconstruction, the noise in normal traffic data can be removed and low-dimensional features can be learned; when training the child model, the error between the implicit variables of the parent model and the child model is minimized, that is, The child model learns the parameters of the parent model, which reduces the data volume requirement of the private dataset and ensures the learning quality of the child model.

[0085] Embodiment 2

[0086] Based on the network intrusion detection model generated by the above network intrusion detection model generation method, this embodiment also provides a network intrusion detection method, such as Figure 3 As shown, the network intrusion detection method includes:

[0087] M100. Standardize the first target flow data sample using the flow data in the public data set to obtain a second target flow data sample.

[0088] Specifically, the first target traffic data sample is a data traffic sample that needs to be used for network intrusion detection, which can be collected in the same way as the private data set, for example, using tools such as Wireshark. Then, the first target traffic data sample is standardized using the same processing method as the public data set and the private data set, that is, the first target traffic data sample is standardized using the mean and variance of several traffic data in the public data set to obtain the second target traffic data sample. Among them, the standardization formula for the first target traffic data sample is: Among them, x raw3 is the first target traffic data sample, μ is the mean of several traffic data in the public data set, σ 2 is the variance of some traffic data in the public dataset, x 3 is the second target traffic data sample.

[0089] M200. Input the second target traffic data sample into the network intrusion detection model to determine the target anomaly index of the first target traffic data sample.

[0090] Specifically, after obtaining the second target traffic data sample, the second target traffic data sample is input into the previously trained network intrusion detection model, and the target anomaly index of the first target traffic data sample is determined according to the output result of the network intrusion detection model, so that in the subsequent steps, it can be judged whether the first target traffic data has been invaded according to the target anomaly index.

[0091] In a specific implementation, the step M200 specifically includes:

[0092] M210, input the second target traffic data sample into the parent model of the network intrusion detection model, obtain the target classification result output by the parent model and the third latent variable of the parent model; wherein the third latent variable is the feature vector output by the Mth layer of the parent model;

[0093] M220, input the second target flow data sample into several sub-models of the network intrusion detection model, obtain the target flow values ​​output by the several sub-models and several fourth implicit variables of the several sub-models; wherein the several fourth implicit variables are the characteristic vectors output by the Mth layer of the several sub-models;

[0094] M230. Determine a target abnormality index of the first target flow data sample according to the target classification result, the target flow value, the third latent variable and the plurality of fourth latent variables.

[0095] In this embodiment, an integrated learning method is adopted to use the prediction results of multiple models to determine the target anomaly index of the first target traffic data sample. Specifically, the second target traffic data sample is input into the parent model of the network intrusion detection model to obtain the target classification result output by the parent model and the third implicit variable of the parent model, that is, the feature vector output by the Mth layer of the parent model after the second target traffic data is input into the parent model. The second target traffic data sample is input into several sub-models of the network intrusion detection model to obtain the target traffic values ​​output by several sub-models and several fourth implicit variables of several sub-models, that is, the feature vectors output by the Mth layer of several sub-models after the second target traffic data sample is input into several sub-models. Then, based on the target classification result, the target traffic value, the third implicit variable and the fourth implicit variable, the target anomaly index of the first target traffic data sample is determined.

[0096] In a specific embodiment, step M230 specifically includes:

[0097] M231. Determine a first abnormality index according to the target classification result;

[0098] M232. Determine a second abnormality index according to the plurality of fourth implicit variables and the third implicit variable;

[0099] M233. Determine a third abnormality index according to the plurality of fourth implicit variables;

[0100] M234. Determine a fourth abnormality index according to the target flow value and the actual flow value of the first target flow data sample;

[0101] M235. Determine a target abnormality index of the first target traffic data sample according to the first abnormality index, the second abnormality index, the third abnormality index and the fourth abnormality index.

[0102] Specifically, the target anomaly index consists of four parts, namely the first anomaly index, the second anomaly index, the third anomaly index and the fourth anomaly index. Among them, the first anomaly index is determined according to the classification result output by the parent model. Assuming that the second target traffic data sample is x 3 , input the second target traffic data sample into the parent model for forward propagation to obtain the classification result output by the parent model The larger the value, the higher the possibility that the first target traffic data sample is abnormal traffic. 1 =F(x 3 ). The second abnormal index is determined based on several fourth latent variables and the third latent variables. Assume that the fourth latent variable of each sub-model is The third implicit variable of the parent model is Then the mean vector of several fourth implicit variables is The error between the mean vector of the fourth latent variable and the third latent variable is the second abnormal index, which can be expressed as When the first target traffic data sample is a normal traffic sample, the third latent variable and the fourth latent variables are similar. On the contrary, when the first target traffic data sample is an abnormal traffic sample, the third latent variable and the fourth latent variables are quite different. 2 The larger the value is, the higher the possibility that the first target traffic data sample is abnormal traffic. The third abnormality index is based on several fourth latent variables. Assume that the fourth latent variable of each sub-model is Variance vectors between several fourth latent variables The third abnormal index I 3 ,Right now When the first target traffic data sample is a normal traffic sample, the fourth implicit variables of each sub-model are similar. On the contrary, when the first target traffic data sample is an abnormal traffic sample, the fourth implicit variables of each sub-model are quite different, that is, I 3 The larger the value is, the higher the possibility that the first target flow data sample is abnormal flow. The fourth abnormality index is determined according to the target flow value and the actual flow value of the first target flow data sample. Assuming that the target flow value is The actual flow value of the first target flow data sample is x, then the fourth abnormal index I 4 is the difference in reconstruction results of each submodule, that is, When the first target flow data sample is a normal flow sample, the reconstruction results of each sub-model are similar. On the contrary, when the first target flow data sample is an abnormal flow sample, the reconstruction results of each sub-model are quite different. 4 The larger the value, the higher the possibility that the sample is abnormal traffic.

[0103] After obtaining the first abnormality index, the second abnormality index, the third abnormality index and the fourth abnormality index, the target abnormality index of the first target flow data sample can be determined according to the first abnormality index, the second abnormality index, the third abnormality index and the fourth abnormality index. The calculation formula of the target abnormality index is: x =λ 1 I 1 +λ 2 I 2 +λ 3 I 3 +λ 4 I 4 , where λ 1 , 2 , 3 and λ 4 is the weight coefficient,

[0104] M300. Determine whether the first target traffic data sample is invaded based on the target anomaly index and a preset index threshold.

[0105] In this embodiment, an index threshold λ is preset p After determining the target anomaly index of the first target traffic data sample, compare the target anomaly index of the first target traffic data sample with the preset index threshold. When the target anomaly index is greater than the preset index threshold, it is judged that the first target traffic data sample has been invaded; when the target anomaly index is less than or equal to the preset index threshold, it is judged that the first target traffic data sample has not been invaded.

[0106] In a specific embodiment, the step of obtaining the index threshold in step M300 includes:

[0107] M310. Input the second training data set into the parent model of the network intrusion detection model, and obtain the private data classification result output by the parent model and the fifth latent variable of the parent model; wherein the fifth latent variable is the feature vector output by the Mth layer of the parent model;

[0108] M320. Input the second training data set into several sub-models of the network intrusion detection model, and obtain the private data flow values ​​output by the several sub-models and several sixth implicit variables output by the several sub-models; wherein the sixth implicit variable is a feature vector output by the Mth layer of the several sub-models;

[0109] M330. Determine a number of private data anomaly indices of the private data set according to the private data classification result, the private data traffic value, the fifth latent variable, and the number of sixth latent variables.

[0110] M340. Select one private data anomaly index from the number of private data anomaly indices as the index threshold.

[0111] The method for obtaining the index threshold is similar to the method for obtaining the target anomaly index. After generating the parent model of network intrusion detection, input the second training data set into the parent model of the network intrusion detection model to obtain the private data classification result output by the parent model and the fifth latent variable of the parent model, that is, the feature vector output by the M-th layer of the parent model after inputting the second training data set. Input the second training data set into a number of sub-models of the network intrusion detection model to obtain the private data traffic values output by the number of sub-models and the number of sixth latent variables of the number of sub-models, that is, the feature vectors output by the M-th layer of the number of sub-models after inputting the second training data set. Then, according to the private data classification result, the private data traffic value, the fifth latent variable, and the number of sixth latent variables, determine a number of private data anomaly indices of the private data set. Among them, the method for determining a number of private data anomaly indices of the private data set is similar to the steps for determining the target anomaly index in the foregoing steps, that is, determine the fifth anomaly index according to the private data classification result output by the parent model, determine the sixth anomaly index according to the number of sixth latent variables and the fifth latent variable, determine the seventh anomaly index according to the variance vector between the number of sixth latent variables, determine the eighth anomaly index according to the private data traffic value and the true value of the data traffic in the second training data set, and determine a number of private data anomaly indices of the private data set according to the fifth anomaly index, the sixth anomaly index, the seventh anomaly index, and the eighth anomaly index.

[0112] After obtaining a number of private data anomaly indices of the private data set, sort the number of private data anomaly indices from smallest to largest, and take the value of the p-th quantile as the preset index threshold λ p , where 0 < p < 1. The larger p is, the higher the threshold is, and the smaller the possibility of judging as abnormal traffic is.

[0113] Embodiment III

[0114] Based on the above embodiments, the present invention also provides a terminal, and its principle block diagram can be as Figure 4As shown. The terminal includes a processor, a memory, a network interface, a display screen and a temperature sensor connected through a system bus. Among them, the processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a network intrusion detection model generation method and a network intrusion detection method are implemented. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor of the terminal is pre-set inside the device to detect the current operating temperature of the internal device.

[0115] Those skilled in the art will understand that Figure 4 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0116] In one embodiment, a terminal is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, at least the following steps can be implemented:

[0117] Obtain a public data set for network intrusion detection and a private data set of the target network; wherein the public data set includes a number of normal traffic data and a number of abnormal traffic data, and the private data set includes a number of normal traffic data;

[0118] The public data set and the private data set are standardized by using the traffic data in the public data set to obtain a first training data set and a second training data set;

[0119] A parent model of a preset neural network model is trained according to the first training data set, and several child models of the preset neural network model are trained according to the second training data set and the first latent variable of the parent model to obtain a network intrusion detection model; wherein the first latent variable is a feature vector output by the Mth layer of the parent model, and M is a positive integer.

[0120] In one embodiment, when the processor executes the computer program, it can also achieve: obtaining the mean and variance of several traffic data in the public data set; standardizing the public data set and the private data set according to the mean and the variance to obtain a first training data set and a second training data set.

[0121] In one embodiment, when the processor executes the computer program, it can also achieve: inputting the first training data set into the parent model of the preset neural network model, obtaining the predicted traffic label of the first training data set output by the parent model; updating the model parameters of the parent model according to the predicted traffic label and the actual traffic label of the first training data set until the training status of the parent model meets the preset conditions.

[0122] In one embodiment, when the processor executes the computer program, it can also achieve: inputting the second training data set into several sub-models of the preset neural network model, obtaining the predicted flow value of the second training data set output by the several sub-models and the second implicit variables of the several sub-models; wherein the second implicit variable is the characteristic vector output by the Mth layer of the several sub-models; updating the model parameters of the several sub-models according to the predicted flow value and the actual flow value of the second training data set and the first implicit variable and the second implicit variable until the training conditions of the several sub-models meet the preset conditions.

[0123] In one embodiment, when the processor executes the computer program, it can also achieve: standardizing the first target traffic data sample through the traffic data in the public data set to obtain the second target traffic data sample; inputting the second target traffic data sample into the network intrusion detection model to determine the target anomaly index of the first target traffic data sample; and determining whether the first target traffic data sample has been invaded based on the target anomaly index and a preset index threshold.

[0124] In one embodiment, when the processor executes the computer program, it can also achieve: inputting the second target traffic data sample into the parent model of the network intrusion detection model, obtaining the target classification result output by the parent model and the third implicit variable of the parent model; wherein the third implicit variable is the characteristic vector output by the Mth layer of the parent model; inputting the second target traffic data sample into several sub-models of the network intrusion detection model, obtaining the target traffic values ​​output by the several sub-models and several fourth implicit variables of the several sub-models; wherein the several fourth implicit variables are the characteristic vectors output by the Mth layer of the several sub-models; determining the target anomaly index of the first target traffic data sample according to the target classification result, the target traffic value, the third implicit variable and the several fourth implicit variables.

[0125] In one embodiment, when the processor executes the computer program, it can also achieve: determining a first abnormality index based on the target classification result; determining a second abnormality index based on the several fourth implicit variables and the third implicit variable; determining a third abnormality index based on the several fourth implicit variables; determining a fourth abnormality index based on the target flow value and the actual flow value of the first target flow data sample; determining the target abnormality index of the first target flow data sample based on the first abnormality index, the second abnormality index, the third abnormality index and the fourth abnormality index.

[0126] In one embodiment, when the processor executes the computer program, it can also achieve: inputting the second training data set into the parent model of the network intrusion detection model, obtaining the private data classification result output by the parent model and the fifth implicit variable of the parent model; wherein the fifth implicit variable is the characteristic vector output by the Mth layer of the parent model; inputting the second training data set into several child models of the network intrusion detection model, obtaining the private data flow values ​​output by the several child models and several sixth implicit variables output by the several child models; wherein the sixth implicit variable is the characteristic vector output by the Mth layer of the several child models; determining several private data anomaly indexes of the private data set according to the private data classification result, the private data flow value, the fifth implicit variable and the several sixth implicit variables; and selecting a private data anomaly index from the several private data anomaly indexes as the index threshold.

[0127] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0128] In summary, the present invention discloses a network intrusion detection model generation method, a detection method and an electronic device, wherein the generation method obtains a public data set for network intrusion detection and a private data set of a target network; the public data set and the private data set are standardized by the traffic data in the public data set to obtain a first training data set and a second training data set; a parent model of a preset neural network model is trained according to the first training data set, and several sub-models of the preset neural network model are trained according to the second training data set and the first latent variable to obtain a network intrusion detection model. The present invention trains a parent model and several sub-models respectively by using a public data set and a private data set, and uses the parent model to guide the learning of several sub-models, thereby improving the training effect while ensuring that several sub-models learn the distribution of the private data set, and can detect traffic data of unknown attack type without collecting abnormal traffic in the target network environment, thereby reducing the demand for data volume collection and improving the accuracy and robustness of abnormal traffic detection.

[0129] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A network intrusion detection model generation method, characterized in that: include: Obtain a public data set for network intrusion detection and a private data set of the target network; wherein the public data set includes a number of normal traffic data and a number of abnormal traffic data, and the private data set includes a number of normal traffic data; The public data set and the private data set are standardized by using the traffic data in the public data set to obtain a first training data set and a second training data set; The parent model of the preset neural network model is trained according to the first training data set, and several sub-models of the preset neural network model are trained according to the second training data set and the first latent variable of the parent model to obtain a network intrusion detection model; wherein the first latent variable is a feature vector output by the Mth layer of the parent model, and M is a positive integer; The step of training a parent model of a preset neural network model according to the first training data set includes: Inputting the first training data set into a parent model of a preset neural network model, and obtaining a predicted traffic label of the first training data set output by the parent model; Updating the model parameters of the parent model according to the predicted traffic label and the actual traffic label of the first training data set until the training condition of the parent model meets the preset conditions; The step of training several sub-models of the preset neural network model according to the second training data set and the first implicit variable of the parent model comprises: Inputting the second training data set into several sub-models of a preset neural network model, obtaining the predicted flow value of the second training data set output by the several sub-models and the second latent variables of the several sub-models; wherein the second latent variable is a feature vector output by the Mth layer of the several sub-models; Updating the model parameters of the plurality of sub-models according to the predicted flow value and the actual flow value of the second training data set as well as the first latent variable and the second latent variable until the training conditions of the plurality of sub-models meet preset conditions; The parent model adopts a fully connected neural network structure, including an input layer and several fully connected layers, and the child model adopts an autoencoder structure.

2. The network intrusion detection model generation method according to claim 1, characterized in that: The step of normalizing the public data set and the private data set by using the traffic data in the public data set to obtain the first training data set and the second training data set comprises: Obtain the mean and variance of several traffic data in the public data set; The public data set and the private data set are standardized according to the mean and the variance to obtain a first training data set and a second training data set.

3. A network intrusion detection method, characterized in that: A network intrusion detection model generated by the network intrusion detection model generation method according to any one of claims 1 to 2, comprising: Standardizing the first target flow data sample by using the flow data in the public data set to obtain a second target flow data sample; Inputting the second target traffic data sample into the network intrusion detection model to determine a target anomaly index of the first target traffic data sample; According to the target anomaly index and a preset index threshold, determine whether the first target traffic data sample is invaded.

4. The network intrusion detection method according to claim 3, characterized in that: The step of inputting the second target traffic data sample into the network intrusion detection model to determine the target anomaly index of the first target traffic data sample comprises: Inputting the second target traffic data sample into the parent model of the network intrusion detection model, obtaining the target classification result output by the parent model and the third latent variable of the parent model; wherein the third latent variable is the feature vector output by the Mth layer of the parent model; Inputting the second target traffic data sample into several sub-models of the network intrusion detection model, obtaining the target traffic values ​​output by the several sub-models and several fourth implicit variables of the several sub-models; wherein the several fourth implicit variables are the characteristic vectors output by the Mth layer of the several sub-models; A target abnormality index of the first target flow data sample is determined according to the target classification result, the target flow value, the third latent variable and the plurality of fourth latent variables.

5. The network intrusion detection method according to claim 4, characterized in that: The step of determining the target abnormality index of the first target flow data sample according to the target classification result, the target flow value, the third latent variable and the plurality of fourth latent variables comprises: Determining a first abnormality index according to the target classification result; Determining a second abnormality index according to the plurality of fourth implicit variables and the third implicit variable; Determining a third abnormality index according to the plurality of fourth implicit variables; determining a fourth abnormality index according to the target flow value and the actual flow value of the first target flow data sample; A target abnormality index of the first target traffic data sample is determined according to the first abnormality index, the second abnormality index, the third abnormality index and the fourth abnormality index.

6. The network intrusion detection method according to claim 4, characterized in that: The step of obtaining the index threshold comprises: Inputting the second training data set into the parent model of the network intrusion detection model, obtaining the private data classification result output by the parent model and the fifth latent variable of the parent model; wherein the fifth latent variable is the feature vector output by the Mth layer of the parent model; Inputting the second training data set into several sub-models of the network intrusion detection model, obtaining the private data flow values ​​output by the several sub-models and several sixth latent variables output by the several sub-models; wherein the sixth latent variable is a feature vector output by the Mth layer of the several sub-models; Determining a plurality of private data anomaly indexes of the private data set according to the private data classification result, the private data flow value, the fifth latent variable and the plurality of sixth latent variables; A private data anomaly index is selected from the plurality of private data anomaly indexes as the index threshold.

7. A terminal, characterized in that: include: A processor and a storage medium communicatively connected to the processor, wherein the storage medium is suitable for storing a plurality of instructions; the processor is suitable for calling the instructions in the storage medium to execute the steps in the network intrusion detection model generation method described in any one of claims 1 to 2, or the steps in the network intrusion detection method described in any one of claims 3 to 6.

8. A storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps in the network intrusion detection model generation method described in any one of claims 1-2, or the steps in the network intrusion detection method described in any one of claims 3-6.

Images