A network intrusion detection method and system based on big data
Through deep convolution, feature selection of adversarial networks and gray wolf algorithms is generated, combined with the federated learning framework model, the problem of lack of authenticity of data samples and low feature selection efficiency in traditional network intrusion detection methods is solved, and efficient and accurate network intrusion is achieved Detect and fast model updates.
Patent Information
- Application Number
- CN202411174558.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-26
AI Technical Summary
Traditional network intrusion detection methods lack authenticity and diversity of data samples, making it difficult to adapt to changes in network traffic data distribution, low feature selection efficiency, slow model update speed, and timely response to changing cyber attack methods.
Deep convolution generation adversarial network is used to synthesize a few types of data samples, combine the improved teaching-learning optimization algorithm of Gray Wolf algorithm for feature selection, and use the federated learning framework model for network intrusion detection to realize distributed computing and incremental learning.
Improve the accuracy and performance of network intrusion detection, can respond to intrusion behavior in a timely manner, adapt to changes in different data distributions and intrusion behavior, and quickly update the model to deal with new threats.
Smart Images

Figure CN118694615B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of network intrusion detection, and in particular to a network intrusion detection method and system based on big data. Background Art
[0002] Network intrusion detection is an information security technology used to detect and record suspicious activities or violations of security policies in a network. It does this by monitoring network traffic and comparing it with known attack patterns or abnormal behaviors to identify potential network attacks or security threats.
[0003] However, the minority data sample synthesis method used by traditional network intrusion detection lacks sufficient authenticity and diversity, and cannot accurately reflect the intrusion situation in the real world. At the same time, it requires manual intervention and experience tuning, and is difficult to adapt to the technical problems of changes in different network traffic data distributions and intrusion behaviors; when performing feature selection, traditional network intrusion detection has the problem that when processing large-scale intrusion features, the global search and search for the optimal feature set are limited, and it is easy to fall into the local optimal solution, resulting in the failure to find the most discriminative feature combination; traditional network intrusion detection methods are inefficient when processing and analyzing large amounts of network traffic data, resulting in insufficient real-time performance and inability to respond to intrusion behaviors in a timely manner. At the same time, traditional methods have slow model update speeds and poor generalization capabilities, making it difficult to keep up with the ever-evolving changes in network attack methods. Summary of the invention
[0004] In view of the above situation, in order to overcome the defects of the prior art, the present invention provides a network intrusion detection method and system based on big data. The method for synthesizing minority data samples used in traditional network intrusion detection lacks sufficient authenticity and diversity, cannot accurately reflect the intrusion situation in the real world, and requires manual intervention and experience tuning, and is difficult to adapt to the changes in different network traffic data distributions and intrusion behaviors. The present invention creatively uses a deep convolutional generative adversarial network to synthesize minority data samples. By autonomously learning the characteristics and patterns of the data, it can better capture complex intrusion features and generate more distinguishable synthetic samples, accurately reflecting the network intrusion situation in the real world. At the same time, it has strong generalization ability and can adapt to changes in different data distributions and intrusion behaviors. In view of the fact that traditional network intrusion detection has limited global search and optimal feature set capabilities when processing large-scale intrusion features during feature selection, it is easy to fall into local optimal solutions, resulting in failure to detect the real-world network intrusions. To solve the technical problem of finding the most discriminative feature combination, this solution creatively adopts an improved teaching-learning optimization algorithm combined with the Gray Wolf Algorithm for feature selection. When processing and analyzing a large number of intrusion features, it has better global search capabilities and faster convergence speed, can effectively overcome the problem of local optimal solutions, find the optimal feature set, and improve the accuracy and performance of network intrusion detection; in view of the fact that traditional network intrusion detection methods are inefficient when processing and analyzing a large amount of network traffic data, resulting in insufficient real-time performance and inability to respond to intrusion behaviors in a timely manner, and at the same time, the traditional method model has slow model update speed and poor generalization ability, and it is difficult to keep up with the changes in the evolving network attack methods. This solution creatively adopts a federated learning framework model for network intrusion detection, adopts distributed computing, allows each local model to calculate in parallel, improves the efficiency of data processing, and can respond to intrusion behaviors in a timely manner. At the same time, it adopts an incremental learning method, which can quickly update the model according to new data and adapt to new threats.
[0005] The technical solution adopted by the present invention is as follows: The present invention provides a network intrusion detection method based on big data, which comprises the following steps:
[0006] Step S1: data collection;
[0007] Step S2: data preprocessing;
[0008] Step S3: feature selection;
[0009] Step S4: data sample synthesis;
[0010] Step S5: Network intrusion detection.
[0011] Furthermore, in step S1, the data collection is used to collect the original data required for network intrusion detection, specifically, to obtain the original data set for network intrusion detection from the security information and event management system through collection;
[0012] The network intrusion detection original data set specifically includes non-DDoS attack malicious traffic data and DDoS attack malicious traffic data.
[0013] Furthermore, in step S2, the data preprocessing is used to preprocess the original data, and specifically includes the following steps:
[0014] Step S21: feature encoding, which is used to convert the original data into digital form, specifically encoding the network intrusion detection original data set by a one-hot encoding method to obtain a network intrusion detection encoded data set;
[0015] Step S22: data normalization, which is used to normalize the data, specifically, normalizing the network intrusion detection coded data set by a minimum-maximum method to obtain a preliminary network intrusion detection data set;
[0016] Step S23: performing data preprocessing, specifically performing data preprocessing on the original data set of network intrusion detection through the feature encoding and the data normalization to obtain a preliminary data set of network intrusion detection.
[0017] Further, in step S3, the feature selection is used to perform feature selection on large-scale intrusion features, specifically using an improved teaching-learning optimization algorithm combined with a gray wolf algorithm to perform feature selection on a preliminary data set for network intrusion detection to obtain a feature optimized data set, specifically including the following steps:
[0018] Step S31: executing the teaching strategy, specifically constructing a teacher unit and a student unit group, and optimizing the parameters of the student unit group by constructing the teacher unit;
[0019] The teacher unit is used to represent an optimal feature set, and the student unit group is used to represent a non-optimal feature set;
[0020] The formula used in the implementation teaching strategy is as follows:
[0021] ;
[0022] In the formula, represents the position calculation function after the iteration of the i-th student unit in the teaching stage, t represents the number of iterations, represents a random vector with a modulus value in [0,1], represents the position calculation function after the teacher iteration in the teaching phase;
[0023] Step S32: Execute the interactive learning strategy combined with the gray wolf algorithm, specifically construct a leader unit group, and optimize the parameters of the student unit group through the leader unit group. The leader unit group specifically includes a first leader , Second Leader and the third leader , the first leader , used to represent the best student unit in the student unit group, the second leader Used to indicate the second best student unit in the student unit group, the third leader Used to represent the third best student unit in the student unit group;
[0024] The interactive learning strategy combined with the grey wolf algorithm comprises the following steps:
[0025] Step S321: construct a linear reduction vector, the formula used is as follows:
[0026] ;
[0027] In the formula, is a linear reduction vector that decreases linearly from 2 to 0 during the iteration process, and MaxT represents the maximum number of iterations;
[0028] Step S322: Calculating the distance between the student units in the student unit group and the leaders in the leader unit group, the steps include:
[0029] Step S3221: Calculate the relationship between the i-th student unit and the first leader The distance between them is calculated using the following formula:
[0030] ;
[0031] In the formula, Indicates the relationship between the i-th student unit and the first leader The distance between represents a random vector with a modulus value in [0,1], Indicates the first leader location, represents the position calculation function after iteration of the i-th student unit in the interactive learning phase;
[0032] Step S3222: Calculate the relationship between the i-th student unit and the second leader The distance between them is calculated using the following formula:
[0033] ;
[0034] In the formula, Indicates the relationship between the i-th student unit and the second leader The distance between represents a random vector with a modulus value in [0,1], Indicates the second leader location;
[0035] Step S3223: Calculate the relationship between the i-th student unit and the third leader The distance between them is calculated using the following formula:
[0036] ;
[0037] In the formula, Indicates the relationship between the i-th student unit and the third leader The distance between represents a random vector with a modulus value in [0,1], Indicates the third leader location;
[0038] Step S323: Calculate the position of the student unit after the t+1th iteration using the following formula:
[0039] ;
[0040] In the formula, represents a random vector with a modulus value in [0,1], represents a random vector with a modulus value in [0,1], Represents a random vector with a modulus value in [0,1];
[0041] Step S33: executing an autonomous learning strategy to optimize the parameters of the student unit group, specifically calculating the mean absolute difference of the student unit as the fitness of the student unit, assigning a ranking to the student unit based on the fitness, and calculating the mutation probability of the student unit based on the ranking to optimize the parameters of the student unit;
[0042] The steps of executing the autonomous learning strategy include:
[0043] Step S331: Calculate the mean of the characteristics of the student unit. The formula used is as follows:
[0044] ;
[0045] In the formula, represents the mean of the features of the i-th student unit, J represents the number of features of the i-th student unit, represents the jth feature of the i-th student unit;
[0046] Step S332: Calculate the average absolute difference of student units using the following formula:
[0047] ;
[0048] In the formula, represents the mean absolute difference of the i-th student unit;
[0049] Step S333: assigning rankings, specifically using the average absolute difference of the student unit as the fitness of the student unit, and ranking the student units in descending order based on the fitness to obtain a student unit ranking list. The formula used is as follows:
[0050] ;
[0051] In the formula, represents the ranked list of student units, represents the worst student unit, represents the second worst student unit, represents the optimal student unit, num represents the total number of student units;
[0052] Step S334: Calculate the mutation probability, using the following formula:
[0053] ;
[0054] In the formula, r represents the ranking of the student unit, represents the mutation probability of the student unit ranked r, represents the maximum mutation probability;
[0055] Step S34: performing feature selection, specifically performing feature selection by executing the teaching strategy, executing the interactive learning strategy combined with the Grey Wolf Algorithm, and executing the autonomous learning strategy to obtain a feature optimized data set.
[0056] Furthermore, in step S4, the data sample synthesis is used to deal with the problem of data imbalance, specifically using a deep convolutional generative adversarial network to generate minority class data samples in the feature optimization data set to obtain a network intrusion detection data set;
[0057] The deep convolutional generative adversarial network is used to generate minority class data samples, specifically including a convolution generator G and a convolution discriminator D. The convolution generator G and the convolution discriminator D are continuously optimized in adversarial training to obtain realistic minority class data samples;
[0058] The data sample synthesis specifically includes the following steps:
[0059] Step S41: feature optimization data set division, which is used to perform data division, specifically, to obtain a majority class data set and a minority class data set by dividing the feature optimization data set;
[0060] Step S42: construct the objective function, the formula used is as follows:
[0061] ;
[0062] In the formula, ~ indicates that the variable follows a probability distribution, is the objective function, represents the data distribution of the minority class dataset, represents the data distribution of the majority class data set, and x represents the data distribution from The actual sample, z represents the data distribution The actual sample, represents the sample generated by the convolution generator G based on the actual sample z, It represents the output probability that the convolution discriminator D correctly classifies the actual sample x as the actual sample, Represents samples that are misclassified by the convolutional discriminator D is the output probability of the actual sample, Represents that for data from the distribution The actual sample x, the convolution discriminator D correctly classifies the actual sample x as the expected value of the actual sample, Represents the sample generated based on the actual sample z , the convolutional discriminator D correctly classifies is the expected value of the non-actual sample;
[0063] Step S43: Construct the loss function of the convolution generator G. The formula used is as follows:
[0064] ;
[0065] In the formula, The character representation of the loss function of the convolutional generator G, N represents the number of training samples, represents the i-th sample generated by the convolution generator G, Represents the i-th sample that the convolution discriminator D will generate The output probability of being misclassified as a real sample;
[0066] Step S44: Construct the loss function of the convolution discriminator D. The formula used is as follows:
[0067] ;
[0068] In the formula, Character representation of the loss function of the convolutional discriminator D, It means that the convolution discriminator D takes the ith actual sample The output probability of correctly classifying as actual samples;
[0069] Step S45: Model training, specifically, performing the deep convolutional generative adversarial network model training through the feature optimization data set partitioning, the construction objective function, the loss function of the convolution generator G, and the loss function of the convolution discriminator D to obtain a realistic minority class data sample generation model ;
[0070] Step S46: constructing a network intrusion detection data set, specifically generating a model through the simulated minority class data samples Generate realistic minority class data samples and obtain a network intrusion detection dataset.
[0071] Further, in step S5, the network intrusion detection is used to detect whether a DDoS attack occurs, specifically by constructing a federated learning framework model and performing model training, and using the trained DDoS attack detection model to perform network intrusion detection to obtain DDoS attack judgment data;
[0072] The federated learning framework model specifically includes a general transformer model, a main model and a local model. Local parameters are obtained by training the general transformer model on the local model, and the local parameters are sent to the main model for local parameter aggregation and global model parameter update to obtain the DDoS attack detection model.
[0073] The universal transformer model is used for model training, the main model is used for aggregating local parameters and updating global model parameters, the local model is used for training the universal transformer model in parallel and sending the local parameters to the main model, and the local parameters are used to represent the model parameters after the universal transformer model training is performed on the local model;
[0074] The network intrusion detection specifically includes the following steps:
[0075] Step S51: data partitioning, which is used to partition the network intrusion detection data set, specifically, partitioning the network intrusion detection data set to obtain a local training set and a network intrusion detection test set;
[0076] Step S52: constructing a universal transformer model, wherein the universal transformer model specifically includes a multi-head attention module and a feedforward neural network module;
[0077] The steps of constructing a general transformer model include:
[0078] Step S521: construct a multi-head attention module, the steps include:
[0079] Step S5211: Calculate the dot product attention, the formula used is as follows:
[0080] ;
[0081] In the formula, Q represents the query vector, K represents the key vector, and V represents the value vector. represents the softmax function, represents the feature dimension, T represents the transposition operation, Represents the dot product attention calculation function;
[0082] Step S5212: Calculate the feature vector of the head using the following formula:
[0083] ;
[0084] In the formula, represents the feature vector of the i-th head, represents the weight matrix in the i-th head used to map the query vector to different subspaces for attention calculation, represents the weight matrix in the i-th head used to map the key vector to different subspaces for attention calculation, represents the weight matrix in the i-th head used to map the value vector to different subspaces for attention calculation;
[0085] Step S5213: Construct a multi-head attention function. The formula used is as follows:
[0086] ;
[0087] In the formula, represents the output function of the multi-head attention module, Concat represents the vector concatenation operation, represents the output weight matrix;
[0088] Step S522: constructing a feedforward neural network module, wherein the feedforward neural network module is specifically composed of two linear transformation layers using a ReLU activation function;
[0089] The feedforward neural network module uses the following formula:
[0090] ;
[0091] In the formula, x represents the input vector, represents the output function of the feedforward neural network module, represents the maximum value function, represents the weight matrix of the first linear transformation layer, represents the weight matrix of the second linear transformation layer, represents the bias term of the first linear transformation layer, represents the bias term of the second linear transformation layer;
[0092] Step S523: Model construction, specifically, establishing the universal transformer model by constructing the multi-head attention module and the feedforward neural network module to obtain the universal transformer model;
[0093] Step S53: training the federated learning framework model, specifically using a local training set to perform model training on a local model to obtain the local parameters, and sending the local parameters to the main model for local parameter aggregation and global model parameter update to obtain the DDoS attack detection model;
[0094] The steps of training the federated learning framework model include:
[0095] Step S531: construct a minimization objective function, the formula used is as follows:
[0096] ;
[0097] In the formula, represents the global model parameters, represents the local parameters of the kth local model, is the Lagrange multiplier, used as the weighting parameter of the constraint condition, Represents the global model parameters and the local parameters of the kth local model The Euclidean distance between represents the objective function of the local model, Represents the objective function value of the local model under the global model parameters;
[0098] Step S532: construct a gradient specification constraint condition to control the amplitude of the model update. The formula used is as follows:
[0099] ;
[0100] In the formula, Represents the global model parameters and the local parameters of the kth local model The gradient norm between represents the local parameters of other local models, Represents local parameters of other local models and global model parameters The gradient norm between Represents the weight of the gradient norm, which is used to constrain the local parameters of other local models and global model parameters The ratio of the gradient norm between ;
[0101] Step S533: construct constraint conditions to balance the consistency of local parameters and global model parameters. The formula used is as follows:
[0102] ;
[0103] In the formula, Indicates that training is performed while satisfying the constraints. Represents the global model parameters and the local parameters of the kth local model The gradient between represents the local parameters of the kth local model The gradient of
[0104] Step S534: Update the global model parameters. The formula used is as follows:
[0105] ;
[0106] In the formula, represents the global model parameters at the e-th communication round, represents the local parameters of the kth local model in the eth communication round, and M represents the total number of local models;
[0107] Step S535: Perform model training, specifically, by constructing the minimization objective function, constructing the gradient specification constraint condition, constructing the constraint condition and updating the global model parameters, and performing federated learning framework model training to obtain a DDoS attack detection model. And the performance is verified using the network intrusion detection test set;
[0108] In step S54, the network intrusion detection is performed, specifically using the DDoS attack detection model Perform DDoS attack detection to obtain the DDoS attack judgment data, and determine whether a DDoS attack occurs based on the DDoS attack judgment data.
[0109] The present invention provides a network intrusion detection system based on big data, comprising a data acquisition module, a data preprocessing module, a feature selection module, a data sample synthesis module and a network intrusion detection module;
[0110] The data acquisition module is used for data acquisition, and obtains a network intrusion detection original data set through data acquisition, and sends the network intrusion detection original data set to the data preprocessing module;
[0111] The data preprocessing module is used for data preprocessing, and obtains a preliminary data set for network intrusion detection through data preprocessing, and sends the preliminary data set for network intrusion detection to the feature selection module;
[0112] The feature selection module is used for feature selection, obtains a feature optimized data set through feature selection, and sends the feature optimized data set to the data sample synthesis module;
[0113] The data sample synthesis module is used to synthesize data samples, obtain a network intrusion detection data set by synthesizing data samples, and send the network intrusion detection data set to the network intrusion detection module;
[0114] The network intrusion detection module is used to detect DDoS attacks and obtain DDoS attack judgment data by detecting DDoS attacks.
[0115] The beneficial effects achieved by the present invention using the above scheme are as follows:
[0116] (1) The traditional method of synthesizing minority data samples used in network intrusion detection lacks sufficient authenticity and diversity, cannot accurately reflect the intrusion situation in the real world, and requires manual intervention and experience tuning, which makes it difficult to adapt to the changes in different network traffic data distribution and intrusion behavior. This solution creatively uses a deep convolutional generative adversarial network to synthesize minority data samples. By autonomously learning the characteristics and patterns of data, it can better capture complex intrusion features and generate more distinguishable synthetic samples, accurately reflecting the network intrusion situation in the real world. At the same time, it has strong generalization ability and can adapt to changes in different data distributions and intrusion behaviors.
[0117] (2) In view of the technical problem that when traditional network intrusion detection performs feature selection, the global search and the ability to find the optimal feature set are limited when processing large-scale intrusion features, and it is easy to fall into the local optimal solution, resulting in the failure to find the most discriminative feature combination, this solution creatively adopts the improved teaching-learning optimization algorithm combined with the gray wolf algorithm for feature selection. When processing and analyzing a large number of intrusion features, it has better global search capabilities and faster convergence speed, can effectively overcome the local optimal solution problem, find the optimal feature set, and improve the accuracy and performance of network intrusion detection;
[0118] (3) In view of the fact that traditional network intrusion detection methods are inefficient in processing and analyzing large amounts of network traffic data, resulting in insufficient real-time performance and inability to respond to intrusions in a timely manner, and that traditional methods have slow model update speeds and poor generalization capabilities, making it difficult to keep up with the ever-evolving changes in network attack methods, this solution creatively adopts a federated learning framework model for network intrusion detection and uses distributed computing, allowing each local model to calculate in parallel, thereby improving data processing efficiency and enabling timely responses to intrusions. At the same time, it uses an incremental learning method, which can quickly update the model based on new data and adapt to new threats. BRIEF DESCRIPTION OF THE DRAWINGS
[0119] Figure 1 A schematic diagram of a flow chart of a network intrusion detection method based on big data provided by the present invention;
[0120] Figure 2 A schematic diagram of a network intrusion detection system based on big data provided by the present invention;
[0121] Figure 3 A schematic diagram of the process of feature selection in step S3;
[0122] Figure 4 A schematic diagram of the process of synthesizing data samples in step S4;
[0123] Figure 5 This is a schematic diagram of the process of network intrusion detection in step S5;
[0124] Figure 6 Schematic diagram of the federated learning framework model in step S5.
[0125] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION
[0126] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0127] In the description of the present invention, it should be understood that terms such as “upper”, “lower”, “front”, “back”, “left”, “right”, “top”, “bottom”, “inside” and “outside” indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as limiting the present invention.
[0128] Example 1, see Figure 1 The present invention provides a network intrusion detection method based on big data, which comprises the following steps:
[0129] Step S1: data collection;
[0130] Step S2: data preprocessing;
[0131] Step S3: feature selection;
[0132] Step S4: data sample synthesis;
[0133] Step S5: Network intrusion detection.
[0134] Example 2, see Figure 1 and Figure 2 In step S1, the data collection is used to collect the original data required for network intrusion detection, specifically, to obtain the original data set for network intrusion detection from the security information and event management system;
[0135] The network intrusion detection original data set specifically includes non-DDoS attack malicious traffic data and DDoS attack malicious traffic data.
[0136] Example 3, see Figure 1 and Figure 2 This embodiment is based on the above embodiment. In step S2, the data preprocessing is used to preprocess the original data, and specifically includes the following steps:
[0137] Step S21: feature encoding, which is used to convert the original data into digital form, specifically encoding the network intrusion detection original data set by a one-hot encoding method to obtain a network intrusion detection encoded data set;
[0138] Step S22: data normalization, which is used to normalize the data, specifically, normalizing the network intrusion detection coded data set by a minimum-maximum method to obtain a preliminary network intrusion detection data set;
[0139] Step S23: performing data preprocessing, specifically performing data preprocessing on the original data set of network intrusion detection through the feature encoding and the data normalization to obtain a preliminary data set of network intrusion detection.
[0140] Example 4, see Figure 1 , Figure 2 and Figure 3 This embodiment is based on the above embodiment. In step S3, the feature selection is used to select large-scale intrusion features. Specifically, the improved teaching-learning optimization algorithm combined with the gray wolf algorithm is used to select features from the preliminary data set of network intrusion detection to obtain a feature optimized data set. Specifically, the following steps are included:
[0141] Step S31: executing the teaching strategy, specifically constructing a teacher unit and a student unit group, and optimizing the parameters of the student unit group by constructing the teacher unit;
[0142] The teacher unit is used to represent an optimal feature set, and the student unit group is used to represent a non-optimal feature set;
[0143] The formula used in the implementation teaching strategy is as follows:
[0144] ;
[0145] In the formula, represents the position calculation function after the iteration of the i-th student unit in the teaching stage, t represents the number of iterations, represents a random vector with a modulus value in [0,1], represents the position calculation function after the teacher iteration in the teaching phase;
[0146] Step S32: Execute the interactive learning strategy combined with the gray wolf algorithm, specifically construct a leader unit group, and optimize the parameters of the student unit group through the leader unit group. The leader unit group specifically includes a first leader , Second Leader and the third leader , the first leader , used to represent the best student unit in the student unit group, the second leader Used to indicate the second best student unit in the student unit group, the third leader Used to represent the third best student unit in the student unit group;
[0147] The interactive learning strategy combined with the grey wolf algorithm comprises the following steps:
[0148] Step S321: construct a linear reduction vector, the formula used is as follows:
[0149] ;
[0150] In the formula, is a linear reduction vector that decreases linearly from 2 to 0 during the iteration process, and MaxT represents the maximum number of iterations;
[0151] Step S322: Calculating the distance between the student units in the student unit group and the leaders in the leader unit group, the steps include:
[0152] Step S3221: Calculate the relationship between the i-th student unit and the first leader The distance between them is calculated using the following formula:
[0153] ;
[0154] In the formula, Indicates the relationship between the i-th student unit and the first leader The distance between represents a random vector with a modulus value in [0,1], Indicates the first leader location, represents the position calculation function after iteration of the i-th student unit in the interactive learning phase;
[0155] Step S3222: Calculate the relationship between the i-th student unit and the second leader The distance between them is calculated using the following formula:
[0156] ;
[0157] In the formula, Indicates the relationship between the i-th student unit and the second leader The distance between represents a random vector with a modulus value in [0,1], Indicates the second leader location;
[0158] Step S3223: Calculate the relationship between the i-th student unit and the third leader The distance between them is calculated using the following formula:
[0159] ;
[0160] In the formula, Indicates the relationship between the i-th student unit and the third leader The distance between represents a random vector with a modulus value in [0,1], Indicates the third leader location;
[0161] Step S323: Calculate the position of the student unit after the t+1th iteration using the following formula:
[0162] ;
[0163] In the formula, represents a random vector with a modulus value in [0,1], represents a random vector with a modulus value in [0,1], Represents a random vector with a modulus value in [0,1];
[0164] Step S33: executing an autonomous learning strategy to optimize the parameters of the student unit group, specifically calculating the mean absolute difference of the student unit as the fitness of the student unit, assigning a ranking to the student unit based on the fitness, and calculating the mutation probability of the student unit based on the ranking to optimize the parameters of the student unit;
[0165] The steps of executing the autonomous learning strategy include:
[0166] Step S331: Calculate the mean of the characteristics of the student unit. The formula used is as follows:
[0167] ;
[0168] In the formula, represents the mean of the features of the i-th student unit, J represents the number of features of the i-th student unit, represents the jth feature of the i-th student unit;
[0169] Step S332: Calculate the average absolute difference of student units using the following formula:
[0170] ;
[0171] In the formula, represents the mean absolute difference of the i-th student unit;
[0172] Step S333: assigning rankings, specifically using the average absolute difference of the student unit as the fitness of the student unit, and ranking the student units in descending order based on the fitness to obtain a student unit ranking list. The formula used is as follows:
[0173] ;
[0174] In the formula, represents the ranked list of student units, represents the worst student unit, represents the second worst student unit, represents the optimal student unit, num represents the total number of student units;
[0175] Step S334: Calculate the mutation probability, using the following formula:
[0176] ;
[0177] In the formula, r represents the ranking of the student unit, represents the mutation probability of the student unit ranked r, represents the maximum mutation probability;
[0178] Step S34: performing feature selection, specifically performing feature selection by executing the teaching strategy, executing the interactive learning strategy combined with the Grey Wolf Algorithm, and executing the autonomous learning strategy to obtain a feature optimized data set.
[0179] By performing the above operations, in view of the technical problem that when traditional network intrusion detection performs feature selection, when processing large-scale intrusion features, the global search and the ability to find the optimal feature set are limited, and it is easy to fall into the local optimal solution, resulting in the failure to find the most discriminative feature combination, this scheme creatively adopts the improved teaching-learning optimization algorithm combined with the gray wolf algorithm for feature selection. When processing and analyzing a large number of intrusion features, it has better global search capabilities and faster convergence speed, can effectively overcome the local optimal solution problem, find the optimal feature subset, and improve the accuracy and performance of network intrusion detection.
[0180] Example 5, see Figure 1 , Figure 2 and Figure 4 , This embodiment is based on the above embodiment. In step S4, the data sample synthesis is used to deal with the problem of data imbalance, specifically using a deep convolutional generative adversarial network to generate minority class data samples in a feature optimization data set to obtain a network intrusion detection data set;
[0181] The deep convolutional generative adversarial network is used to generate minority class data samples, specifically including a convolution generator G and a convolution discriminator D. The convolution generator G and the convolution discriminator D are continuously optimized in adversarial training to obtain realistic minority class data samples;
[0182] The data sample synthesis specifically includes the following steps:
[0183] Step S41: feature optimization data set division, which is used to perform data division, specifically, to obtain a majority class data set and a minority class data set by dividing the feature optimization data set;
[0184] Step S42: construct the objective function, the formula used is as follows:
[0185] ;
[0186] In the formula, ~ indicates that the variable follows a probability distribution, is the objective function, represents the data distribution of the minority class dataset, represents the data distribution of the majority class data set, and x represents the data distribution from The actual sample, z represents the data distribution The actual sample, represents the sample generated by the convolution generator G based on the actual sample z, It represents the output probability that the convolution discriminator D correctly classifies the actual sample x as the actual sample, Represents samples that are misclassified by the convolutional discriminator D is the output probability of the actual sample, Represents that for data from the distribution The actual sample x, the convolution discriminator D correctly classifies the actual sample x as the expected value of the actual sample, Represents the sample generated based on the actual sample z , the convolutional discriminator D correctly classifies is the expected value of the non-actual sample;
[0187] Step S43: Construct the loss function of the convolution generator G. The formula used is as follows:
[0188] ;
[0189] In the formula, The character representation of the loss function of the convolutional generator G, N represents the number of training samples, represents the i-th sample generated by the convolution generator G, Represents the i-th sample that the convolution discriminator D will generate The output probability of being misclassified as a real sample;
[0190] Step S44: Construct the loss function of the convolution discriminator D. The formula used is as follows:
[0191] ;
[0192] In the formula, Character representation of the loss function of the convolutional discriminator D, It means that the convolution discriminator D takes the ith actual sample The output probability of correctly classifying as actual samples;
[0193] Step S45: Model training, specifically, performing the deep convolutional generative adversarial network model training through the feature optimization data set partitioning, the construction objective function, the loss function of the convolution generator G, and the loss function of the convolution discriminator D to obtain a realistic minority class data sample generation model ;
[0194] Step S46: constructing a network intrusion detection data set, specifically generating a model through the simulated minority class data samples Generate realistic minority class data samples and obtain a network intrusion detection dataset.
[0195] By performing the above operations, the minority data sample synthesis method used in traditional network intrusion detection lacks sufficient authenticity and diversity, cannot accurately reflect the intrusion situation in the real world, and requires manual intervention and experience tuning, and is difficult to adapt to the changes in different network traffic data distributions and intrusion behaviors. This solution creatively uses a deep convolutional generative adversarial network to synthesize minority data samples. By autonomously learning the characteristics and patterns of the data, it can better capture complex intrusion features and generate more discriminative synthetic samples, accurately reflecting the real-world network intrusion situation. At the same time, it has strong generalization ability and can adapt to changes in different data distributions and intrusion behaviors.
[0196] Example 6, see Figure 1 , Figure 2 , Figure 5 and Figure 6 This embodiment is based on the above embodiment. In step S5, the network intrusion detection is used to detect whether a DDoS attack occurs. Specifically, a federated learning framework model is constructed and the model is trained. The DDoS attack detection model obtained after training is used to perform network intrusion detection to obtain DDoS attack judgment data.
[0197] The federated learning framework model specifically includes a general transformer model, a main model and a local model. Local parameters are obtained by training the general transformer model on the local model, and the local parameters are sent to the main model for local parameter aggregation and global model parameter update to obtain the DDoS attack detection model.
[0198] The universal transformer model is used for model training, the main model is used for aggregating local parameters and updating global model parameters, the local model is used for training the universal transformer model in parallel and sending the local parameters to the main model, and the local parameters are used to represent the model parameters after the universal transformer model training is performed on the local model;
[0199] The network intrusion detection specifically includes the following steps:
[0200] Step S51: data partitioning, which is used to partition the network intrusion detection data set, specifically, partitioning the network intrusion detection data set to obtain a local training set and a network intrusion detection test set;
[0201] Step S52: constructing a universal transformer model, wherein the universal transformer model specifically includes a multi-head attention module and a feedforward neural network module;
[0202] The steps of constructing a general transformer model include:
[0203] Step S5211: Calculate the dot product attention, the formula used is as follows:
[0204] ;
[0205] In the formula, Q represents the query vector, K represents the key vector, and V represents the value vector. represents the softmax function, represents the feature dimension, T represents the transposition operation, Represents the dot product attention calculation function;
[0206] Step S5212: Calculate the feature vector of the head using the following formula:
[0207] ;
[0208] In the formula, represents the feature vector of the i-th head, represents the weight matrix in the i-th head used to map the query vector to different subspaces for attention calculation, represents the weight matrix in the i-th head used to map the key vector to different subspaces for attention calculation, represents the weight matrix in the i-th head used to map the value vector to different subspaces for attention calculation;
[0209] Step S5213: Construct a multi-head attention function. The formula used is as follows:
[0210] ;
[0211] In the formula, represents the output function of the multi-head attention module, Concat represents the vector concatenation operation, represents the output weight matrix;
[0212] Step S522: constructing a feedforward neural network module, wherein the feedforward neural network module is specifically composed of two linear transformation layers using a ReLU activation function;
[0213] The feedforward neural network module uses the following formula:
[0214] ;
[0215] In the formula, x represents the input vector, represents the output function of the feedforward neural network module, represents the maximum value function, represents the weight matrix of the first linear transformation layer, represents the weight matrix of the second linear transformation layer, represents the bias term of the first linear transformation layer, represents the bias term of the second linear transformation layer;
[0216] Step S523: Model construction, specifically, establishing the universal transformer model by constructing the multi-head attention module and the feedforward neural network module to obtain the universal transformer model;
[0217] Step S53: training the federated learning framework model, specifically using a local training set to perform model training on a local model to obtain the local parameters, and sending the local parameters to the main model for local parameter aggregation and global model parameter update to obtain the DDoS attack detection model;
[0218] The steps of training the federated learning framework model include:
[0219] Step S531: construct a minimization objective function, the formula used is as follows:
[0220] ;
[0221] In the formula, represents the global model parameters, represents the local parameters of the kth local model, is the Lagrange multiplier, used as the weighting parameter of the constraint condition, Represents the global model parameters and the local parameters of the kth local model The Euclidean distance between represents the objective function of the local model, Represents the objective function value of the local model under the global model parameters;
[0222] Step S532: construct a gradient specification constraint condition to control the amplitude of the model update. The formula used is as follows:
[0223] ;
[0224] In the formula, Represents the global model parameters and the local parameters of the kth local model The gradient norm between represents the local parameters of other local models, Represents local parameters of other local models and global model parameters The gradient norm between Represents the weight of the gradient norm, which is used to constrain the local parameters of other local models and global model parameters The ratio of the gradient norm between ;
[0225] Step S533: construct constraint conditions to balance the consistency of local parameters and global model parameters. The formula used is as follows:
[0226] ;
[0227] In the formula, Indicates that training is performed while satisfying the constraints. Represents the global model parameters and the local parameters of the kth local model The gradient between represents the local parameters of the kth local model The gradient of
[0228] Step S534: Update the global model parameters. The formula used is as follows:
[0229] ;
[0230] In the formula, represents the global model parameters at the e-th communication round, represents the local parameters of the kth local model in the eth communication round, and M represents the total number of local models;
[0231] Step S535: Perform model training, specifically, by constructing the minimization objective function, constructing the gradient specification constraint condition, constructing the constraint condition and updating the global model parameters, and performing federated learning framework model training to obtain a DDoS attack detection model. And the performance is verified using the network intrusion detection test set;
[0232] In step S54, the network intrusion detection is performed, specifically using the DDoS attack detection model Perform DDoS attack detection to obtain the DDoS attack judgment data, and determine whether a DDoS attack occurs based on the DDoS attack judgment data.
[0233] By performing the above operations, the traditional network intrusion detection method has the problem of low efficiency in processing and analyzing large amounts of network traffic data, resulting in insufficient real-time performance and inability to respond to intrusion behaviors in a timely manner. At the same time, the traditional method has slow model update speed and poor generalization ability, and it is difficult to keep up with the ever-evolving changes in network attack methods. This solution creatively adopts a federated learning framework model for network intrusion detection and uses distributed computing, allowing each local model to calculate in parallel, thereby improving the efficiency of data processing and being able to respond to intrusion behaviors in a timely manner. At the same time, it uses an incremental learning method, which can quickly update the model according to new data and adapt to new threats.
[0234] Embodiment 7, see Figure 1 and Figure 2 , this embodiment is based on the above embodiment, and the present invention provides a network intrusion detection system based on big data, including a data acquisition module, a data preprocessing module, a feature selection module, a data sample synthesis module and a network intrusion detection module;
[0235] The data acquisition module is used for data acquisition, and obtains a network intrusion detection original data set through data acquisition, and sends the network intrusion detection original data set to the data preprocessing module;
[0236] The data preprocessing module is used for data preprocessing, and obtains a preliminary data set for network intrusion detection through data preprocessing, and sends the preliminary data set for network intrusion detection to the feature selection module;
[0237] The feature selection module is used for feature selection, obtains a feature optimized data set through feature selection, and sends the feature optimized data set to the data sample synthesis module;
[0238] The data sample synthesis module is used to synthesize data samples, obtain a network intrusion detection data set by synthesizing data samples, and send the network intrusion detection data set to the network intrusion detection module;
[0239] The network intrusion detection module is used to detect DDoS attacks and obtain DDoS attack judgment data by detecting DDoS attacks.
[0240] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0241] While the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that many changes, modifications, substitutions and variations can be made to the embodiments without departing from the principles and spirit of the invention.
[0242] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.
Claims
1. A network intrusion detection method based on big data, characterized in that: The method comprises the following steps: Step S1: data collection; Step S2: data preprocessing; Step S3: feature selection; Step S4: data sample synthesis; Step S5: network intrusion detection; In step S1, the data collection is used to collect the original data required for network intrusion detection, specifically, to obtain the original data set for network intrusion detection from the security information and event management system through collection; the original data set for network intrusion detection specifically includes non-DDoS attack malicious traffic data and DDoS attack malicious traffic data; In step S2, the data preprocessing is used to perform data preprocessing on the original data, specifically, to perform data preprocessing on the original data set of network intrusion detection to obtain a preliminary data set of network intrusion detection; In step S3, the feature selection specifically includes the following steps: Step S31: executing the teaching strategy, specifically constructing a teacher unit and a student unit group, and optimizing the parameters of the student unit group by constructing the teacher unit; The teacher unit is used to represent an optimal feature set, and the student unit group is used to represent a non-optimal feature set; The formula used in the implementation teaching strategy is as follows: In the formula, represents the position calculation function after the iteration of the i-th student unit in the teaching stage, t represents the number of iterations, represents a random vector with a modulus value in [0,1], represents the position calculation function after the teacher iteration in the teaching phase; Step S32: Execute an interactive learning strategy combined with the gray wolf algorithm, specifically construct a leader unit group, and optimize the parameters of the student unit group through the leader unit group, the leader unit group specifically includes a first leader α, a second leader β and a third leader δ, the first leader α is used to represent the best student unit in the student unit group, the second leader β is used to represent the second best student unit in the student unit group, and the third leader δ is used to represent the third best student unit in the student unit group; The interactive learning strategy combined with the grey wolf algorithm comprises the following steps: Step S321: construct a linear reduction vector, the formula used is as follows: In the formula, is a linear reduction vector that decreases linearly from 2 to 0 during the iteration process, and MaxT represents the maximum number of iterations; Step S322: Calculating the distance between the student units in the student unit group and the leaders in the leader unit group, the steps include: Step S3221: Calculate the distance between the i-th student unit and the first leader α, using the following formula: In the formula, represents the distance between the i-th student unit and the first leader α, represents a random vector with a modulus value in [0,1], represents the position of the first leader α, represents the position calculation function after iteration of the i-th student unit in the interactive learning phase; Step S3222: Calculate the distance between the i-th student unit and the second leader β, using the following formula: In the formula, represents the distance between the i-th student unit and the second leader β, represents a random vector with a modulus value in [0,1], represents the position of the second leader β; Step S3223: Calculate the distance between the i-th student unit and the third leader δ, using the following formula: In the formula, represents the distance between the i-th student unit and the third leader δ, represents a random vector with a modulus value in [0,1], represents the position of the third leader δ; Step S323: Calculate the position of the student unit after the t+1th iteration using the following formula: In the formula, represents a random vector with a modulus value in [0,1], represents a random vector with a modulus value in [0,1], Represents a random vector with a modulus value in [0,1]; Step S33: executing an autonomous learning strategy to optimize the parameters of the student unit group, specifically calculating the mean absolute difference of the student unit as the fitness of the student unit, assigning a ranking to the student unit based on the fitness, and calculating the mutation probability of the student unit based on the ranking to optimize the parameters of the student unit; The steps of executing the autonomous learning strategy include: Step S331: Calculate the mean of the characteristics of the student unit. The formula used is as follows: In the formula, represents the mean of the features of the i-th student unit, J represents the number of features of the i-th student unit, and y ij represents the jth feature of the i-th student unit; Step S332: Calculate the average absolute difference of student units using the following formula: In the formula, MAD i represents the mean absolute difference of the i-th student unit; Step S333: assigning rankings, specifically using the average absolute difference of the student unit as the fitness of the student unit, and ranking the student units in descending order based on the fitness to obtain a student unit ranking list. The formula used is as follows: list r =[s num ,S num-1 ,...,s1]; In the formula, list r represents the student unit ranking list, s num Represents the worst student unit, s num-1 represents the second worst student unit, s1 represents the best student unit, and num represents the total number of student units; Step S334: Calculate the mutation probability, using the following formula: In the formula, r represents the ranking of the student unit, p rank represents the mutation probability of the student unit ranked r, p max represents the maximum mutation probability; Step S34: performing feature selection, specifically performing feature selection by executing the teaching strategy, executing the interactive learning strategy combined with the Grey Wolf Algorithm, and executing the autonomous learning strategy to obtain a feature optimized data set; In step S4, the data sample synthesis is used to deal with the problem of data imbalance, specifically using a deep convolutional generative adversarial network to generate minority class data samples in the feature optimization data set to obtain a network intrusion detection data set; In step S5, the network intrusion detection is used to detect whether a DDoS attack occurs. Specifically, a federated learning framework model is constructed and model training is performed, and the DDoS attack detection model obtained after training is used to perform network intrusion detection to obtain DDoS attack judgment data.
2. According to the big data based network intrusion detection method of claim 1, it is characterized by: In step S2, the data preprocessing specifically includes the following steps: Step S21: feature encoding, which is used to convert the original data into digital form, specifically encoding the network intrusion detection original data set by a one-hot encoding method to obtain a network intrusion detection encoded data set; Step S22: data normalization, which is used to normalize the data, specifically, normalizing the network intrusion detection coded data set by a minimum-maximum method to obtain a preliminary network intrusion detection data set; Step S23: performing data preprocessing, specifically performing data preprocessing on the original data set of network intrusion detection through the feature encoding and the data normalization to obtain a preliminary data set of network intrusion detection.
3. A network intrusion detection method based on big data according to claim 2, characterized in that: In step S4, the data sample synthesis specifically includes the following steps: Step S41: feature optimization data set division, which is used to perform data division, specifically, to obtain a majority class data set and a minority class data set by dividing the feature optimization data set; Step S42: construct the objective function, the formula used is as follows: In the formula, ~ indicates that the variable follows the probability distribution, V(D, G) is the objective function, and p datax represents the data distribution of the minority class dataset, p z represents the data distribution of the majority class data set, and x represents the data distribution p datax The actual sample, z represents the sample from the data distribution p z The actual sample, G z Denotes the sample generated by the convolution generator G based on the actual sample z, D x It represents the output probability that the convolution discriminator D correctly classifies the actual sample x as the actual sample, Denotes the sample G generated by the convolution discriminator D misclassification z is the output probability of the actual sample, For data from the distribution p datax The actual sample x, the convolution discriminator D correctly classifies the actual sample x as the expected value of the actual sample, Represents the sample G generated based on the actual sample z z , the convolutional discriminator D correctly classifies G z is the expected value of the non-actual sample; Step S43: Construct the loss function of the convolution generator G. The formula used is as follows: Where, Ls G Character representation of the loss function of the convolutional generator G, where N represents the number of training samples and G zi represents the i-th sample generated by the convolution generator G, Represents the i-th sample G that the convolution discriminator D will generate zi The output probability of being misclassified as a real sample; Step S44: Construct the loss function of the convolution discriminator D. The formula used is as follows: Where, Ls D Character representation of the loss function of the convolutional discriminator D, D xi represents the output probability that the convolution discriminator D correctly classifies the i-th actual sample xi as an actual sample; Step S45: Model training, specifically, performing the deep convolutional generative adversarial network model training through the feature optimization data set partitioning, the construction of the objective function, the loss function of the convolution generator G, and the loss function of the convolution discriminator D, to obtain the simulated minority class data sample generation model Model G ; Step S46: construct a network intrusion detection data set, specifically generating a model Model through the simulated minority class data samples G Generate realistic minority class data samples and obtain a network intrusion detection dataset.
4. A network intrusion detection method based on big data according to claim 3, characterized in that: In step S5, the network intrusion detection includes the following steps: Step S51: data division; Step S52: constructing a general transformer model; Step S53: Federated learning framework model training; Step S54: Perform network intrusion detection.
5. A network intrusion detection method based on big data according to claim 4, characterized in that: In step S51, the data partitioning is used to partition the network intrusion detection data set, specifically, by partitioning the network intrusion detection data set to obtain a local training set and a network intrusion detection test set; In step S52, the general transformer model is constructed, and the steps include: Step S521: construct a multi-head attention module, the steps include: Step S5211: Calculate the dot product attention, the formula used is as follows: In the formula, Q represents the query vector, K represents the key vector, V represents the value vector, softmax(·) represents the softmax function, and d k represents the feature dimension, T represents the transposition operation, A att (·) represents the dot product attention calculation function; Step S5212: Calculate the feature vector of the head using the following formula: In the formula, h i represents the feature vector of the i-th head, represents the weight matrix in the i-th head used to map the query vector to different subspaces for attention calculation, represents the weight matrix in the i-th head used to map the key vector to different subspaces for attention calculation, represents the weight matrix in the i-th head used to map the value vector to different subspaces for attention calculation; Step S5213: Construct a multi-head attention function. The formula used is as follows: M MHA (Q,K,V)=Concat(h1,h2,...,h m )ω o ; Where M MHA (·) represents the output function of the multi-head attention module, Concat represents the vector concatenation operation, ω o represents the output weight matrix; Step S522: constructing a feedforward neural network module, wherein the feedforward neural network module is specifically composed of two linear transformation layers using a ReLU activation function; The feedforward neural network module uses the following formula: F FFN (x)=max(0,xω1+a1)ω2+a2; In the formula, x represents the input vector, F FFN (·) represents the output function of the feedforward neural network module, max(·) represents the maximum value function, ω1 represents the weight matrix of the first linear transformation layer, ω2 represents the weight matrix of the second linear transformation layer, a1 represents the bias term of the first linear transformation layer, and a2 represents the bias term of the second linear transformation layer; Step S523: Model construction, specifically, establishing the universal transformer model by constructing the multi-head attention module and the feedforward neural network module to obtain the universal transformer model; In step S53, the federated learning framework model training is specifically to use the local training set to perform model training on the local model to obtain local parameters, and send them to the main model for local parameter aggregation and global model parameter update to obtain the DDoS attack detection model; The steps of training the federated learning framework model include: Step S531: construct a minimization objective function, the formula used is as follows: In the formula, θ represents the global model parameter, τ k represents the local parameters of the kth local model, σ is the Lagrange multiplier, which is used as the weighting parameter of the constraint condition, and ||τ k -θ|| represents the global model parameter θ and the local parameter τ of the kth local model k The Euclidean distance between represents the objective function of the local model, φ k (τ k ; θ) represents the objective function value of the local model under the global model parameters; Step S532: construct a gradient specification constraint condition to control the amplitude of the model update. The formula used is as follows: In the formula, represents the global model parameters θ and the local parameters τ of the kth local model k The gradient norm between el represents the local parameters of other local models, represents the local parameters τ of other local models el and the gradient norm between the global model parameters θ, represents the weight of the gradient norm, which is used to constrain the local parameters τ of other local models el The ratio of the gradient norm between θ and the global model parameter θ; Step S533: construct constraint conditions to balance the consistency of local parameters and global model parameters. The formula used is as follows: In the formula, st means training under the condition of satisfying the constraints. represents the global model parameters θ and the local parameters τ of the kth local model k The gradient between represents the local parameter τ of the kth local model k The gradient of Step S534: Update the global model parameters. The formula used is as follows: In the formula, θ e represents the global model parameters at the e-th communication round, represents the local parameters of the kth local model in the eth communication round, and M represents the total number of local models; Step S535: Perform model training, specifically, perform federated learning framework model training by constructing the minimization objective function, constructing the gradient specification constraint condition, constructing the constraint condition and updating the global model parameters to obtain the DDoS attack detection model Model F And the performance is verified using the network intrusion detection test set; In step S54, the network intrusion detection is performed, specifically using the DDoS attack detection model Model F Perform DDoS attack detection to obtain the DDoS attack judgment data, and determine whether a DDoS attack occurs based on the DDoS attack judgment data.
6. A network intrusion detection system based on big data, used to implement a network intrusion detection method based on big data as claimed in any one of claims 1 to 5, characterized in that: It includes a data acquisition module, a data preprocessing module, a feature selection module, a data sample synthesis module and a network intrusion detection module.
7. A network intrusion detection system based on big data according to claim 6, characterized in that: The data acquisition module is used for data acquisition, and obtains a network intrusion detection original data set through data acquisition, and sends the network intrusion detection original data set to the data preprocessing module; The data preprocessing module is used for data preprocessing, and obtains a preliminary data set for network intrusion detection through data preprocessing, and sends the preliminary data set for network intrusion detection to the feature selection module; The feature selection module is used for feature selection, obtains a feature optimized data set through feature selection, and sends the feature optimized data set to the data sample synthesis module; The data sample synthesis module is used to synthesize data samples, obtain a network intrusion detection data set by synthesizing data samples, and send the network intrusion detection data set to the network intrusion detection module; The network intrusion detection module is used to detect DDoS attacks and obtain DDoS attack judgment data by detecting DDoS attacks.
