Network security data analysis method based on big data
Data expansion and feature extraction through adversarial network algorithm and adaptive fractal exploration optimization algorithm, combined with the extreme learning machine algorithm of gradient enhancement learning, the instability problem of network security data analysis methods in the existing technology is solved, and more efficient Cybersecurity data classification and identification to respond quickly to cyber attacks.
Patent Information
- Application Number
- CN202410961511.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-12
AI Technical Summary
Existing network security data analysis methods are difficult to adapt to emerging threats and cannot fully cover data distribution. Gradient optimization algorithms are prone to local optimization. Feature extraction methods cannot effectively compress and retain important information, resulting in unstable model performance.
Using a network security data analysis method based on big data, data augmentation is performed through adversarial network algorithms, neural network feature extraction model using adaptive fractal exploration optimization algorithm, and data feature extraction and model training are performed in combination with the extreme learning machine algorithm of gradient enhancement learning. , ensure that the model focuses on difficult-to-classify samples during training.
It improves the accuracy of network security data analysis and generalization capabilities of models, improves the efficiency and accuracy of network security data classification and identification, realizes rapid identification and response to potential network attacks, and realizes automatic blocking or mitigation of attacks.
Smart Images

Figure CN118944919A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data technology, and more specifically, to a network security data analysis method based on big data. Background Art
[0002] As the complexity of networks and applications increases, network security vulnerabilities become more difficult to identify and manage. Existing network security data analysis methods have been unable to adapt to emerging threats or different types of network environments. Summary of the invention
[0003] According to the present invention, a network security data analysis method based on big data is provided to solve the problems that the data generation method in the prior art may not fully cover the potential data distribution and limit the generalization ability of the model; to solve the problem that the traditional gradient-based optimization algorithm is prone to fall into the local optimum and is insufficient to cope with complex parameter spaces; to solve the problem that the existing feature extraction method may not be able to effectively compress and retain the most important information in the data, affecting the classification performance; to solve the problem that the existing classifier may not be able to effectively identify and classify samples that are difficult to distinguish, resulting in unstable model performance.
[0004] According to the present invention, a network security data analysis method based on big data is provided, comprising:
[0005] Collect characteristic data from network traffic logs, system access records, abnormal login attempts, and various network attacks, and annotate the data;
[0006] Using an adversarial network algorithm to perform data expansion on the original data in the original data set to obtain preprocessed data;
[0007] The three-layer fully connected neural network is used to extract data features, and the neural network feature extraction model of the adaptive fractal exploration optimization algorithm is used to achieve efficient optimization of the neural network weights and biases, explore a wider parameter space, and effectively improve the parameter optimization efficiency of the neural network;
[0008] The feature-extracted data is input into the feature dimensionality reduction model for training, ensuring that the encoded feature vector can more accurately capture the key information of the input data;
[0009] The reduced-dimensional data is input into the classifier for classifier training. The extreme learning machine algorithm based on gradient enhancement learning uses the gradient enhancement mechanism to adjust the weights of the hidden layer, so that the model pays more attention to samples that are difficult to classify during the training process.
[0010] Use the trained model to process new samples, classify the new samples, and then achieve fast and accurate network security analysis, quickly identify and respond to potential network attacks, and achieve automatic blocking or mitigation of attacks.
[0011] Optionally, collect characteristic data from network traffic logs, system access records, abnormal login attempts, and various network attacks, and annotate the data; including:
[0012] Collect multiple raw data from the network, and the data storage format adopts SQL structured database format, including multiple attribute fields and multiple data attributes;
[0013] The data set is determined based on the original data, and 10 attributes in the network traffic data packet are extracted. The data attributes are:
[0014] a x is the source IP address, indicating the source of the data packet; a y is the target IP address, indicating the receiving destination of the data packet; a z is the transmission protocol type, such as TCP, UDP, etc.; a aa is the source port number, indicating the port from which the data packet is sent; a ab is the destination port number, indicating the port where the data packet is received; a ac is the packet size in bytes; a ad Session duration, which records the length of time from connection establishment to disconnection; a ae is the data packet transmission rate, in packets per second; a af A feature summary of the payload data, such as a hash value, used to quickly identify the data content; a ag A flag to mark whether the data packet contains known attack features.
[0015] In this raw data, the collected data is labeled, and each feature is assigned a label category, which includes three categories: "normal", "suspicious", and "malicious".
[0016] Optionally, the collected original data and the labeled training data are expanded, which is characterized in that the collected labeled data are expanded to obtain optimized generated data so that it is closer to the real data distribution, including:
[0017] The algorithm structure of the generative adversarial network based on quantum information quantum state reuse consists of two parts: the generator (G) and the discriminator (D). The generator is responsible for generating data, and the discriminator's task is to distinguish the generated data from the real data. The generator receives the noise signal encoded in the quantum state as input, and improves the diversity and quality of the data through quantum state reuse technology; the discriminator evaluates the authenticity of the data to optimize the performance of the generator.
[0018] Initialize the parameters of the generator G and the discriminator D to be b G0 , b D0 , the parameters are continuously updated during the training process to improve the model performance. The input noise signal is quantum-encoded, and the traditional noise signal is converted into a quantum state using the principle of quantum computing to increase the density and complexity of the information. The quantum-encoding process can be expressed as:
[0019] c encoded =Q(c input )
[0020] In the formula, c encoded represents the noise signal after quantum state encoding, Q represents the quantum state encoding operation, c input is the original noise signal;
[0021] Generator G receives the noise signal encoded in the quantum state and generates data through the internal network structure, simulating the high efficiency of quantum information processing to produce high-quality data samples.
[0022] The discriminator D evaluates the difference between generated data and real data, trying to distinguish whether the input data is generated by the generator G or comes from the real data set. Specifically, the authenticity evaluation of the generated data by the discriminator D can be expressed as:
[0023] D score =D(b D p decoded ||p real )
[0024] Where D score represents the score given by the discriminator D, b D is the parameter of the discriminator D, p decoded is the generated data after quantum state decoding, P real is the real data, and || indicates the merging of data.
[0025] Optionally, a three-layer fully connected neural network is used to extract data features, and a neural network feature extraction model of an adaptive fractal exploration optimization algorithm is used to achieve efficient optimization of neural network weights and biases, explore a wider parameter space, and effectively improve the parameter optimization efficiency of the neural network; including:
[0026] Step 1: Select the initial parameter set, randomly generate a set of parameters as the starting point according to the structure of the neural network, and set the initial granularity and search radius of the exploration. Specifically, set the initial parameter set to θ0, including the weight w and bias b of the neural network, and initialize the search granularity δ0 and search radius r0.
[0027] Step 2: In each iteration, a set of self-similar exploration points is generated according to the current parameter point to generate a fractal structure. The exploration points are distributed around the current point according to fractal rules to form a pattern similar to the fractal structure in nature. Specifically, from the current parameter point θ i , generate a set of self-similar exploration points {θ i,j}, where j indicates the jth exploration point, and sine and cosine functions are used to generate points distributed on the sphere of high-dimensional space to simulate the distribution of fractal structures in nature.
[0028] Step 3: Use the dynamic information entropy feedback mechanism to optimize the search process of neural network parameters. By calculating the information entropy of the current exploration point set, dynamically adjust the search strategy to respond to the complexity changes in the parameter space, dynamically adjust the search granularity and direction, and enhance the global search ability and adaptability of the algorithm, so as to explore the parameter space more effectively. Specifically, let the set of all exploration points in the current iteration be S = {θ i,j}, where i is the number of iterations, j is the index of the exploration point, and for each exploration point, its loss function value L(θ i,j ), and normalize the loss function values so that their sum is 1. The normalization process can be expressed as:
[0029]
[0030] In the formula, p(θ i,j ) represents the exploration point θ i,j The normalized weight of .
[0031] Step 4: Adaptively and dynamically adjust the search granularity according to the loss function value of the current exploration point. If the loss of multiple exploration points is low, reduce the search granularity to refine the search; if the loss is high, increase the search granularity to speed up the search. Specifically, the adaptive dynamic adjustment of the search granularity can ensure that the search granularity is refined when a better solution is found, and increased when it is difficult to find a better solution to speed up the exploration. According to the loss function L at the exploration point {θ i,j}, dynamically adjust the search granularity δ i+1 can be expressed as:
[0032]
[0033] In the formula, α self is the reduction factor, β selfis the expansion factor, L(θ i ) is the parameter θ i The loss value at that time, Ω is an indicator based on the historical search efficiency.
[0034] Step 5: Calculate the loss function value of all exploration points and select the point with the smallest loss as the new current point, which can be expressed as:
[0035]
[0036] In the formula, θ i+1 is the optimal parameter exploration point of the current iteration, T(θ, Λ) is the transformation function, which improves the search efficiency and accuracy of the algorithm in different areas by dynamically adjusting the representation of the parameter space. By dynamically transforming the parameter space nonlinearly, the algorithm can more effectively adapt to the local characteristics of the loss function.
[0037] Step 6: According to the effect of the current iteration, dynamically adjust the search radius to optimize the breadth and depth of the search to more effectively explore the parameter space. Specifically, the search radius is dynamically adjusted according to the degree of improvement of the loss function, and the adjustment method can be expressed as:
[0038]
[0039] In the formula, ΔL=L(θ i )-L(θ i+1 ),∈ ad It is the set threshold used to judge whether the loss function has been significantly improved.
[0040] Optionally, the feature-extracted data is input into a feature dimensionality reduction model for training, using an improved autoencoder structure, which includes two main parts: an encoder and a decoder. The encoder's task is to compress the input data into a smaller representation, namely a feature vector, and the decoder attempts to reconstruct the original data from this feature vector. In view of the data feature information that may be ignored by traditional autoencoders, in the encoding stage, an information filtering layer is used to ensure that only the most useful information is passed to the feature vector, which helps to improve the quality and representativeness of the feature vector. In the decoding stage, the decoded output is compared with the original input and fed back to the encoder to further optimize the encoding process and ensure that the encoded feature vector can more accurately capture the key information of the input data.
[0041] Specifically, the training process of the improved autoencoder algorithm is as follows:
[0042] Step 1: Initialize the parameters of the encoder and decoder. Set the weight and bias of the encoder to and The decoder weights and biases are and The initialization process can be expressed as:
[0043]
[0044]
[0045] In the formula, Indicates that the mean is 0 and the variance is σ 2 The normal distribution of .
[0046] Step 2: Use the training data to learn the information filter layer and determine which information is most useful for the feature vector so that it can effectively select and transmit information. Specifically, the information filter layer aims to extract the information from the input data p input Extract the most useful information from the information filter layer, and set the output of the information filter layer to be p filtered , and its calculation method can be expressed as:
[0047]
[0048] In the formula, and They represent the weight and bias of the information filtering layer respectively, and Sig( ) is the Sigmoid activation function.
[0049] Step 3: Input the training data into the encoder and generate a compressed feature vector through the information filtering layer. Specifically, the encoding process converts the filtered data p filtered Compressed into feature vector p features , which can be expressed as:
[0050]
[0051] For the encoder weights The update is calculated by the gradient descent algorithm. Assuming that the goal of the encoder is to minimize the difference between the original data and the reconstructed data, its update method can be expressed as:
[0052]
[0053] In the formula, is the updated encoder weight, η as is the learning rate, is the loss function, is the loss function for gradient.
[0054] Step 4: Input the feature vector into the decoder, reconstruct the data, compare the reconstructed data with the original data, and adjust the parameters of the encoder and decoder through error feedback to reduce information loss and reconstruction error.
[0055] Step 5: Repeat the above steps until the preset stop iteration condition is met, indicating that the model training is completed.
[0056] Optionally, the reduced-dimensional data is fed into a classifier for classifier training, so that the model pays more attention to samples that are difficult to classify during training, including:
[0057] Initialize the weights and biases of the extreme learning machine model, including the weights from the input layer to the hidden layer and bias And the weights from the hidden layer to the output layer Specifically, the process of initializing parameters can be expressed as:
[0058]
[0059]
[0060]
[0061] In the formula, and They represent the weight from the input layer to the hidden layer, the bias of the hidden layer, and the weight from the hidden layer to the output layer during initialization. The above parameters are distributed through a normal distribution. Random initialization, where and is the corresponding variance, controlling the dispersion of the initialization.
[0062] The training data is input into the extreme learning machine model, and the activation value of the hidden layer and the final output value are calculated through forward propagation. Specifically, during the forward propagation process, the activation value of the hidden layer And the predicted value q of the output layer y The calculation method can be expressed as:
[0063]
[0064]
[0065] In the formula, q x is the input feature vector, and Sig() is the Sigmoid activation function.
[0066] In each iteration, for misclassified samples, the weights of the hidden layer are adjusted The model is made to enhance its learning ability for these samples, and interactive optimization is performed based on the performance of the model on the validation set. Specifically, if the classification accuracy of a certain category of samples is lower than expected, the weights from the hidden layer to the output layer are fine-tuned. To improve the recognition rate of a specific category, the adjustment of the output layer weight can be expressed as:
[0067]
[0068] In the formula, is the weight adjustment based on the feedback.
[0069] Furthermore, the weight adjustment amount Dynamic calculation is performed based on the performance feedback of the model. If the classification accuracy of a certain category is not high and the model's ability to distinguish this category needs to be enhanced, then, The calculation of is based on the feedback adjustment of category performance and can be expressed as:
[0070]
[0071] Where β is the coefficient for adjusting the intensity, and Represent the target performance and current performance respectively, q dir is the adjustment direction determined according to performance feedback. Preferably, β is set to 0.1.
[0072] Thus, comprehensive and targeted improvements have been made from data preprocessing to feature extraction, and then to model training and optimization. By introducing innovative data preprocessing algorithms, data expansion methods, and adopting a neural network feature extraction model based on an adaptive fractal exploration optimization algorithm, the present invention not only improves the accuracy of network security data analysis, but also enhances the generalization ability of the model. This largely solves a series of problems in the prior art, can greatly improve the efficiency and accuracy of network security data classification and identification, quickly identify and respond to potential network attacks, and achieve automated blocking or mitigation of attacks. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:
[0074] Figure 1 A schematic diagram of a process of a network security data analysis method based on big data described in this embodiment;
[0075] Figure 2 This is a flow chart of feature extraction model training described in this implementation mode; DETAILED DESCRIPTION
[0076] Now, exemplary embodiments of the present invention are described with reference to the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided to disclose the present invention in detail and completely and to fully convey the scope of the present invention to those skilled in the art. The terms used in the exemplary embodiments shown in the accompanying drawings are not intended to limit the present invention. In the accompanying drawings, the same units / elements are marked with the same reference numerals.
[0077] Unless otherwise specified, the terms (including technical terms) used herein have the commonly understood meanings to those skilled in the art. In addition, it is understood that the terms defined in commonly used dictionaries should be understood to have the same meanings as those in the context of the relevant fields, and should not be understood as idealized or overly formal meanings.
[0078] According to the present invention, a network security data analysis method based on big data is provided, referring to Figure 1 As shown, the method includes:
[0079] S1: Collect characteristic data from network traffic logs, system access records, abnormal login attempts, and various network attacks, and annotate the data;
[0080] S2: using an adversarial network algorithm to perform data expansion on the original data in the original data set to obtain preprocessed data;
[0081] S3: Utilize a three-layer fully connected neural network to extract data features and a neural network feature extraction model based on an adaptive fractal exploration optimization algorithm to achieve efficient optimization of neural network weights and biases, explore a wider parameter space, and effectively improve the parameter optimization efficiency of the neural network;
[0082] S4: Input the feature-extracted data into the feature dimensionality reduction model to train the feature dimensionality reduction model, ensuring that the encoded feature vector can more accurately capture the key information of the input data;
[0083] S5: The reduced-dimensional data is input into the classifier for classifier training. The extreme learning machine algorithm based on gradient enhancement learning adjusts the weights of the hidden layer by adopting the gradient enhancement mechanism, so that the model pays more attention to samples that are difficult to classify during the training process;
[0084] S6: Use the trained model to process new samples, classify the new samples, and then achieve fast and accurate network security analysis, quickly identify and respond to potential network attacks, and achieve automatic blocking or mitigation of attacks.
[0085] Specifically, step 1: data collection and annotation
[0086] The data source of the present invention mainly comes from network traffic logs, system access records, abnormal login attempts, and characteristic data of various network attacks. The data attributes mainly include:
[0087] a x : Source IP address, indicating the source of the data packet.
[0088] a y : Destination IP address, indicating the receiving destination of the data packet.
[0089] a z : Transmission protocol type, such as TCP, UDP, etc.
[0090] a aa : Source port number, indicating the port from which the data packet is sent.
[0091] a ab : Destination port number, indicating the port where the data packet is received.
[0092] a ac : Packet size in bytes.
[0093] a ad : Session duration, which records the length of time from connection establishment to disconnection.
[0094] a ae : Data packet transmission rate, in packets per second.
[0095] a af : A feature summary of the payload data, such as a hash value, used to quickly identify the data content.
[0096] a ag : Marks whether the data packet contains known attack features.
[0097] It should be noted that this embodiment is only used to illustrate one data format and type of the present invention. In actual applications, the attributes of data are usually more than 10 attributes, and the number of attributes of data may reach dozens or even hundreds.
[0098] The collected data is labeled. The labeling method of the present invention is manual labeling. In one embodiment, the labeled categories include three categories: "normal", "suspicious", and "malicious".
[0099] Step 2: Data expansion processing
[0100] In the task of identifying network security vulnerabilities, data collection and labeling are time-consuming and labor-intensive, and the insufficient amount of training sample data can easily lead to the difficulty of rapid convergence of artificial intelligence models. Therefore, it is necessary to expand the collected and labeled training data using a generative adversarial network algorithm based on quantum information quantum state reuse. The algorithm structure consists of two parts: the generator (G) and the discriminator (D). The generator is responsible for generating data, and the discriminator's task is to distinguish the generated data from the real data. The generator receives a noise signal encoded in a quantum state as input, and improves the diversity and quality of the data through quantum state reuse technology; the discriminator evaluates the authenticity of the data to optimize the performance of the generator.
[0101] Initialize the parameters of the generator G and the discriminator D to be b G0 , b D0 , the parameters are continuously updated during the training process to improve the model performance. The input noise signal is quantum-encoded, and the traditional noise signal is converted into a quantum state using the principle of quantum computing to increase the density and complexity of the information. Specifically, the quantum state encoding process can be expressed as:
[0102] First, initialize the parameters of the generator G and the discriminator D to b respectively. G0 , b D0 , the parameters are continuously updated during the training process to improve the model performance.
[0103] Furthermore, the input noise signal is quantum-encoded, and the traditional noise signal is converted into a quantum state using the principle of quantum computing to increase the density and complexity of information. Specifically, the quantum state encoding process can be expressed as:
[0104] c encoded =Q(c input ) where c encoded represents the noise signal after quantum state encoding, Q represents the quantum state encoding operation, c input is the original noise signal.
[0105] In one embodiment, the calculation method of the quantum state encoding operation Q can be expressed as:
[0106]
[0107] In the formula, α i represents the probability amplitude of the quantum state, |ψ i > represents the base state of the quantum state.
[0108] Furthermore, the generator G receives the noise signal encoded in the quantum state, generates data through the internal network structure, and simulates the efficiency of quantum information processing to produce high-quality data samples. Specifically, the way in which the generator G receives the noise signal encoded in the quantum state and generates data can be expressed as:
[0109] p generated =G(b G , c encoded )
[0110] In the formula, p generated is the data generated by generator G, b G is the parameter of the generator G, c encoded It is a noise signal encoded by quantum state.
[0111] Furthermore, the operation of the generator G can be expressed as:
[0112]
[0113] In the formula, w j represents the weight in the generator G, b j is the bias term, and Sig( ) is the Sigmoid activation function.
[0114] Furthermore, the generated data is quantum-decoded and converted back into traditional data form. The quantum-decoding process can be expressed as:
[0115] p decoded =Q -1 (P generated )
[0116] In the formula, p decoded Represents the data after quantum state decoding, Q -1 represents the quantum state decoding operation, p generated is the data produced by the generator.
[0117] Furthermore, the discriminator D evaluates the difference between the generated data and the real data, trying to distinguish whether the input data is generated by the generator G or comes from the real data set. Specifically, the authenticity evaluation of the generated data by the discriminator D can be expressed as:
[0118] D score =D(b D , p decoded ||p real )
[0119] Where D score represents the score given by the discriminator D, b D is the parameter of the discriminator D, p decoded is the generated data after quantum state decoding, p realis the real data, and || indicates the merging of data.
[0120] Furthermore, the operation of the discriminator D can be expressed as:
[0121]
[0122] In the formula, and are the weights and bias terms in the discriminator D, respectively, and Re() is the ReLU activation function.
[0123] Furthermore, for the generator G, the loss function aims to minimize the ability of the discriminator D to correctly identify the data it generates; for the discriminator D, its loss function aims to maximize its ability to distinguish between generated data and real data. The present invention uses quantum state information hiding loss to further guide the generator G to generate high-quality data close to the real distribution. Specifically, the loss function of the generator G is:
[0124] L G = -log(D(b D G(b G , c encoded )))
[0125] And, the loss function of the discriminator D is:
[0126] L D = -[log(D(b D , p real ))+log(1-D(b D , G(b G , c encoded )))]
[0127] Where, L G and L D are the loss functions of the generator and the discriminator, respectively, and b G and b D are the parameters of the generator and discriminator, c encoded is the noise signal encoded in the quantum state, p real is real data. Similarly, D(b D ,p real ) and 1-D(b D ,G(b G ,C encoded The calculation of )) is also based on the calculation method of the aforementioned discriminator D operation.
[0128] Furthermore, the back propagation algorithm is used to update the parameters of the generator G and the discriminator D according to the result of the loss function. Specifically, the parameter update uses the gradient descent method, which can be expressed as:
[0129]
[0130]
[0131] In the formula, and denote the updated parameters of the generator and discriminator, α G and α D is the learning rate, and is the loss function L D and L D About b G and b D Preferably, the learning rate α of the discriminator D is D is set to 0.001.
[0132] Further, the above steps are repeated until the preset stop iteration condition is met, indicating that the model training is completed. In one embodiment, the preset stop iteration condition is reaching a preset maximum number of iterations, preferably, the preset maximum number of iterations is set to 1000 times.
[0133] Step 3: Feature extraction model training
[0134] The present invention adopts a neural network feature extraction model based on an adaptive fractal exploration optimization algorithm, which is inspired by the self-similarity and ubiquitous characteristics of fractal structures in nature, as well as the complexity and self-organization in fractal theory. Fractal structures are widely present in nature, such as the shape of snowflakes, the outline of mountains, etc., which show similarities at different scales. The present invention introduces this self-similarity principle into the parameter optimization process of the neural network, explores the parameter space of the neural network by simulating the generation process of fractal structures in nature, and thus realizes efficient optimization of the weights and biases of the neural network.
[0135] The adaptive fractal exploration optimization algorithm proposed in the present invention does not rely on gradient information, but uses a self-similar exploration mode to search in the parameter space, so that the algorithm has a higher probability of jumping out of the local optimal solution and exploring a wider parameter space. In addition, the adaptive fractal exploration optimization algorithm adopts an adaptive mechanism to dynamically adjust the granularity and direction of the search according to the current search state, so as to adapt to different task scenarios more flexibly and effectively improve the parameter optimization efficiency of the neural network.
[0136] Specifically, the training process of the neural network feature extraction model based on the adaptive fractal exploration optimization algorithm is as follows: Figure 2 :
[0137] 1. Select an initial parameter set, randomly generate a set of parameters as the starting point according to the structure of the neural network, and set the initial granularity and search radius of the exploration. Specifically, set the initial parameter set to θ0, including the weight w and bias b of the neural network, and initialize the search granularity δ0 and search radius r0. Preferably, the parameter δ0 is set to 0.1, r0 is set to 1, and the parameter set θ0 is initialized by random initialization.
[0138] 2. In each iteration, a set of self-similar exploration points is generated according to the current parameter point to generate a fractal structure. The exploration points are distributed around the current point according to fractal rules to form a pattern similar to the fractal structure in nature. Specifically, from the current parameter point θ i , generate a set of self-similar exploration points {θ i,j}, where j indicates the jth exploration point, and sine and cosine functions are used to generate points distributed on the sphere of a high-dimensional space to simulate the distribution of fractal structures in nature. In one embodiment, the exploration points are distributed in the following form:
[0139] θ i,j =θ i +r i ·δ i ·cos(2πj / N)+r i ·δ i ·sin(2πj / N)
[0140] Where N is the number of exploration points generated in the current iteration, δ i is the current search granularity, r i is the current search radius.
[0141] Furthermore, the two items cos(2πj / N) and sin(2πj / N) actually define the position of a point on a unit circle in a high-dimensional parameter space. In order to make the adjustment to make the distribution of the exploration points richer and more flexible, and to make the exploration points not only distributed on a standard circular trajectory, but also adjusted based on the fractal dimension D, in another embodiment, the present invention uses the fractal dimension D to adjust the angle distribution, and the calculation method can be expressed as:
[0142]
[0143] Where D represents the fractal dimension, which affects the distribution density of the exploration points in space and how to adjust the distribution of these points according to fractal theory.
[0144] 3. Use dynamic information entropy feedback mechanism to optimize the search process of neural network parameters. By calculating the information entropy of the current exploration point set, dynamically adjust the search strategy to respond to the complexity changes of the parameter space, dynamically adjust the search granularity and direction, enhance the global search ability and adaptability of the algorithm, and thus explore the parameter space more effectively. Specifically, let the set of all exploration points in the current iteration be S = {θ i,j}, where i is the number of iterations and j is the index of the exploration point. Further, for each exploration point, calculate its loss function value L(θ i,j ), and normalize the loss function values so that their sum is 1. The normalization process can be expressed as:
[0145]
[0146] In the formula, p(θ i,j ) represents the exploration point θ i,j The normalized weight of .
[0147] Furthermore, the information entropy H(S) of the current exploration point set is calculated, which can be expressed as:
[0148]
[0149] Furthermore, f(H)S)) and g(H(S)) are defined as adjustment functions based on information entropy H(S), which are used to adjust the search granularity and search radius respectively. The calculation method can be expressed as:
[0150]
[0151]
[0152] Wherein, H0 is a preset information entropy threshold. Preferably, the information entropy threshold H0 is set to 0.8.
[0153] 4. Adaptively and dynamically adjust the search granularity according to the loss function value of the current exploration point. In one embodiment, if the loss of multiple exploration points is low, the search granularity is reduced to refine the search; if the loss is high, the search granularity is increased to speed up the search. Specifically, the adaptive and dynamic adjustment of the search granularity can ensure that the search granularity is refined when a better solution is found, and is increased when it is difficult to find a better solution to speed up the exploration. According to the loss function L at the exploration point {θ i,j}, dynamically adjust the search granularity δ i+1 can be expressed as:
[0154]
[0155] In the formula, α self is the reduction factor, β selfis the expansion factor, L(θ i ) is the parameter θ i The loss value at this time, Ω is an indicator based on the historical search efficiency. Preferably, the reduction factor α self is set to 0.9, the expansion factor β self is set to 1.1.
[0156] Furthermore, the calculation method of the index Ω based on historical search efficiency is as follows:
[0157]
[0158] Where k represents the number of historical iterations.
[0159] 5. Calculate the loss function value of all exploration points and select the point with the smallest loss as the new current point, which can be expressed as:
[0160]
[0161] In the formula, θ i+1 is the optimal parameter exploration point of the current iteration, and T(θ, Λ) is the transformation function.
[0162] Furthermore, the dynamic adjustment function λ(θ,Λ) improves the search efficiency and accuracy of the algorithm in different areas by dynamically adjusting the representation of the parameter space, and enables the algorithm to more effectively adapt to the local characteristics of the loss function by dynamically transforming the parameter space. Specifically, the dynamic nonlinear transformation can be regarded as the use of an adaptive mirror in the parameter space, and the adaptive mirror dynamically adjusts its shape according to the local characteristics of the loss function, thereby achieving local magnification or reduction of the search space and accelerating the algorithm to find the optimal solution. Furthermore, in the transformation function T(θ,Λ), θ represents a point in the original parameter space, Λ represents a set of transformation parameters, and the purpose of the transformation function T(θ,Λ) is to map the original parameter space to a new space for searching in the new space. The transformation function can be expressed as:
[0163] T(θ,Λ)=θ·exp(λ(θ,Λ))
[0164] Where λ(θ, Λ) is a dynamic adjustment function based on the current point θ and the transformation parameter set Λ, which is used to control the local enlargement or reduction of the parameter space.
[0165] Furthermore, the dynamic adjustment function λ(θ,Λ) adjusts the strength of the transformation according to the loss value of the current search point and the loss value of its surrounding points. The calculation method can be expressed as:
[0166]
[0167] In the formula, κ is a regulating factor, Σ-1 It is the inverse of the loss gradient covariance matrix of the current exploration point set, which is used to measure the heterogeneity of the loss function changes in different directions in the parameter space.
[0168] 6. According to the effect of the current iteration, dynamically adjust the search radius to optimize the breadth and depth of the search to more effectively explore the parameter space. Specifically, the search radius is dynamically adjusted according to the degree of improvement of the loss function, and the adjustment method can be expressed as:
[0169]
[0170] In the formula, ΔL=L(θ i )-L(θ i+1 ),∈ ad is a threshold value set to determine whether the loss function has been significantly improved. Preferably, the expansion factor γ ad is set to 1.05, the reduction factor λ ad is set to 0.95, ∈ ad Set to 0.01.
[0171] In order to make the adjustment of the search radius more detailed, so that the search radius can be adaptively adjusted according to the actual improvement of the loss function, in another embodiment, a nonlinear adjustment factor Φ(ΔL,∈) is defined to consider the relationship between the loss function improvement ΔL and the threshold ∈, which can be expressed as:
[0172]
[0173] The search radius is adjusted by the nonlinear adjustment factor, which can be expressed as:
[0174] r i+1 =r i ·(γ·Φ(ΔL,∈)+λ·(1-Φ(ΔL,∈)))
[0175] 7. Repeat the above steps until the preset stop iteration condition is met, indicating that the model training is completed. In one embodiment, the preset stop iteration condition is reaching a preset maximum number of iterations. Preferably, the preset maximum number of iterations is set to 1000 times.
[0176] Step 4: Feature Dimensionality Reduction Model Training
[0177] The data after feature extraction is input into the feature dimensionality reduction model for training the feature dimensionality reduction model. The present invention adopts an improved autoencoder structure, which includes two main parts: an encoder and a decoder. The task of the encoder is to compress the input data into a smaller representation, namely a feature vector, and the decoder attempts to reconstruct the original data from this feature vector. In view of the data feature information that may be ignored by the traditional autoencoder, the present invention ensures that only the most useful information is passed to the feature vector through an information filtering layer during the encoding stage, which helps to improve the quality and representativeness of the feature vector. In addition, during the decoding stage, the decoded output is compared with the original input and fed back to the encoder to further optimize the encoding process and ensure that the encoded feature vector can more accurately capture the key information of the input data.
[0178] Specifically, the training process of the improved autoencoder algorithm is as follows:
[0179] 1. Initialize the parameters of the encoder and decoder, and set the weight and bias of the encoder to and The decoder weights and biases are and The initialization process can be expressed as:
[0180]
[0181]
[0182] In the formula, Indicates that the mean is 0 and the variance is σ 2 The normal distribution of .
[0183] 2. The information filter layer is trained through training data to determine which information is most useful to the feature vector so that it can effectively select and transmit information. Specifically, the information filter layer aims to extract information from the input data p input Extract the most useful information from the information filter layer, and set the output of the information filter layer to be p filtered , and its calculation method can be expressed as:
[0184]
[0185] In the formula, and They represent the weight and bias of the information filtering layer respectively, and Sig() is the Sigmoid activation function.
[0186] In one embodiment, depends on the input data p input The main components of the input data are selected as The initial value of The calculation method can be expressed as:
[0187]
[0188] Wherein, PCA ( ) is a principal component analysis operation. Preferably, the principal component analysis algorithm is set to retain the first 128 principal components.
[0189] 3. Input the training data into the encoder and generate the compressed feature vector through the information filtering layer. Specifically, the encoding process converts the filtered data P filtered Compressed into feature vector p faetures , which can be expressed as:
[0190]
[0191] Furthermore, for the encoder weights The update is calculated by the gradient descent algorithm. Assuming that the goal of the encoder is to minimize the difference between the original data and the reconstructed data, its update method can be expressed as:
[0192]
[0193] In the formula, is the updated encoder weight, η as is the learning rate, is the loss function, is the loss function for Preferably, the loss function Select the cross entropy loss function and the learning rate η as Set to 0.01.
[0194] 4. Input the feature vector into the decoder to reconstruct the data. Further, the reconstructed data is compared with the original data, and the parameters of the encoder and decoder are adjusted through error feedback to reduce information loss and reconstruction error. Specifically, the decoding process starts from the feature vector p features Reconstruct the data to get p reconstructed can be expressed as:
[0195]
[0196] Similarly, The update also uses the gradient descent method, which can be expressed as:
[0197]
[0198] In the formula, is the updated decoder weight, Reflects the loss function on the decoder weight sensitivity.
[0199] Furthermore, the error feedback is based on the original data p input and reconstruct the data p reconstructed The difference between the two, adjusting the codec parameters to reduce the reconstruction error, can be expressed as:
[0200]
[0201] Where λ is the learning rate, represents the gradient of the corresponding parameter, represents the L2 norm.
[0202] 5. Repeat the above steps until the preset stop iteration condition is met, indicating that the model training is completed. In one embodiment, the preset stop iteration condition is reaching a preset maximum number of iterations. Preferably, the preset maximum number of iterations is set to 1000 times.
[0203] Step 5: Classifier model training
[0204] The reduced-dimensional data is input into a classifier for training the classifier. The present invention adopts an extreme learning machine algorithm based on gradient enhancement learning. The extreme learning machine mainly consists of an input layer, a hidden layer, and an output layer. It uses a single hidden layer to accelerate the learning process and can effectively handle classification problems. The present invention uses a gradient enhancement mechanism to adjust the weight of the hidden layer, so that the model pays more attention to samples that are difficult to classify during the training process.
[0205] Specifically, the training process of the extreme learning machine algorithm based on gradient reinforcement learning is as follows:
[0206] 1. Initialize the weights and biases of the extreme learning machine model, including the weights from the input layer to the hidden layer and bias And the weights from the hidden layer to the output layer Specifically, the process of initializing parameters can be expressed as:
[0207]
[0208]
[0209]
[0210] In the formula, and They represent the weight from the input layer to the hidden layer, the bias of the hidden layer, and the weight from the hidden layer to the output layer during initialization. The above parameters are distributed through a normal distribution. Random initialization, where and is the corresponding variance, controlling the dispersion of the initialization.
[0211] 2. Input the training data into the extreme learning machine model and calculate the activation value of the hidden layer and the final output value through forward propagation. Specifically, during the forward propagation process, the activation value of the hidden layer And the predicted value q of the output layer y The calculation method can be expressed as:
[0212]
[0213]
[0214] In the formula, q x is the input feature vector, and Sig() is the Sigmoid activation function.
[0215] 3. In each iteration, adjust the weights of the hidden layer for misclassified samples The model is made to enhance its learning ability for these samples. Preferably, the gradient enhancement learning rate q η is set to a relatively small value, such as 0.1, to ensure stable learning of the model. Specifically, the loss of each sample in gradient reinforcement learning is To update the weights, the update method can be expressed as:
[0216]
[0217]
[0218] In the formula, and represents the updated weight, is the loss function of the i-th sample, q η is the learning rate for gradient boosting, and Represent the loss function about and Preferably, the gradient enhancement learning rate q η is set to 0.01.
[0219] In one embodiment, the loss function Using the mean square error loss function, the loss for a single sample can be expressed as:
[0220]
[0221] In the formula, is the output predicted by the model, is the true label. Gradient It can be calculated using the chain rule:
[0222]
[0223] Similarly, yes Gradient for:
[0224]
[0225] In the formula, Represents the activation value of the hidden layer, which serves as the output layer weight direct influencing factors.
[0226] 4. Perform interactive optimization based on the performance of the model on the validation set. Specifically, if the classification accuracy of a certain category of samples is lower than expected, fine-tune the weights from the hidden layer to the output layer. To improve the recognition rate of a specific category, the adjustment of the output layer weight can be expressed as:
[0227]
[0228] In the formula, is the weight adjustment based on the feedback.
[0229] Furthermore, the weight adjustment amount Dynamic calculation is performed based on the performance feedback of the model. If the classification accuracy of a certain category is not high and the model's ability to distinguish this category needs to be enhanced, then, The calculation of is based on the feedback adjustment of category performance and can be expressed as:
[0230]
[0231] Where β is the coefficient for adjusting the intensity, and Represent the target performance and current performance respectively, q dir is the adjustment direction determined according to performance feedback. Preferably, β is set to 0.1.
[0232] 5. Repeat the above steps until the preset stop iteration condition is met, indicating that the model training is completed. In one embodiment, the preset stop iteration condition is reaching a preset maximum number of iterations. Preferably, the preset maximum number of iterations is set to 1000 times.
[0233] Step 6: Cybersecurity Data Analysis
[0234] The above model is trained using training samples. After the model training is completed, the trained model is used to process new samples. By classifying the new samples, network security vulnerabilities are identified to achieve fast and accurate network security analysis, helping security experts to quickly identify and respond to potential network attacks and achieve automated blocking or mitigation of attacks.
[0235] Thus, comprehensive and targeted improvements have been made from data preprocessing to feature extraction, and then to model training and optimization. By introducing innovative data identification processing, a data expansion method based on the generative adversarial network algorithm of quantum information quantum state reuse, and a feature extraction algorithm using a 3-layer fully connected neural network, the present invention not only improves the accuracy of network security data analysis, but also enhances the generalization ability of the model. This solves a series of problems in the prior art to a large extent, can greatly improve the efficiency and accuracy of network security data classification and identification, quickly identify and respond to potential network attacks, and achieve automatic blocking or mitigation of attacks.
[0236] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.
[0237] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0238] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0239] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0240] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0241] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A network security data analysis method based on big data, characterized in that: include: Collect characteristic data from network traffic logs, system access records, abnormal login attempts, and various network attacks, and annotate the data; Using an adversarial network algorithm to perform data expansion on the original data in the original data set to obtain preprocessed data; The three-layer fully connected neural network is used to extract data features, and the neural network feature extraction model of the adaptive fractal exploration optimization algorithm is used to achieve efficient optimization of the neural network weights and biases, explore a wider parameter space, and effectively improve the parameter optimization efficiency of the neural network; The feature-extracted data is input into the feature dimensionality reduction model for training, ensuring that the encoded feature vector can more accurately capture the key information of the input data; The reduced-dimensional data is input into the classifier for classifier training. The extreme learning machine algorithm based on gradient enhancement learning uses the gradient enhancement mechanism to adjust the weights of the hidden layer, so that the model pays more attention to samples that are difficult to classify during the training process. Use the trained model to process new samples, classify the new samples, and then achieve fast and accurate network security analysis, quickly identify and respond to potential network attacks, and achieve automatic blocking or mitigation of attacks.
2. The method according to claim 1, characterized in that Collect various raw data from the network, determine the original data set according to the raw data, and obtain a data model with multiple attributes, including: Collect multiple raw data from the network, and the data storage format adopts SQL structured database format, including multiple attribute fields and multiple data attributes; A data set is determined based on the original data, and 10 data attributes are selected, specifically: a x is the source IP address, indicating the source of the data packet; a y is the target IP address, indicating the receiving destination of the data packet; a z is the transmission protocol type, such as TCP, UDP, etc.; a aa is the source port number, indicating the port from which the data packet is sent; a ab is the destination port number, indicating the port where the data packet is received; a ac is the packet size in bytes; a ad Session duration, which records the length of time from connection establishment to disconnection; a ae is the data packet transmission rate, in packets per second; a af A feature summary of the payload data, such as a hash value, used to quickly identify the data content; a ag A flag to mark whether the data packet contains known attack features.
3. The method according to claim 2, characterized in that The collected original data is expanded and processed to obtain optimized data, making it closer to the actual data distribution, including: The collected and annotated training data is expanded using a generative adversarial network algorithm based on quantum information quantum state reuse. The algorithm structure consists of a generator (G) and a discriminator (D). The generator is responsible for generating data and receiving quantum state-encoded noise signals as input. The quantum state reuse technology is used to improve the diversity and quality of data. The discriminator evaluates the authenticity of the data to optimize the performance of the generator. Initialize the parameters of the generator G and the discriminator D to be b G0 , b D0 , the parameters are continuously updated during the training process to improve the model performance, and the input noise signal is quantum-encoded. The traditional noise signal is converted into a quantum state using the principle of quantum computing to increase the density and complexity of the information. The quantum state encoding process can be expressed as: c encoded =Q(c input ) In the formula, c encoded represents the noise signal after quantum state encoding, Q represents the quantum state encoding operation, c input is the original noise signal; Generator G receives the noise signal encoded in the quantum state and generates data through the internal network structure, simulating the high efficiency of quantum information processing to produce high-quality data samples; The discriminator D evaluates the difference between generated data and real data, trying to distinguish whether the input data is generated by the generator G or comes from the real data set. The authenticity evaluation of the generated data can be expressed as: D score =D(b D ,p decoded ||p real ) Where D score represents the score given by the discriminator D, b D is the parameter of the discriminator D, p decoded is the generated data after quantum state decoding, p real is the real data, and || indicates the merging of data.
4. The method according to claim 2, characterized in that: The three-layer fully connected neural network is used to extract data features, and the neural network feature extraction model of the adaptive fractal exploration optimization algorithm is used to achieve efficient optimization of the neural network weights and biases, explore a wider parameter space, and effectively improve the parameter optimization efficiency of the neural network; Step 1: Select an initial parameter set, randomly generate a set of parameters as the starting point according to the structure of the neural network, and set the initial granularity and search radius of the exploration. Specifically, set the initial parameter set to θ0, including the weight w and bias b of the neural network, and initialize the search granularity δ0 and search radius r0; Step 2: In each iteration, a set of self-similar exploration points is generated according to the current parameter point to generate a fractal structure. The exploration points are distributed around the current point according to the fractal rule to form a pattern similar to the fractal structure in nature. Specifically, from the current parameter point θ i , generate a set of self-similar exploration points {θ i,j }, where j indicates the jth exploration point, and sine and cosine functions are used to generate points distributed on the sphere of high-dimensional space to simulate the distribution of fractal structures in nature; Step 3: Use the dynamic information entropy feedback mechanism to optimize the search process of neural network parameters. By calculating the information entropy of the current exploration point set, dynamically adjust the search strategy to respond to the complexity changes of the parameter space, dynamically adjust the search granularity and direction, and enhance the global search ability and adaptability of the algorithm, so as to explore the parameter space more effectively. Specifically, let the set of all exploration points in the current iteration be S = {θ i,j }, where i is the number of iterations, j is the index of the exploration point, and for each exploration point, its loss function value L(θ i,j ), and normalize the loss function values so that their sum is 1. The normalization process can be expressed as: In the formula, p(θ i,j ) represents the exploration point θ i,j The normalized weight of Step 4: Adaptively and dynamically adjust the search granularity according to the loss function value of the current exploration point. If the loss of multiple exploration points is low, reduce the search granularity to refine the search; if the loss is high, increase the search granularity to speed up the search. Specifically, the adaptive dynamic adjustment of the search granularity can ensure that the search granularity is refined when a better solution is found, and is increased when it is difficult to find a better solution to speed up the exploration. According to the loss function L at the exploration point {θ i,j }, dynamically adjust the search granularity δ i+1 can be expressed as: In the formula, α self is the reduction factor, β self is the expansion factor, L(θ i ) is the parameter θ i The loss value at the time, Ω is an indicator based on the historical search efficiency; Step 5: Calculate the loss function value of all exploration points and select the point with the smallest loss as the new current point, which can be expressed as: In the formula, θ i+1 is the optimal parameter exploration point of the current iteration, and T(θ, A) is the transformation function. The algorithm can improve its search efficiency and accuracy in different regions by dynamically adjusting the representation of the parameter space, and can more effectively adapt to the local characteristics of the loss function by dynamically transforming the parameter space. Step 6: According to the effect of the current iteration, dynamically adjust the search radius to optimize the breadth and depth of the search to more effectively explore the parameter space. Specifically, the search radius is dynamically adjusted according to the degree of improvement of the loss function. The adjustment method can be expressed as: In the formula, ΔL=L(θ i )-L(θ i+1 ),∈ ad It is the set threshold used to judge whether the loss function has been significantly improved.
5. The method according to claim 2, characterized in that: The feature-extracted data is input into the feature dimensionality reduction model for training the feature dimensionality reduction model, including: The training of the feature dimensionality reduction model includes two main parts: the encoder and the decoder. The encoder's task is to compress the input data into a smaller representation, namely the feature vector, and the decoder tries to reconstruct the original data from this feature vector. In view of the data feature information that the traditional autoencoder may ignore, in the encoding stage, the information filtering layer is used to ensure that only the most useful information is passed to the feature vector, which helps to improve the quality and representativeness of the feature vector. In addition, in the decoding stage, the decoded output is compared with the original input and fed back to the encoder to further optimize the encoding process and ensure that the encoded feature vector can more accurately capture the key information of the input data.
6. The method according to claim 5, characterized in that The reduced-dimensional data is input into the classifier for classifier training, so that the model pays more attention to samples that are difficult to classify during training, including: Initialize the weights and biases of the extreme learning machine model, including the weights from the input layer to the hidden layer and bias And the weights from the hidden layer to the output layer Specifically, the process of initializing parameters can be expressed as: In the formula, and They represent the weight from the input layer to the hidden layer, the bias of the hidden layer, and the weight from the hidden layer to the output layer during initialization. The above parameters are distributed normally. Random initialization, where and is the corresponding variance, controlling the dispersion of the initialization; The training data is input into the extreme learning machine model, and the activation value of the hidden layer and the final output value are calculated through forward propagation. Specifically, during the forward propagation process, the activation value of the hidden layer And the predicted value q of the output layer y The calculation method can be expressed as: In the formula, q x is the input feature vector, and Sig() is the Sigmoid activation function.
7. The method according to claim 6, characterized in that Determine the classification of network security data and conduct network data security identification, including: Use the trained model to process new samples and classify them to achieve fast and accurate network security analysis, helping security experts quickly identify and respond to potential network attacks and automatically block or mitigate attacks.
Citation Information
Cited By
Market dishonesty information knowledge graph entity identification method
CN119272775A
Network security situation assessment method based on data mining
CN120074967A
A Network Security Situation Assessment Method Based on Data Mining
CN120074967B
