A method and system for detecting abnormal traffic in a database based on an improved 1DCNN network.
By combining an improved 1DCNN with a BiLSTM network, the problems of information loss and insufficient temporal capture in traditional 1DCNN in complex nonlinear traffic patterns are solved, achieving higher accuracy and robustness in abnormal traffic detection.
Patent Information
- Application Number
- CN202411066969.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2025-10-31
- Estimated Expiration
- 2044-08-05
AI Technical Summary
Traditional one-dimensional convolutional neural networks (1DCNNs) are prone to losing key information when processing complex nonlinear traffic patterns and have difficulty capturing temporal information, resulting in insufficient accuracy in abnormal traffic detection and an inability to effectively deal with complex and covert network attacks.
By combining an improved one-dimensional convolutional neural network (1DCNN) with a bidirectional long short-term memory network (BiLSTM), and through residual connections and adaptive hybrid pooling, the information transmission mechanism is enhanced, complex nonlinear and temporal patterns are captured, and the network's expressive power is improved.
It significantly improves the accuracy and robustness of abnormal flow detection, can capture data features at a deeper level, and enhances the model's adaptability to complex nonlinear patterns.
Smart Images
Figure CN119030760B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method, system, device, and readable storage medium for detecting abnormal traffic in databases based on an improved one-dimensional convolutional neural network and a bidirectional long short-term memory network, belonging to the field of database detection. Background Technology
[0002] With the rapid development of network technology, anomaly traffic detection plays a crucial role in maintaining network security and stability. Anomaly traffic often indicates potential network attacks or malfunctions; therefore, early detection and response to anomaly traffic are essential to ensuring normal network operation. Currently, various methods have emerged for anomaly traffic detection, including those based on statistics, machine learning, and deep learning.
[0003] Traditional anomaly detection methods have limitations when handling complex nonlinear traffic patterns. While commonly used one-dimensional convolutional neural networks (1DCNNs) can effectively extract spatial features from data, their convolution operations are prone to losing crucial information, limiting the network's expressive power. Especially when facing increasingly complex and covert network attacks, traditional 1DCNNs perform poorly, failing to provide sufficiently accurate anomaly detection results. 1DCNNs use convolutional kernels of different sizes to perform convolution operations on the input data to extract its spatial features. The input data format is n×1×m, representing n samples, each a one-dimensional sequence of length m. Convolution operations help capture data features at a deeper level. However, in the convolution calculations of 1DCNN convolutional layers, weights and biases are directly applied after the convolution operation, easily causing the loss of important information during the convolution process, and the network's capabilities are relatively weak. Therefore, to improve performance, traditional 1DCNNs need to be improved by introducing a more efficient information transmission mechanism to better capture complex nonlinear traffic patterns, thereby enhancing the network's expressive power. This improvement is of great significance for addressing the evolving threats posed by cyber attackers and enhancing the accuracy and robustness of abnormal traffic detection.
[0004] Furthermore, while 1DCNNs primarily focus on extracting spatial features from data, they have limitations in capturing temporal information, such as the trends and periodicity of traffic flow. Bidirectional Long Short-Term Memory (BiLSTM) networks, on the other hand, are better able to capture the temporal dependencies and contextual information in the input data. 1DCNNs struggle to accurately identify and classify complex nonlinear traffic patterns, while BiLSTMs can effectively handle complex temporal patterns through memory and forgetting mechanisms. However, BiLSTM models alone are weak in extracting spatial features from data. Database traffic data may contain multiple dimensions and complex spatial patterns, which are crucial for anomaly detection. A lack of spatial feature extraction capabilities can lead to inaccurate identification of certain anomaly patterns. Therefore, combining 1DCNNs with BiLSTMs can fully leverage the advantages of both, enhance the model's adaptability to complex nonlinear patterns, and improve the accuracy of anomaly traffic detection. Attached Figure Description
[0005] Figure 1 This is a system framework diagram.
[0006] Figure 2 This is the algorithm flowchart. Summary of the Invention
[0007] To address the aforementioned technical problems, this invention proposes a database anomaly traffic detection method based on an improved 1DCNN network, with the following specific steps:
[0008] (1) Network traffic data collection;
[0009] (2) Transmitting and processing network traffic data;
[0010] (3) The detection of nonlinear traffic patterns is carried out using the improved 1DCNN algorithm, and single traffic session detection and malicious event alarm are performed.
[0011] (4) Visual display of test results;
[0012] (5) System management.
[0013] Furthermore, in step (3), the specific algorithm steps are as follows:
[0014] 3.1 Define the number of input categories n.
[0015] 3.2 Calculate the output of the z-th layer Where ReLU is the activation function; c z-1 For the (z-1)th layer, the c-th channel is used; Attention is the attention mechanism. This is the input vector of the z-th layer, which is the output vector of the i-th channel of the (z-1)-th layer; Let z be the weight matrix of the convolution kernel parameters, where z represents the z-th layer, i is the i-th channel of the z-th layer, and c is the channel in the (z-1)-th layer. This is the bias term for the i-th channel of the z-th layer; This is the output vector of the i-th channel in the (z-2)-th layer;
[0016] Calculate the mixed pooling output of the j-th neuron in the i-th channel of the z-th layer.
[0017] Where σ() is the activation function, p is the width of the pooling kernel, and x z(i,t) This is the input of the t-th neuron in the i-th channel of the z-th layer;
[0018] Calculate the output of the fully connected layer z in, The weight of the line connecting the i-th neuron in layer z-1 to the j-th neuron in layer z; This represents the output value of the i-th neuron in layer z-1. This is the bias term for the j-th neuron in layer z-1; This is the output value of the i-th neuron in the (z-2)-th layer.
[0019] Finally, the classification result is calculated, and the final category of the sample is the position of the maximum value among all values.
[0020] 3.3 Input data is processed through BiLSTM twice, and Dropout is used in both processes to prevent overfitting.
[0021] 3.4 The output data from 1DCNN and BiLSTM are concatenated and then input into a fully connected layer. The number of neurons in the fully connected layer is set according to the number of classifications n.
[0022] 3.5 Based on the number of categories n, determine whether to use the binary sigmoid loss function or the multi-class softmax loss function.
[0023] 3.6 Output the number representing the category.
[0024] 3.7 Classify and interpret the output results.
[0025] Based on the above method, this invention further proposes a database abnormal traffic detection system based on an improved 1DCNN network, the system comprising:
[0026] (1) Network traffic data acquisition module;
[0027] (2) Network traffic data transmission and processing module;
[0028] (3) Detection module based on improved 1DCNN algorithm. This module uses the improved 1DCNN algorithm to detect non-linear traffic patterns, and performs single traffic session detection and malicious event alarm.
[0029] (4) Visualization module for test results;
[0030] (5) System Management Module.
[0031] Furthermore, in module (3), the steps of the improved 1DCNN algorithm are as follows:
[0032] 3.1 Define the number of input categories n.
[0033] 3.2 Calculate the output of the z-th layer Where ReLU is the activation function; c z-1 For the (z-1)th layer, the c-th channel is used; Attention is the attention mechanism. This is the input vector of the z-th layer, which is the output vector of the i-th channel of the (z-1)-th layer; Let z be the weight matrix of the convolution kernel parameters, where z represents the z-th layer, i is the i-th channel of the z-th layer, and c is the channel in the (z-1)-th layer. This is the bias term for the i-th channel of the z-th layer; This is the output vector of the i-th channel in the (z-2)-th layer;
[0034] Calculate the mixed pooling output of the j-th neuron in the i-th channel of the z-th layer.
[0035] Where σ() is the activation function, p is the width of the pooling kernel, and x z(i,t) This is the input of the t-th neuron in the i-th channel of the z-th layer;
[0036] Calculate the output of the fully connected layer z in, The weight of the line connecting the i-th neuron in layer z-1 to the j-th neuron in layer z; This represents the output value of the i-th neuron in layer z-1. This is the bias term for the j-th neuron in layer z-1; This is the output value of the i-th neuron in the (z-2)-th layer.
[0037] Finally, the classification result is calculated, and the final category of the sample is the position of the maximum value among all values.
[0038] 3.3 Input data is processed through BiLSTM twice, and Dropout is used in both processes to prevent overfitting.
[0039] 3.4 The output data from 1DCNN and BiLSTM are concatenated and then input into a fully connected layer. The number of neurons in the fully connected layer is set according to the number of classifications n.
[0040] 3.5 Based on the number of categories n, determine whether to use the binary sigmoid loss function or the multi-class softmax loss function.
[0041] 3.6 Output the number representing the category.
[0042] 3.7 Classify and interpret the output results.
[0043] Compared with the prior art, the present invention has produced significant beneficial effects:
[0044] (1) Deeply capture the features of the data. The problem of key information loss in convolution calculation in existing 1DCNNs has been perfectly solved, and the prediction accuracy of the network model has been improved.
[0045] (2) Introduce a more effective information transmission mechanism to better capture complex nonlinear flow patterns. To address the shortcomings of existing 1DCNNs in nonlinear flow models, a method using residual connections is proposed to introduce cross-layer information transmission, thereby improving the network's expressive power.
[0046] Introducing cross-layer information transfer through residual connections helps the network better capture complex nonlinear traffic patterns and improves the network's expressive power. Detailed Implementation
[0047] This invention constructs a database abnormal traffic detection system based on an improved 1DCNN algorithm, such as... Figure 1 As shown. The functions of the database abnormal traffic detection system during database use are as follows:
[0048] (1) Data Acquisition: The NSL-KDD Dataset and UNSW-NB15 Dataset were selected. These datasets provide raw data (pcap files) on network traffic. These datasets contain normal traffic and different types of anomalous traffic for training and testing the model. For example, a TCP traffic record from the NSL-KDD dataset was selected:
[0049] Source IP address: 192.168.1.1
[0050] Target IP address: 10.0.0.1
[0051] Source port: 12345
[0052] Target port: 80
[0053] Protocol type: TCP
[0054] Data packet size: 1500 bytes
[0055] ...(other characteristics)
[0056] (2) Data Transmission and Processing: Utilizing distributed and asynchronous processing mechanisms, data is distributed to multiple nodes for asynchronous processing, improving system speed and throughput. Lightweight data formats, such as Apache Avro or Protocol Buffers, are used to reduce data transmission overhead. The pcap file is parsed to extract relevant features. This may include source IP address, destination IP address, port information, etc. Simultaneously, tags need to be processed, marking normal traffic as 0 and abnormal traffic as 1. The TCP traffic records above are preprocessed as follows:
[0057] Feature extraction: [192.168.1.1,10.0.0.1,12345,80,TCP,1500,...]
[0058] Label: label:0 (Normal traffic)
[0059] Data format conversion: [0.2,0.8,0.5,0.3,0.0,0.6,...]
[0060] (3) Detection Function: Based on the improved 1DCNN algorithm, the detection function under non-linear traffic mode is implemented, as well as the detection of single traffic sessions and the alarm of malicious events. The model detects each preprocessed data to determine whether it is a malicious event and provides an alarm with the corresponding probability. Finally, the malicious attack event is persisted.
[0061] Specifically, for the detection algorithm, after determining the number of categories n, the traffic detection algorithm uses two parts of input data: the first part extracts the spatial features of the traffic through an improved one-dimensional convolutional neural network (1DCNN) model, and the second part extracts the temporal features of the traffic through a bidirectional long short-term memory (BiLSTM) network model. The outputs of these two models are concatenated to form a comprehensive feature representation. Finally, the result of classifying the traffic data is obtained through a fully connected layer and the classifier output. This design can effectively capture spatial and temporal features and improve the accuracy of traffic anomaly detection. The algorithm flow is as follows: Figure 2 As shown, the specific algorithm steps are as follows:
[0062] 3.1 Define the number of input categories n.
[0063] 3.2 1DCNN Section: Perform one-dimensional convolution on the input data. Pool the convolutional data. Classify the samples using the Softmax function, repeating this operation twice.
[0064] This invention proposes a novel convolution calculation method that introduces cross-layer information transfer through residual connections. This helps the network better capture complex nonlinear traffic patterns and improves the network's expressive power. The specific calculation formula is as follows:
[0065]
[0066] Among them, y z Let c represent the output of the z-th layer, ReLU represent the activation function, and c represent the output of the z-th layer. z-1 This represents the c-th channel of layer z-1, where Attention is the attention mechanism. It is the input vector of the z-th layer and also the output vector of the i-th channel of the (z-1)-th layer. Let z represent the weight matrix of the convolution kernel parameters, where z represents the z-th layer, i represents the i-th channel in the z-th layer, and c represents the c-th channel in the (z-1)-th layer. Let b represent the bias term b of the i-th channel in the z-th layer. It is the output vector of the i-th channel of the z-2-th layer.
[0067] Single average or max pooling methods may fail to capture key information comprehensively and effectively when faced with different input feature distributions. This invention proposes adaptive hybrid pooling, which dynamically adjusts the model to better adapt to diverse input feature distributions. When the mean of a local region is large, the method favors average pooling; when the mean is small, it favors max pooling. This balance helps retain useful feature information and suppress noise. The hybrid pooling formula is calculated as follows:
[0068]
[0069] Among them, y z(i,j) Let σ() represent the mixed pooling output of the j-th neuron in the i-th channel of layer z, where σ is the activation function, p represents the width of the pooling kernel, and x... z(i,t) This represents the input of the t-th neuron in the i-th channel of the z-th layer.
[0070] After multiple rounds of convolution and pooling, the fully connected layer enhances the expressive power of extracting features from the data. Similarly, the use of residual connections and stronger non-linear representation capabilities in the fully connected layer helps improve the network's performance in complex tasks, thus improving the performance of one-dimensional convolutional neural networks in database anomaly traffic detection and enhancing the model's expressive and generalization abilities. The formula for the fully connected layer is as follows:
[0071]
[0072] Among them, y z This represents the output of the fully connected layer z, where n represents the number of neurons in the layer preceding the fully connected layer, i.e., layer z-1. This represents the weight of the connection between the i-th neuron in layer z-1 and the j-th neuron in layer z. Let x represent the output value x of the i-th neuron in layer z-1, and This represents the bias term for the j-th neuron in layer z-1. It is the output value x of the i-th neuron in the z-2-th layer.
[0073] Finally, the softmax function is used to output the classification results of the samples. Where z i p(x) represents the input of the i-th neuron in the output layer, K represents the total number of all classes, and p(x) represents the input of the i-th neuron in the output layer. i ) represents the probability of classifying the corresponding sample on the classifier, and the final class of the sample is the position of the maximum value among all values.
[0074] 3.3 BiLSTM Part: Input data passes through two Bidirectional Long Short-Term Memory (BiLSTM) networks. Dropout is used in each BiLSTM part to prevent overfitting.
[0075] 3.4 The output data from the 1DCNN and BiLSTM are concatenated. The concatenated data is then input into a fully connected layer. The number of neurons in the fully connected layer is set according to the number of classifications, n. For binary classification, it is set to 1; for multi-class classification, it is set to n.
[0076] 3.5 Based on the number of categories n, determine whether to use the binary sigmoid loss function or the multi-class softmax loss function.
[0077] 3.6 Output the number representing the category. For binary classification, the output is 0 or 1; for multi-class classification, the output is an integer in [0, n-1].
[0078] 3.7 Based on the output results, perform classification and interpretation. In binary classification, 0 represents normal traffic, and 1 represents abnormal traffic. In multi-class classification, the specific classification is determined based on the mapped labels.
[0079] The model's input is a tensor of size n×28×28, where n is the batch size. The convolutional layers use 3×3 kernels, and both the output of the convolutional parts and the output of the BiLSTM part have a dimension of 128. These two outputs are merged to form a tensor of size n×256. This merged tensor is then passed through a fully connected layer, with an output dimension of 128. Finally, depending on whether it's binary or multi-class classification, the Sigmoid and Softmax loss functions are selected to perform the classification interpretation. Classifying the aforementioned TCP traffic records yields a classification result of label = 0.
[0080] (4) Visualization function: The detection data is displayed intuitively, including traffic data and statistics and visualization of malicious events, event query and statistical visualization of offline traffic data detection results, query and visualization of past malicious attack events, generation of custom reports, and provision of customized services for users.
[0081] (5) System Management: Utilize existing identity authentication and permission management mechanisms to ensure system security, including configurations such as user registration and login, user management, and system parameter settings, and provide an easy-to-use user interface.
[0082] The present invention also provides an apparatus comprising: a data acquisition device, a processor, and a memory; the data acquisition device is used to acquire data; the memory is used to store one or more program instructions; and the processor is used to execute one or more program instructions to perform any of the methods described above.
[0083] The present invention also provides a computer-readable storage medium containing one or more program instructions for performing any of the methods described above.
[0084] The units, devices, or modules described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above devices are described by dividing them into various modules according to their functions. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection between the devices or units shown or discussed can be through some interfaces, and the indirect coupling or communication connection between the devices or units can be electrical, mechanical, or other forms.
[0085] Those skilled in the art will also know that, besides implementing the controller using purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller function as logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices within it used to implement various functions can also be considered structures within that hardware component. Alternatively, the devices used to implement various functions can be considered as both software modules implementing the method and structures within a hardware component.
[0086] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0087] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.
[0088] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. This application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.
[0089] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A method for detecting abnormal traffic in a database based on an improved 1DCNN network, characterized in that: The specific steps of this method are as follows: (1) Network traffic data collection; (2) Transmitting and processing network traffic data; (3) The detection of nonlinear traffic patterns is carried out using the improved 1DCNN algorithm, and single traffic session detection and malicious event alarm are performed; (4) Visual display of test results; (5) System management; In step (3), the specific algorithm steps are as follows: 3.1 Define the number of input categories n ; 3.2 Calculate the first z Layer output Where ReLU is the activation function; For the first z -1st floor c Each channel; Attention is the attention mechanism; For the first z The input vector of the layer, i.e., the first layer z -1st floor i The output vector of each channel; Let be the weight matrix of the convolution kernel parameters, where z Representing the z layer, i For the first z The first layer i One channel; c For the first z The passage in the -1 level; For the first z The first layer i The bias term for each channel; For the first z -2nd floor i The output vector of each channel; Calculate the first z The first in the layer i The first in the channel j Hybrid pooling output of neurons Where σ() is the activation function, p The width of the pooling kernel. For the first z The first layer i The first channel t The input of each neuron; Calculate the fully connected layer z Output ,in, For the first z -1st floor i The first neuron to the second z Layer j The weights of the connections between neurons; for z -1st floor i The output value of each neuron; for z -1st floor j Bias terms for each neuron; For the first z -2nd floor i The output value of each neuron; Finally, the classification results are calculated. The final sample's category is determined by the position of the maximum value among all values; 3.3 Input data is processed through BiLSTM twice, with Dropout used in both processes to prevent overfitting; 3.4 The output data from the 1DCNN and BiLSTM are concatenated, and then the concatenated data is input into a fully connected layer. The number of neurons in the fully connected layer depends on the number of classifications. n Configure settings; 3.5 Based on the number of categories n Determine whether to use the binary sigmoid loss function or the multi-class softmax loss function; 3.6 Output the number representing the category; 3.7 Classify and interpret the output results.
2. A database anomaly traffic detection system based on an improved 1DCNN network, the system comprising: (1) Network traffic data acquisition module; (2) Network traffic data transmission and processing module; (3) Abnormal traffic detection module; (4) Visualization module for test results; (5) System Management Module; Its features are as follows: Module (3) uses an improved 1DCNN algorithm to perform detection under nonlinear traffic patterns, and performs single traffic session detection and malicious event alarm. The steps of the improved 1DCNN algorithm are as follows: 3.1 Define the number of input categories n ; 3.2 Calculate the first z Layer output Where ReLU is the activation function; For the first z -1st floor c Each channel; Attention is the attention mechanism; For the first z The input vector of the layer, i.e., the first layer z -1st floor i The output vector of each channel; Let be the weight matrix of the convolution kernel parameters, where z Representing the z layer, i For the first z The first layer i One channel; c For the first z The passage in the -1 level; For the first z The first layer i The bias term for each channel; For the first z -2nd floor i The output vector of each channel; Calculate the first z The first in the layer i The first in the channel j Hybrid pooling output of neurons Where σ() is the activation function, p The width of the pooling kernel. For the first z The first layer i The first channel t The input of each neuron; Calculate the fully connected layer z Output ,in, For the first z -1st floor i The first neuron to the second z Layer j The weights of the connections between neurons; for z -1st floor i The output value of each neuron; for z -1st floor j Bias terms for each neuron; For the first z -2nd floor i The output value of each neuron; Finally, the classification results are calculated. The final sample's category is determined by the position of the maximum value among all values; 3.3 Input data is processed through BiLSTM twice, with Dropout used in both processes to prevent overfitting; 3.4 The output data from the 1DCNN and BiLSTM are concatenated, and then the concatenated data is input into a fully connected layer. The number of neurons in the fully connected layer depends on the number of classifications. n Configure settings; 3.5 Based on the number of categories n Determine whether to use the binary sigmoid loss function or the multi-class softmax loss function; 3.6 Output the number representing the category; 3.7 Classify and interpret the output results.