Traffic detection method and device and electronic equipment

Abstract

The invention discloses a traffic detection method, a traffic detection device and electronic equipment. The method comprises the following steps: acquiring network flow data; a stacked auto-encoder model in the traffic detection model is adopted to carry out feature extraction on the network traffic data, the stacked auto-encoder model comprises a plurality of auto-encoders, and each auto-encoder comprises an encoder used for mapping input features into implicit features and a decoder used for restoring the implicit features, the implicit features are flow features obtained after feature dimensionality reduction is carried out on the input features; and a deep neural network model in the traffic detection model is adopted to classify the network traffic data after feature extraction to obtain a traffic identification result, and the traffic identification result is used for reflecting the traffic type to which the network traffic data belongs. According to the method and the device, the technical problem of low network security protection efficiency caused by low detection precision, high false alarm rate and poor adaptability to novel attacks in an abnormal traffic detection method in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and more specifically, to a flow detection method, device, and electronic device. Background Art

[0002] With the rapid development of computer technology, the scale of internet data continues to expand, and network environments are becoming increasingly complex, leading to increasingly serious network security issues. Abnormal traffic detection, as a key component of network security protection, monitors network flows in real time and accurately identifies abnormal behavior, enabling timely action to block potential threats and protect the network from harm.

[0003] However, most existing abnormal traffic detection technologies rely on static rule matching or simple statistical analysis. While these methods can be effective in specific scenarios, they still have the following limitations when dealing with complex and diverse network attacks: First, detection accuracy is insufficient, making it easy to overlook new and more subtle network attacks. Second, the false alarm rate is high, frequently generating false positive signals within massive amounts of data, increasing the troubleshooting burden on network administrators. Third, they lack the ability to adapt to emerging threats, making it difficult to quickly respond to unknown attack patterns, resulting in lagging protection mechanisms.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present application provide a flow detection method, device, and electronic device to at least solve the technical problem of low network security protection efficiency caused by low detection accuracy, high false alarm rate, and poor adaptability to new attacks in abnormal flow detection methods in related technologies.

[0006] According to one aspect of an embodiment of the present application, a traffic detection method is provided, including: obtaining network traffic data; using a stacked autoencoder model in a traffic detection model to extract features from the network traffic data, wherein the stacked autoencoder model includes multiple autoencoders, each autoencoder includes an encoder for mapping input features into implicit features and a decoder for restoring the implicit features, and the implicit features are traffic features after feature dimensionality reduction of the input features; using a deep neural network model in the traffic detection model to classify the network traffic data after feature extraction to obtain a traffic identification result, and the traffic identification result is used to reflect the traffic type to which the network traffic data belongs.

[0007] Optionally, the traffic detection model is trained in the following manner: obtaining historical traffic logs, wherein the historical traffic logs include normal traffic and different types of abnormal traffic; training an initial autoencoder model based on the historical traffic logs to obtain a stacked autoencoder model; obtaining target latent features of the stacked autoencoder model, and training an initial neural network model based on the target latent features to obtain a deep neural network model, wherein the target latent features are used to represent the latent features in the last layer of the autoencoder of the stacked autoencoder model; and determining the traffic detection model based on the stacked autoencoder and the deep neural network model.

[0008] Optionally, after obtaining the historical traffic log, the method also includes: performing data parsing on the traffic log to obtain field data, wherein the field data includes text items and numerical items; preprocessing the field data to obtain a data set for training the initial autoencoder model, wherein the preprocessing is used to convert the text information in the field data into numerical information, and each traffic data in the data set has the same feature dimension and corresponding traffic label, and the traffic label is used to reflect the traffic type to which each traffic data belongs.

[0009] Optionally, the field data is preprocessed, including: replacing space symbols in the field data with preset symbols; replacing abnormal traffic types in the field data with digital identifiers; extracting traffic labels in the field data, binary encoding text items in the remaining field data, and one-hot encoding the traffic labels.

[0010] Optionally, an initial autoencoder model is trained based on historical traffic logs to obtain a stacked autoencoder model, including: obtaining an initial autoencoder model, wherein the initial autoencoder model includes at least a first autoencoder and a second autoencoder; training the first autoencoder with a data set to obtain a first latent feature, wherein the first latent feature is used to represent the traffic feature obtained after preliminary feature dimensionality reduction by the first autoencoder; training the second autoencoder with the first latent feature to obtain a second latent feature, wherein the second latent feature is used to represent the traffic feature obtained after secondary feature dimensionality reduction by the second autoencoder; and fusing the trained first autoencoder and second autoencoder to obtain a stacked autoencoder model.

[0011] Optionally, the first autoencoder is trained using a data set, including: determining a first input feature corresponding to the first autoencoder from the data set; mapping the first input feature to a first latent feature through forward propagation, wherein the feature dimension of the first latent feature is smaller than the feature dimension of the first input feature; reconstructing the first latent feature into a first output feature through back propagation, wherein the feature dimension of the first output feature is the same as the feature dimension of the first input feature; determining a first error between the first input feature and the first output feature, and optimizing the first error by adjusting the weight matrix corresponding to the first autoencoder.

[0012] Optionally, the second autoencoder is trained using the first latent feature, including: determining the first latent feature as the second input feature of the second autoencoder; mapping the second input feature to the second latent feature through forward propagation, wherein the feature dimension of the second latent feature is smaller than the feature dimension of the second input feature; reconstructing the second latent feature into the second output feature through back propagation, wherein the feature dimension of the second output feature is the same as the feature dimension of the second input feature; determining a second error between the second input feature and the second output feature, and optimizing the second error by adjusting the weight matrix corresponding to the second autoencoder.

[0013] Optionally, an initial neural network model is trained based on the target latent feature to obtain a deep neural network model, including: determining the initial neural network model, wherein the initial neural network model includes multiple fully connected layers; determining the second latent feature as the third input feature of the initial neural network model, and processing the third input feature through the initial neural network model to obtain a third output feature; determining a third error between the third output feature and the corresponding traffic label through a cross entropy loss function, and optimizing the third error by adjusting the model weights corresponding to the initial neural network model to obtain a deep neural network model.

[0014] Optionally, the method also includes: when the traffic identification result indicates that there is abnormal traffic in the network traffic data, extracting metadata information corresponding to the abnormal traffic, wherein the metadata information includes at least one of the following: the source IP, destination IP and corresponding abnormal traffic type of the abnormal traffic; and generating an abnormal traffic report based on the metadata information.

[0015] According to another aspect of an embodiment of the present application, a traffic detection device is also provided, including: an acquisition module for acquiring network traffic data; an extraction module for using a stacked autoencoder model in a traffic detection model to extract features from the network traffic data, wherein the stacked autoencoder model includes multiple autoencoders, each autoencoder includes an encoder for mapping input features into implicit features and a decoder for restoring the implicit features, and the implicit features are traffic features after feature dimensionality reduction of the input features; a classification module for using a deep neural network model in the traffic detection model to classify the network traffic data after feature extraction to obtain a traffic identification result, and the traffic identification result is used to reflect the traffic type to which the network traffic data belongs.

[0016] According to another aspect of the embodiments of the present application, an electronic device is provided, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned flow detection method.

[0017] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided. The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned traffic detection method by running the computer program.

[0018] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned traffic detection method when executed by a processor.

[0019] In an embodiment of the present application, network traffic data is obtained; a stacked autoencoder model in a traffic detection model is used to extract features from the network traffic data, wherein the stacked autoencoder model includes multiple autoencoders, each autoencoder includes an encoder for mapping input features into implicit features and a decoder for restoring the implicit features, and the implicit features are traffic features after feature dimensionality reduction of the input features; a deep neural network model in the traffic detection model is used to classify the network traffic data after feature extraction to obtain a traffic identification result, which is used to reflect the traffic type to which the network traffic data belongs, thereby achieving the purpose of accurately identifying normal and abnormal traffic types, thereby achieving the technical effect of improving network security protection efficiency, reducing false alarm rate and missed alarm rate, and enhancing the adaptive detection capability of new attacks, thereby solving the technical problem of low network security protection efficiency caused by low detection accuracy, high false alarm rate and poor adaptability to new attacks in the abnormal traffic detection method in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 is a hardware structure diagram of a computer terminal for implementing a flow detection method according to an embodiment of the present application;

[0022] Figure 2 is a flow chart of a flow detection method according to an embodiment of the present application;

[0023] Figure 3 is a flow chart of another flow detection method according to an embodiment of the present application;

[0024] Figure 4 Schematic diagram of the classification principle of a flow detection model according to an embodiment of the present application;

[0025] Figure 5 It is a structural diagram of a flow detection device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] First, some nouns or terms that appear in the process of explaining the embodiments of this application are subject to the following explanations:

[0029] SAE (Stacked Autoencoder): A deep learning architecture primarily used for feature extraction and dimensionality reduction in unsupervised learning. It constructs a deep network structure by training multiple autoencoders layer by layer. The autoencoder consists of two parts: an encoder and a decoder. The encoder maps the input data to a low-dimensional latent space, while the decoder restores the representation of the latent space to the original data. The characteristic of stacked autoencoders is that they can capture the deep features of the data layer by layer, thereby improving the quality of feature representation. In SAE, the decoded output of the previous layer serves as the encoded input of the next layer. Through layer-by-layer training and stacking, a high-level abstract representation of the data can be learned, which is helpful for subsequent classification or other machine learning tasks. It is widely used in fields such as image recognition, natural language processing, and anomaly detection.

[0030] DNN (Deep Neural Network): A neural network structure composed of multiple layers of nonlinear transformations that mimics the workings of neurons in the human brain to represent and learn complex data. A DNN consists of an input layer, multiple hidden layers, and an output layer. Each layer contains multiple neurons that interact through weighted connections. Deep neural networks are characterized by their powerful feature extraction and representation learning capabilities. Through layers of abstraction and transformation, they can extract high-level feature representations from raw data. They are widely used in fields such as image classification, speech recognition, and natural language processing.

[0031] One-Hot Encoding: A method of encoding categorical variables into continuous values. In one-hot encoding, each categorical value is converted into a new binary column, with only one column containing a value of 1 and all other columns containing 0. This method is often used to convert nominal variables (such as attack type) into a form that can be processed by machine learning models, helping to improve model performance and accuracy.

[0032] Stochastic Gradient Descent (SGD): An iterative optimization algorithm commonly used to train machine learning models. Unlike batch gradient descent, SGD uses only one example or a small batch of examples at a time to calculate the gradient and then update the model weights. This method has the advantages of fast computation and can converge effectively on large datasets. However, its disadvantage is that the path fluctuates significantly, and it may require more iterations to reach the optimal solution.

[0033] In order to solve the problem of poor efficiency of abnormal flow detection in related technologies, the present invention provides a flow detection method that can be run on Figure 1 Among the computer terminals shown, the computer terminal will be described below.

[0034] The flow detection method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal for implementing a flow detection method. Figure 1 As shown, the computer terminal 10 may include one or more (illustrated by 102a, 102b, ..., 102n in the figure) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0035] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0036] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the flow detection method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned flow detection method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0037] The transmission module 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission module 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission module 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0038] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0039] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0040] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a flow detection method. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] Figure 2 is a flow chart of a flow detection method according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0042] Step S202: Acquire network traffic data.

[0043] In the above step S202, real-time network traffic data may be obtained through a database interface.

[0044] In step S204, the stacked autoencoder model in the traffic detection model is used to extract features of the network traffic data, wherein the stacked autoencoder model includes multiple autoencoders, each autoencoder includes an encoder for mapping input features into implicit features and a decoder for restoring the implicit features, and the implicit features are traffic features after feature dimensionality reduction of the input features.

[0045] In the above step S204, the traffic detection model is a two-layer SAE-DNN model, the first layer is a stacked autoencoder (SAE) model, and the second layer is a deep neural network (DNN) model.

[0046] This step is primarily responsible for preprocessing and feature extraction of the collected network traffic data through the first-layer stacked autoencoder (SAE) model. The SAE model consists of multiple stacked autoencoders, each of which contains two components: an encoder and a decoder. The encoder learns a low-dimensional latent feature representation from the high-dimensional input data. This representation concentrates information crucial for anomaly detection and filters out irrelevant or redundant features. The decoder attempts to reconstruct the original input from the latent features to quantify the amount of information lost during the encoding process. Through layer-by-layer training and optimization, the SAE model can gradually extract deep features of network traffic data, laying the foundation for subsequent deep neural network training.

[0047] In step S206, the deep neural network model in the traffic detection model is used to classify the network traffic data after feature extraction to obtain a traffic identification result. The traffic identification result is used to reflect the traffic type to which the network traffic data belongs.

[0048] In the above step S206, the network traffic data after SAE feature extraction is mainly responsible for being classified through the second-layer deep neural network (DNN) model. Among them, the DNN model is a multi-layer neural network that can learn more subtle and advanced patterns from the implicit features provided by the SAE model through internal complex nonlinear transformations to distinguish normal traffic from different types of abnormal traffic types (such as SQL injection, denial of service, etc.). DNN usually contains multiple layers of fully connected layers, each layer has a large number of neurons and their weights, and the weights and biases are optimized through the back propagation algorithm to achieve the goal of minimizing classification errors. The output layer uses the Softmax function to convert the output into a probability distribution, indicating the possibility that each data belongs to a different type, thereby determining its traffic type.

[0049] Optionally, the above method also includes: when the traffic identification result indicates that there is abnormal traffic in the network traffic data, extracting metadata information corresponding to the abnormal traffic, wherein the metadata information includes at least one of the following: the source IP, destination IP and corresponding abnormal traffic type of the abnormal traffic; and generating an abnormal traffic report based on the metadata information.

[0050] In an embodiment of the present application, when abnormal traffic is detected in the network traffic data, the system will further extract metadata information related to the abnormal traffic, including but not limited to the source IP address, destination IP address, and the specific abnormal traffic type. These metadata information are the core components of the abnormal traffic report, which can help network security managers quickly locate the source and nature of abnormal activities and take targeted countermeasures. For example, if the system identifies a SQL injection attack, the report will clearly indicate the source IP of the attack, the IP of the target server, and this specific type of attack, allowing administrators to immediately block the malicious IP, check the affected database, and initiate the corresponding security audit process.

[0051] Generating abnormal traffic reports not only fulfills the basic function of anomaly detection but also provides critical information for subsequent security response and incident management. Based on the model's identification results and extracted metadata, reports are generated to automate alerting and preliminary analysis, reducing the burden of manual monitoring and improving the speed and efficiency of network security incident handling. Furthermore, abnormal traffic reports provide a basis for adjusting network policies, helping organizations continuously optimize their security systems and strengthen their resilience to potential attacks.

[0052] Through the above steps S202 to S206, the purpose of accurately identifying normal and abnormal traffic types is achieved, thereby realizing the technical effects of improving network security protection efficiency, reducing false alarm rate and missed alarm rate, and enhancing the adaptive detection capability of new attacks, thereby solving the technical problem of low network security protection efficiency caused by low detection accuracy, high false alarm rate and poor adaptability to new attacks of abnormal traffic detection methods in related technologies.

[0053] Figure 3 is a flow chart of another flow detection method according to an embodiment of the present application. Figure 3 As shown in the figure, the training process of the traffic detection model (SAE-DNN model) is included, and a more detailed description is given from data collection to model training, to result evaluation and report generation. Each link is closely connected and together constitutes an efficient abnormal traffic detection system.

[0054] Specifically, the system first obtains historical traffic logs through a database interface. Next, a preprocessing phase involves creating a dataset through log parsing, handling missing values, digitizing text, and implementing numerical standards to ensure that the data meets the SAE-DNN model input requirements. The SAE-DNN model is then trained, using stacked autoencoders for feature dimensionality reduction and extraction, and deep neural networks for classification. Next, the model's performance is checked to determine whether the detection results meet the standards. If so, an abnormal traffic report is generated, automatically including metadata such as the source IP address, destination IP address, and anomaly type to assist in security management. Otherwise, the SAE-DNN model is retrained for adjustments and optimization. The entire process is highly integrated, aiming to achieve an integrated abnormal traffic detection and management mechanism, from data to intelligent detection to timely response. The following details the SAE-DNN model training process.

[0055] Optionally, the traffic detection model (SAE-DNN model) can be trained by the following steps:

[0056] S1: Obtain historical traffic logs, where the historical traffic logs include normal traffic and different types of abnormal traffic.

[0057] In step S1, a large amount of historical traffic logs can be obtained through the database interface or network monitoring equipment, including normal traffic and various types of abnormal traffic, such as SQL injection, backdoor programs, botnets, cross-site scripting attacks (XSS), denial of service attacks, etc., to provide rich learning samples for subsequent model training and ensure that the model can fully understand the behavior patterns and characteristics of normal and abnormal traffic.

[0058] Optionally, after obtaining the historical traffic log, it also includes: performing data parsing on the traffic log to obtain field data, wherein the field data includes text items and numerical items; preprocessing the field data to obtain a data set for training the initial autoencoder model, wherein the preprocessing is used to convert the text information in the field data into numerical information, and each traffic data in the data set has the same feature dimension and corresponding traffic label, and the traffic label is used to reflect the traffic type to which each traffic data belongs.

[0059] In an embodiment of the present application, by performing log parsing on historical traffic logs, multiple field data such as source IP (attack_sip), destination IP (alarm_sip), data packet protocol (origin.proto), data packet payload content request header (payload.req_header), data packet payload content request body (payload.req_body), data packet payload content response header (payload.rsp_header), and data packet payload content response body (payload.rsp_body) can be extracted.

[0060] Furthermore, the field data after log parsing is preprocessed and a dataset is created for training the SAE model. The parsed field data contains text items and numerical items, which together describe various dimensions of traffic behavior, such as source IP, destination IP, packet protocol, request header, and request body.

[0061] For example, this dataset contains 146,756 packets, with six types of abnormal traffic: "normal," "SQL injection," "backdoor," "botnet," "cross-site scripting (XSS)," and "denial of service." Each packet contains 41 fields, including 13 numerical items and 28 text items.

[0062] Optionally, the field data is preprocessed, including: replacing space symbols in the field data with preset symbols; replacing abnormal traffic types in the field data with digital identifiers; extracting traffic labels in the field data, binary encoding text items in the remaining field data, and one-hot encoding the traffic labels.

[0063] In the embodiment of this application, in order to make the parsed field data more suitable for model training, it is comprehensively preprocessed. The specific steps are as follows:

[0064] 1) Unify the spaces in the field data into a preset replacement symbol (such as "-") to standardize the data format and avoid unnecessary errors caused by format differences during model training.

[0065] 2) Use numerical identifiers such as 0, 1, 2, 3, ..., y to replace the attack type (i.e., abnormal traffic type) and digitize the category information. For example, 0 represents normal traffic and 1 represents SQL injection, which simplifies the model processing process.

[0066] 3) Extract the traffic label (i.e., traffic type) from the field data, and perform binary encoding on the text items and category items in the remaining fields to achieve full numerical processing.

[0067] 4) Perform one-hot encoding on the extracted traffic labels to meet the requirements of the classification model.

[0068] It should be noted that each data entry in the final dataset has the same feature dimension, and each piece of data is assigned a traffic label, which clearly marks the type of the traffic data.

[0069] For example, after preprocessing, each piece of data includes 296-dimensional features and 6-dimensional labels.

[0070] In an embodiment of the present application, the obtained dataset can also be divided into a training set and a test set by using the train_test_split tool in sklearn.model_selection to prepare data for model training and verification.

[0071] S2: Train the initial autoencoder model based on historical traffic logs to obtain a stacked autoencoder model.

[0072] In step S2, the SAE model is trained on the partitioned training set to extract meaningful features from the raw traffic data and reduce its dimensionality. This process is accomplished by constructing multiple autoencoders and training them layer by layer. The encoder portion of each autoencoder converts high-dimensional data into low-dimensional latent features, while the decoder portion attempts to reconstruct the original data from the latent features. After layer-by-layer training, the autoencoders are stacked to form the SAE model. The goal of SAE is to minimize data dimensionality while preserving key information, providing optimized feature representation for subsequent deep neural network training.

[0073] Optionally, an initial autoencoder model is trained based on historical traffic logs to obtain a stacked autoencoder model, including: obtaining an initial autoencoder model, wherein the initial autoencoder model includes at least a first autoencoder and a second autoencoder; training the first autoencoder with a data set to obtain a first latent feature, wherein the first latent feature is used to represent the traffic feature obtained after preliminary feature dimensionality reduction by the first autoencoder; training the second autoencoder with the first latent feature to obtain a second latent feature, wherein the second latent feature is used to represent the traffic feature obtained after secondary feature dimensionality reduction by the second autoencoder; and fusing the trained first autoencoder and second autoencoder to obtain a stacked autoencoder model.

[0074] In this embodiment of the present application, the initial autoencoder model includes at least two layers of autoencoders: a first autoencoder and a second autoencoder. The first autoencoder is first trained to obtain a first latent feature. The second autoencoder is then trained using the first latent feature as input to obtain a second latent feature. It should be noted that this second latent feature will serve as input for subsequent training of the DNN model. Finally, the trained first and second encoders are stacked to obtain the SAE model.

[0075] Furthermore, all training data can be used to fine-tune the SAE model as a whole to further optimize model parameters and reduce reconstruction errors.

[0076] Optionally, the first autoencoder is trained using a data set, including: determining a first input feature corresponding to the first autoencoder from the data set; mapping the first input feature to a first latent feature through forward propagation, wherein the feature dimension of the first latent feature is smaller than the feature dimension of the first input feature; reconstructing the first latent feature into a first output feature through back propagation, wherein the feature dimension of the first output feature is the same as the feature dimension of the first input feature; determining a first error between the first input feature and the first output feature, and optimizing the first error by adjusting the weight matrix corresponding to the first autoencoder.

[0077] In the embodiment of the present application, the training analysis of the first encoder is as follows:

[0078] Input layer: The feature dimension of the first input feature is d=296, which is the feature dimension of the above dataset.

[0079] Hidden layer: Set the number of units (for example, 30) to reduce the feature dimension of the first input feature from d to 30.

[0080] Output layer: The feature dimension of the first output feature is d, the same as that of the input layer.

[0081] Forward propagation: Map the first input feature y to the first latent feature z. The specific expression is as follows:

[0082] z=σ(W1y+b1)

[0083] Where z represents the first latent feature, y represents the first input feature, W1 is the weight matrix, b1 is the bias vector, and σ represents the nonlinear activation function.

[0084] Back propagation: Reconstruct the first latent feature z into the first output feature y′. The specific expression is as follows:

[0085] y′=σ(W2z+b2)

[0086] Where y′ represents the first output feature, W2 is the weight matrix, and b2 is the bias vector.

[0087] Reconstruction error: Calculate the first error between the first input feature y and the first output feature y′. The mean square error (MSE) is usually used. The specific expression is as follows:

[0088] L(y,y′)=||yy′|| 2

[0089] Where L(y,y′) represents the first error.

[0090] In order to minimize the first error, the back propagation algorithm can be used to adjust the weight matrix (W1, W2) of the first autoencoder, and gradually optimize the model parameters until the model can effectively reduce the dimension and reconstruct the input features.

[0091] Optionally, the second autoencoder is trained using the first latent feature, including: determining the first latent feature as the second input feature of the second autoencoder; mapping the second input feature to the second latent feature through forward propagation, wherein the feature dimension of the second latent feature is smaller than the feature dimension of the second input feature; reconstructing the second latent feature into the second output feature through back propagation, wherein the feature dimension of the second output feature is the same as the feature dimension of the second input feature; determining a second error between the second input feature and the second output feature, and optimizing the second error by adjusting the weight matrix corresponding to the second autoencoder.

[0092] In the embodiment of the present application, the training analysis of the second encoder is as follows:

[0093] Input layer: The first latent feature obtained by training the first autoencoder is used as the second input feature, and its feature dimension is 30.

[0094] Hidden layer: Set the number of units (for example, 20) to reduce the feature dimension of the second input feature from 30 to 20.

[0095] Output layer: The feature dimension of the second output feature is 30, the same as that of the input layer.

[0096] Forward propagation: Map the second input feature (i.e., the first latent feature z) to the second latent feature z′. The specific expression is as follows:

[0097] z′=σ(W3z+b3)

[0098] Where z′ represents the second latent feature, W3 is the weight matrix, and b3 is the bias vector.

[0099] Back propagation: Reconstruct the second latent feature z′ into the second output feature z″. The specific expression is as follows:

[0100] z″=σ(W4z′+b4)

[0101] Where z″ represents the second output feature, W4 is the weight matrix, and b4 is the bias vector.

[0102] Reconstruction error: Calculate the first error between the second input feature z and the second output feature z″. The mean square error (MSE) is usually used. The specific expression is as follows:

[0103] L(z,z″)=||zz″|| 2

[0104] Where L(z,z″) represents the second error.

[0105] Similarly, in order to minimize the second error, the backpropagation algorithm can be used to adjust the weight matrix (W3, W4) of the second autoencoder, and gradually optimize the model parameters until the model can effectively reduce the dimension and reconstruct the input features.

[0106] S3: Obtain target latent features of the stacked autoencoder model, and train an initial neural network model based on the target latent features to obtain a deep neural network model, wherein the target latent features are used to represent the latent features in the last layer of the autoencoder of the stacked autoencoder model.

[0107] In the embodiment of the present application, the second latent feature obtained by training the second autoencoder is used as the target latent feature to train the DNN model. Through multiple layers of abstraction and learning, the DNN model can further extract high-level patterns and classify them, ultimately outputting the type of traffic.

[0108] Optionally, an initial neural network model is trained based on the target latent feature to obtain a deep neural network model, including: determining the initial neural network model, wherein the initial neural network model includes multiple fully connected layers; determining the second latent feature as the third input feature of the initial neural network model, and processing the third input feature through the initial neural network model to obtain a third output feature; determining a third error between the third output feature and the corresponding traffic label through a cross entropy loss function, and optimizing the third error by adjusting the model weights corresponding to the initial neural network model to obtain a deep neural network model.

[0109] In the embodiment of the present application, the initial neural network model is designed as an architecture containing multiple fully connected layers, aiming to further abstract and learn the deep patterns in the second latent features through multi-layer nonlinear transformations to distinguish different traffic types. The details are as follows:

[0110] First fully connected layer: set the number of units (e.g. 1024), use the ReLU activation function, and apply batch normalization and Dropout regularization.

[0111] Second fully connected layer: set the number of units (e.g. 512), use the ReLU activation function, and apply batch normalization and Dropout regularization.

[0112] Third fully connected layer: Set the number of units (e.g. 128), use the ReLU activation function, and apply batch normalization and Dropout regularization.

[0113] Output layer: Set the number of nodes to be equal to the number of classified traffic categories y, and use the softmax activation function to output the probability of each category.

[0114] In this embodiment of the present application, the cross entropy loss function can be used to calculate the difference between the third output feature and the actual traffic label, that is, the third error. The cross entropy loss function can quantify the difference between the model's predicted probability distribution and the true label probability distribution, and is a commonly used loss function in multi-classification tasks. The specific expression is as follows:

[0115]

[0116] Where, represents the third error, y represents the unique hot encoding of the traffic label, It represents the output probability predicted by the model, that is, the third output feature mentioned above; o is the sample index, and c is the traffic category index.

[0117] In order to optimize the classification performance of the DNN model, a backpropagation algorithm can also be used, such as using the SGD (stochastic gradient descent) optimizer to adjust the model weights to minimize the third error. The specific expression is as follows:

[0118]

[0119] Where w t Indicates the current model weight; w t+1 Represents the optimized model weight; η is the learning rate, which is used to control the step size of each update; Indicates the current model weight w t The gradient of the loss function L.

[0120] In the embodiment of the present application, a test set can also be used to evaluate the model performance, calculate performance indicators such as accuracy, recall rate, F1 score, etc., adjust the model parameters and training strategy according to the verification results, and further optimize the model.

[0121] In summary, by inputting the second latent features generated by the SAE model into the DNN model and combining it with the cross-entropy loss function for model training and optimization, we achieve efficient detection and classification of network traffic anomalies. This process leverages the feature extraction advantages of SAE and the powerful classification capabilities of DNN, providing an automated and highly accurate solution for network security protection.

[0122] S4: Determine the traffic detection model based on the stacked autoencoder and deep neural network model.

[0123] In the embodiment of the present application, after the SAE model and DNN model are trained separately and their performance is verified, the two models are combined to form the final abnormal traffic detection model (SAE-DNN model). The SAE-DNN model can receive real-time network traffic data, first perform feature extraction and dimensionality reduction through stacked autoencoders, and then perform classification through deep neural networks to quickly and accurately identify abnormal traffic.

[0124] The integration of the SAE-DNN model ensures the automation and intelligence of the entire process from data preprocessing to feature extraction and final classification, thereby improving the efficiency and accuracy of abnormal traffic detection, reducing the risk of false positives and missed reports, and providing a powerful tool for network security management.

[0125] Figure 4 This is a classification diagram of a flow detection model according to an embodiment of the present application. Figure 4 As shown, the SAE-DNN model includes the SAE model and the DNN model. The SAE model includes at least two autoencoder layers. The first encoder is used to transform the input features (y1, y2, y3, ..., y d ) is mapped to the first latent feature (z1,z2,…,z m ), the second autoencoder is used to map the first latent feature to the second latent feature (z'1, z'2, ..., z' n The DNN model uses the second latent feature as the model input and performs classification and recognition through a three-layer network structure (DNN-1, DNN-2, and DNN-3, each layer of which includes Dropout regularization, batch normalization BatchNorm, and a fully connected layer FCLayer), outputting the final traffic recognition results, such as normal traffic and abnormal traffic (including but not limited to SQL injection, botnets, Trojan viruses, etc.).

[0126] In the embodiment of the present application, the SAE model is used to perform layer-by-layer feature dimensionality reduction and extraction, which can not only effectively reduce the data dimension and remove redundant information, but also mine the key features of abnormal traffic, providing high-quality input for subsequent classification. Subsequently, the powerful classification ability of the DNN model is utilized to accurately distinguish the type of abnormal traffic based on the implicit features output by the SAE model. The present application not only overcomes the limitations of traditional detection methods, such as the static nature of rule matching and the limitations of statistical analysis, but also greatly improves the accuracy and response speed of detection, significantly reduces the false alarm rate, and shows outstanding results, especially when processing high-dimensional and complex network traffic data.

[0127] According to an embodiment of the present application, a flow detection device is provided. It should be noted that the flow detection device of the embodiment of the present application can be used to execute the flow detection method provided in the embodiment of the present application. The flow detection device provided in the embodiment of the present application is introduced below.

[0128] Figure 5 This is a structural diagram of a flow detection device provided according to an embodiment of the present application. Figure 5 As shown, the device includes:

[0129] An acquisition module 50 is used to acquire network traffic data;

[0130] An extraction module 52 is configured to extract features from network traffic data using a stacked autoencoder model in a traffic detection model, wherein the stacked autoencoder model includes multiple autoencoders, each of which includes an encoder for mapping input features to implicit features and a decoder for restoring the implicit features, where the implicit features are traffic features obtained by performing feature dimensionality reduction on the input features;

[0131] The classification module 54 is used to classify the network traffic data after feature extraction using the deep neural network model in the traffic detection model to obtain a traffic identification result. The traffic identification result is used to reflect the traffic type to which the network traffic data belongs.

[0132] Through the acquisition module, extraction module and classification module in the above-mentioned traffic detection device, the purpose of accurately identifying normal and abnormal traffic types is achieved, thereby realizing the technical effects of improving network security protection efficiency, reducing false alarm rate and missed alarm rate, and enhancing the adaptive detection capability of new attacks, thereby solving the technical problem of low network security protection efficiency caused by low detection accuracy, high false alarm rate and poor adaptability to new attacks in the abnormal traffic detection method in related technologies.

[0133] In the traffic detection device provided in the embodiment of the present application, the classification module is also used to extract metadata information corresponding to the abnormal traffic when the traffic identification result indicates that there is abnormal traffic in the network traffic data, wherein the metadata information includes at least one of the following: the source IP, destination IP and corresponding abnormal traffic type of the abnormal traffic; and generate an abnormal traffic report based on the metadata information.

[0134] The traffic detection device provided in the embodiment of the present application also includes a training module 56, which is used to obtain historical traffic logs, wherein the historical traffic logs include normal traffic and different types of abnormal traffic; train the initial autoencoder model based on the historical traffic logs to obtain a stacked autoencoder model; obtain the target implicit features of the stacked autoencoder model, and train the initial neural network model based on the target implicit features to obtain a deep neural network model, wherein the target implicit features are used to represent the implicit features in the last layer of the autoencoder of the stacked autoencoder model; and determine the traffic detection model based on the stacked autoencoder and the deep neural network model.

[0135] In the traffic detection device provided in the embodiment of the present application, the training module is also used to perform data analysis on the traffic log to obtain field data, wherein the field data includes text items and numerical items; the field data is preprocessed to obtain a data set for training the initial autoencoder model, wherein the preprocessing is used to convert the text information in the field data into numerical information, and each traffic data in the data set has the same feature dimension and corresponding traffic label, and the traffic label is used to reflect the traffic type to which each traffic data belongs.

[0136] In the traffic detection device provided in the embodiment of the present application, the training module is also used to replace the space symbols in the field data with preset symbols; replace the abnormal traffic types in the field data with digital identifiers; extract the traffic labels in the field data, binary encode the text items in the remaining field data, and one-hot encode the traffic labels.

[0137] In the flow detection device provided in an embodiment of the present application, the training module is also used to obtain an initial autoencoder model, wherein the initial autoencoder model includes at least a first autoencoder and a second autoencoder; the first autoencoder is trained using a data set to obtain a first latent feature, wherein the first latent feature is used to represent the flow feature obtained after preliminary feature dimensionality reduction by the first autoencoder; the second autoencoder is trained using the first latent feature to obtain a second latent feature, wherein the second latent feature is used to represent the flow feature obtained after secondary feature dimensionality reduction by the second autoencoder; the trained first autoencoder and the second autoencoder are fused to obtain a stacked autoencoder model.

[0138] In the flow detection device provided in an embodiment of the present application, the training module is also used to determine a first input feature corresponding to a first autoencoder from a data set; map the first input feature to a first latent feature through forward propagation, wherein the feature dimension of the first latent feature is smaller than the feature dimension of the first input feature; reconstruct the first latent feature into a first output feature through back propagation, wherein the feature dimension of the first output feature is the same as the feature dimension of the first input feature; determine a first error between the first input feature and the first output feature, and optimize the first error by adjusting the weight matrix corresponding to the first autoencoder.

[0139] In the flow detection device provided in an embodiment of the present application, the training module is also used to determine the first implicit feature as the second input feature of the second autoencoder; map the second input feature to the second implicit feature through forward propagation, wherein the feature dimension of the second implicit feature is smaller than the feature dimension of the second input feature; reconstruct the second implicit feature into the second output feature through back propagation, wherein the feature dimension of the second output feature is the same as the feature dimension of the second input feature; determine the second error between the second input feature and the second output feature, and optimize the second error by adjusting the weight matrix corresponding to the second autoencoder.

[0140] In the traffic detection device provided in an embodiment of the present application, the training module is also used to determine an initial neural network model, wherein the initial neural network model includes multiple fully connected layers; the second latent feature is determined as the third input feature of the initial neural network model, and the third input feature is processed by the initial neural network model to obtain a third output feature; the third error between the third output feature and the corresponding traffic label is determined by a cross entropy loss function, and the third error is optimized by adjusting the model weight corresponding to the initial neural network model to obtain a deep neural network model.

[0141] An embodiment of the present application further provides an electronic device, comprising: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned flow detection method.

[0142] It should be noted that the above electronic equipment is used to perform Figure 2 The flow detection method shown in the figure, therefore the relevant explanations in the above flow detection method are also applicable to the electronic device and will not be repeated here.

[0143] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned traffic detection method by running the computer program.

[0144] It should be noted that the above non-volatile storage medium is used to execute Figure 2The flow detection method shown in the figure, therefore the relevant explanations in the above flow detection method are also applicable to the non-volatile storage medium and will not be repeated here.

[0145] An embodiment of the present application also provides a computer program product, including computer instructions, which implement the above-mentioned traffic detection method when executed by a processor.

[0146] It should be noted that the above-mentioned computer program product is used to execute Figure 2 The flow detection method shown in the figure, therefore the relevant explanations in the above flow detection method are also applicable to the computer program product and will not be repeated here.

[0147] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0148] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0149] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0150] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0151] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0153] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A flow detection method, characterized in that: include: Get network traffic data; Using a stacked autoencoder model in a traffic detection model to extract features from the network traffic data, wherein the stacked autoencoder model includes multiple autoencoders, each autoencoder includes an encoder for mapping input features to implicit features and a decoder for restoring the implicit features, wherein the implicit features are traffic features after feature dimensionality reduction of the input features; The deep neural network model in the traffic detection model is used to classify the network traffic data after feature extraction to obtain a traffic identification result, which is used to reflect the traffic type to which the network traffic data belongs.

2. The method according to claim 1, characterized in that The traffic detection model is trained in the following way: Obtaining historical traffic logs, wherein the historical traffic logs include normal traffic and different types of abnormal traffic; Training an initial autoencoder model according to the historical traffic log to obtain the stacked autoencoder model; Obtaining target latent features of the stacked autoencoder model, and training an initial neural network model based on the target latent features to obtain the deep neural network model, wherein the target latent features are used to represent the latent features in the last layer of the autoencoder of the stacked autoencoder model; The traffic detection model is determined based on the stacked autoencoder and the deep neural network model.

3. The method according to claim 2, characterized in that After obtaining the historical traffic log, the method further includes: Performing data parsing on the traffic log to obtain field data, wherein the field data includes text items and numerical items; The field data is preprocessed to obtain a data set for training the initial autoencoder model, wherein the preprocessing is used to convert text information in the field data into numerical information, and each traffic data in the data set has the same feature dimension and corresponding traffic label, and the traffic label is used to reflect the traffic type to which each traffic data belongs.

4. The method according to claim 3, characterized in that Preprocessing the field data includes: Replace the space symbols in the field data with preset symbols; Replacing the abnormal traffic type in the field data with a digital identifier; The traffic labels in the field data are extracted, the text items in the remaining field data are binary-encoded, and the traffic labels are one-hot-encoded.

5. The method according to claim 3, characterized in that Training an initial autoencoder model based on the historical traffic log to obtain the stacked autoencoder model includes: Acquire the initial autoencoder model, wherein the initial autoencoder model includes at least a first autoencoder and a second autoencoder; Training the first autoencoder using the data set to obtain a first latent feature, wherein the first latent feature is used to represent a traffic feature obtained after preliminary feature dimensionality reduction by the first autoencoder; The second autoencoder is trained using the first latent feature to obtain a second latent feature, wherein the second latent feature is used to represent the traffic feature obtained after secondary feature dimensionality reduction is performed by the second autoencoder; The first autoencoder and the second autoencoder after training are integrated to obtain the stacked autoencoder model.

6. The method according to claim 5, characterized in that Training the first autoencoder using the data set includes: Determining a first input feature corresponding to the first autoencoder from the dataset; Mapping the first input feature to the first latent feature through forward propagation, wherein the feature dimension of the first latent feature is smaller than the feature dimension of the first input feature; Reconstructing the first latent feature into a first output feature through back propagation, wherein the feature dimension of the first output feature is the same as the feature dimension of the first input feature; A first error between the first input feature and the first output feature is determined, and the first error is optimized by adjusting a weight matrix corresponding to the first autoencoder.

7. The method according to claim 5, characterized in that Training the second autoencoder using the first latent feature includes: Determining the first latent feature as a second input feature of the second autoencoder; Mapping the second input feature to the second latent feature through forward propagation, wherein the feature dimension of the second latent feature is smaller than the feature dimension of the second input feature; Reconstructing the second latent feature into a second output feature through back propagation, wherein the feature dimension of the second output feature is the same as the feature dimension of the second input feature; A second error between the second input feature and the second output feature is determined, and the second error is optimized by adjusting a weight matrix corresponding to the second autoencoder.

8. The method according to claim 5, characterized in that Training an initial neural network model based on the target implicit features to obtain the deep neural network model includes: Determining the initial neural network model, wherein the initial neural network model includes a plurality of fully connected layers; Determining the second latent feature as a third input feature of the initial neural network model, and processing the third input feature through the initial neural network model to obtain a third output feature; A third error between the third output feature and the corresponding traffic label is determined by a cross entropy loss function, and the third error is optimized by adjusting the model weights corresponding to the initial neural network model to obtain the deep neural network model.

9. The method according to claim 1, characterized in that The method further comprises: When the traffic identification result indicates that the network traffic data has abnormal traffic, extracting metadata information corresponding to the abnormal traffic, wherein the metadata information includes at least one of the following: a source IP address, a destination IP address, and a corresponding abnormal traffic type of the abnormal traffic; Generate an abnormal traffic report based on the metadata information.

10. A flow detection device, characterized in that: include: Acquisition module, used to obtain network traffic data; an extraction module, configured to extract features from the network traffic data using a stacked autoencoder model in a traffic detection model, wherein the stacked autoencoder model includes a plurality of autoencoders, each of which includes an encoder for mapping input features to implicit features and a decoder for restoring the implicit features, wherein the implicit features are traffic features obtained by performing feature dimensionality reduction on the input features; A classification module is used to classify the network traffic data after feature extraction using the deep neural network model in the traffic detection model to obtain a traffic identification result, which is used to reflect the traffic type to which the network traffic data belongs.

11. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; The processor is connected to the memory and is used to execute the flow detection method described in any one of claims 1 to 9.

12. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the flow detection method according to any one of claims 1 to 9 by running the computer program.

13. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the flow detection method according to any one of claims 1 to 9 is implemented.

Images