Multi-dimensional network anomaly monitoring method and device and storage medium
Through the multi-dimensional network anomaly monitoring method, combined with neural network model and rare classification technology, the shortcomings of traditional methods in the face of unknown attacks and rare pattern recognition are solved, and more efficient and accurate network traffic anomaly detection is achieved.
Patent Information
- Application Number
- CN202411530924.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-30
- Publication Date
- 2025-02-28
AI Technical Summary
Traditional network traffic anomaly detection methods have problems such as inability to effectively deal with unknown attacks or variant attacks, difficulty in adapting to changes in network security threats, high false alarm rates and missed alarm rates, high computing resources consumption when processing large-scale data, low real-time and efficiency, and insufficient ability to identify rare network traffic patterns.
The multi-dimensional network anomaly monitoring method is used to identify and classify the data segments by obtaining network traffic data, extracting feature vectors and inputting them into neural network models. Combining rare classification technology and local density judgment, the data segments are identified and classified.
It improves the accuracy and efficiency of network traffic anomaly detection, can more effectively identify rare categories, reduce false alarms and missed alarms, improves the system's adaptability to new attacks, and saves computing resource consumption.
Smart Images
Figure CN119544268A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of network security, and in particular to a multi-dimensional network anomaly monitoring method, device and storage medium. Background Art
[0002] With the rapid development of information technology and the popularization of the Internet, network security faces increasingly complex and diverse threats. Traditional network traffic anomaly detection methods mainly include rule-based detection and statistical methods. These methods can identify known attack patterns to a certain extent, but they also have significant limitations:
[0003] Limitations of static rules: Detection methods based on fixed rules can only deal with known attack patterns, and lack effective response strategies for unknown attacks or variant attacks. This static rule-based approach is difficult to adapt to the increasingly changing and complex trend of network security threats. For example, common signature-based detection systems need to update their databases to include new attack patterns, but this update process may be delayed, making it impossible for the system to identify and prevent new attacks in a timely manner.
[0004] Statistical methods that rely on specific thresholds: Traditional statistical methods usually rely on pre-set thresholds to judge abnormal behavior. This static threshold setting is often difficult to cope with the dynamic changes and seasonal fluctuations of network traffic, resulting in a high false positive rate or false negative rate. For example, some statistical methods may mistake normal network traffic fluctuations as abnormal activities, causing unnecessary interference or misjudgment.
[0005] Large data volume and slow processing speed: With the rapid increase in network traffic data volume, traditional methods often face the problem of large computing resource consumption and slow processing speed when processing large-scale data, affecting real-time performance and efficiency. This problem is particularly prominent when deep data analysis or complex algorithm calculations are required, limiting the detection system's ability to respond to large-scale attacks or high traffic loads.
[0006] Insufficient recognition of rare classifications: Traditional methods perform poorly in identifying rare or uncommon network traffic patterns. These rare patterns may contain the characteristics of new attacks, which are difficult for traditional methods to effectively detect and identify. For example, some advanced threats may use rare network traffic patterns to avoid the recognition of traditional detection systems, which requires new detection technologies to be more flexible in adapting to and identifying these changing attack patterns.
[0007] In addition, since rare categories have low prior probabilities and are likely to overlap with common categories, this overlap can easily cause rare samples to be classified into common categories, ultimately resulting in the inability to effectively detect network attacks and anomalies. Summary of the invention
[0008] The purpose of the present invention is to provide a multi-dimensional network anomaly monitoring method, device and storage medium.
[0009] The purpose of the present invention can be achieved by the following technical solutions:
[0010] A multi-dimensional network anomaly monitoring method, comprising:
[0011] Get network traffic data;
[0012] Based on network traffic data collection, feature vectors of each data segment are extracted;
[0013] Inputting the extracted feature vector into the first model to obtain a first classification result;
[0014] Based on the extracted feature vectors, determine whether the data segments belong to any rare category one by one. If the data segments belong to any rare category, the rare category is used as the second classification result;
[0015] If the data segment has a second classification result, the second classification result is used as the final classification result; otherwise, the first classification result is used as the final classification result;
[0016] Output monitoring results based on the final classification results.
[0017] The process of determining whether the data segment belongs to any rare category includes:
[0018] Select a rare category i as the target category and obtain the number of known data segments Ki of the target category;
[0019] Calculate the distance between the data segment to be classified and the Ki-th nearest neighbor as the first distance, and calculate the local density of the data segment to be classified using the first distance as the radius;
[0020] Neighbor search step: searching neighbor data segments near the data segment to be classified as first data segments with the initial neighbor radius, and calculating the local density of each first data segment respectively;
[0021] A first data segment is selected based on the local density of the data segment to be classified and all the first data segments, and it is determined whether the first data segment belongs to the target category. If so, the data segment to be classified is determined to be the target category. Otherwise, the first neighbor radius is increased and the process returns to the neighbor search step until the neighborhood radius increases to a preconfigured radius threshold, and then it is determined that the data segment to be classified is not the target category.
[0022] The selecting a first data segment based on the local densities of the data segment to be classified and all the first data segments specifically includes: selecting a first data segment having a local density closest to the data segment to be classified from among all the first data segments.
[0023] When it is determined that the data segment to be classified is not the target category, another rare category that has not been traversed is selected as the target category, and the data segment is re-determined whether it belongs to the target category until the data segment to be classified belongs to any rare category or the traversal of all rare categories is completed.
[0024] In the process of determining whether the first data segment belongs to the target category, if the first data segment belongs to the target category, the data segment near the first data segment is excluded with the second distance as the radius.
[0025] The second distance is determined based on a local density of the first data segment.
[0026] The feature vector is extracted from at least the following information:
[0027] Basic features, including source and destination IP addresses, source and destination ports, protocol type, transport layer data size, and timestamp;
[0028] Traffic statistics, including the number of packets, number of bytes, and size of packets;
[0029] Transport layer characteristics, including packet latency, packet transmission rate, and connection establishment and closing times;
[0030] Application layer characteristics, including HTTP request method, HTTP response status code, and DNS query type and response time;
[0031] The time series characteristics of network traffic, including the volatility of traffic within a certain period of time and the periodicity or regularity of time series data;
[0032] Structural characteristics of network traffic, including the data structure of packets and the payload content of packets;
[0033] Traffic statistics, including packet arrival rate, packet sending rate, and packet transmission rate;
[0034] Network behavior characteristics, including the duration of network connections and the frequency and pattern of service requests.
[0035] The first model is a neural network model.
[0036] A multi-dimensional network anomaly monitoring device includes a memory, a processor, and a program stored in the memory, and the processor implements the above method when executing the program.
[0037] A storage medium stores a program, which implements the above method when executed.
[0038] Compared with the prior art, the present invention has the following beneficial effects:
[0039] 1. Based on the classification of the first model, a rare classification method is also introduced. When any sample to be classified is determined to be a rare category, the final result adopts the rare category, which can effectively solve the problem of overlap between rare categories and regular categories and improve the classification accuracy.
[0040] 2. Whether it belongs to the target category is determined by local density, which solves the problem of low prior probability of rare categories and can improve the probability and accuracy of mining rare categories.
[0041] 3. By improving the rare classification method, the surrounding samples are excluded after being positioned as the target category, thus solving the problem of slow rare classification speed and improving the operating efficiency of the computer system. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 Schematic diagram of the main steps of this method. DETAILED DESCRIPTION
[0043] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0044] A multi-dimensional network anomaly monitoring method, such as Figure 1 As shown, including:
[0045] (1) Obtaining network traffic data;
[0046] This process mainly obtains raw network traffic data in real time from network devices or traffic capture devices.
[0047] In almost all embodiments, the acquired network traffic data needs to be preprocessed and cleaned to ensure the accuracy and completeness of the data. Since the preprocessing and data cleaning process is a common practice in the field, this application does not make any improvements to this part. In order to avoid blurring the purpose of this application, this application will not elaborate on it.
[0048] (ii) extracting feature vectors of each data segment based on network traffic data collection;
[0049] During preprocessing and cleaning, feature vectors of data segments are extracted, where a data segment can be a separate data packet or several aggregated data packets.
[0050] The extraction of feature vectors depends on certain features of the data segment. Generally, the dimensions of features considered in the prior art are incomplete. In this embodiment, feature vectors are extracted from features of multiple dimensions, including basic features, traffic statistical features, transport layer features, application layer features, timing features of network traffic, structural features of network traffic, traffic statistical features, and network behavior features, so that the information in the data segment can be used more deeply. The specific features of each dimension are as follows:
[0051] Basic features, including source and destination IP addresses, source and destination ports, protocol type, transport layer data size, and timestamp;
[0052] Traffic statistics, including the number of packets, number of bytes, and size of packets;
[0053] Transport layer characteristics, including packet latency, packet transmission rate, and connection establishment and closing times;
[0054] Application layer characteristics, including HTTP request method, HTTP response status code, and DNS query type and response time;
[0055] The time series characteristics of network traffic, including the volatility of traffic within a certain period of time and the periodicity or regularity of time series data;
[0056] Structural characteristics of network traffic, including the data structure of packets and the payload content of packets;
[0057] Traffic statistics, including packet arrival rate, packet sending rate, and packet transmission rate;
[0058] Network behavior characteristics, including the duration of network connections and the frequency and pattern of service requests.
[0059] However, since this application considers more feature information, the problem of too low prior probability of rare categories and overlap with conventional categories is further aggravated. Therefore, this application needs to solve this problem through rare classification technology.
[0060] (3) inputting the extracted feature vector into the first model to obtain a first classification result;
[0061] This process uses advanced data mining and machine learning techniques to extract and select features from pre-processed network traffic data.
[0062] In some embodiments, the first model is a neural network model, and its autoencoder structure is designed as follows:
[0063] Encoder: Use a multi-layer neural network (such as multiple fully connected layers) to map the input features to a low-dimensional hidden representation. Consider using ReLU as the activation function.
[0064] Decoder: Decodes the encoded hidden representation and reconstructs it into data that is as similar as possible to the original input. The structure of the decoder layer is symmetrical to that of the encoder, and ultimately outputs reconstructed data with the same dimension as the original input data.
[0065] Training strategy:
[0066] Normal network traffic data is used for training. The goal is to minimize the reconstruction error between the encoder and decoder, usually using the mean squared error (MSE) as the loss function.
[0067] During training, stochastic gradient descent (SGD) or adaptive optimizers such as Adam can be used to optimize model parameters.
[0068] (iv) based on the extracted feature vectors, determining whether the data segments belong to any rare category one by one; if the data segments belong to any rare category, taking the rare category as the second classification result;
[0069] The process of determining whether a data segment belongs to any rare category includes:
[0070] Select a rare category i as the target category and obtain the number of known data segments Ki of the target category;
[0071] Calculate the distance between the data segment to be classified and the Ki-th nearest neighbor as the first distance, and calculate the local density of the data segment to be classified using the first distance as the radius;
[0072] Neighbor search step: searching neighbor data segments near the data segment to be classified as first data segments with the initial neighbor radius, and calculating the local density of each first data segment respectively;
[0073] A first data segment is selected based on the local density of the data segment to be classified and all the first data segments, and it is determined whether the first data segment belongs to the target category. If so, the data segment to be classified is determined to be the target category. Otherwise, the first neighbor radius is increased and the process returns to the neighbor search step until the neighborhood radius increases to a preconfigured radius threshold, and then it is determined that the data segment to be classified is not the target category.
[0074] In some embodiments, selecting a first data segment based on the local density of the data segment to be classified and all the first data segments specifically includes: selecting a first data segment that is closest to the local density of the data segment to be classified among all the first data segments.
[0075] When it is determined that the data segment to be classified is not the target category, another rare category that has not been traversed is selected as the target category, and the data segment is re-determined whether it belongs to the target category until the data segment to be classified belongs to any rare category or the traversal of all rare categories is completed.
[0076] In particular, in some embodiments, in order to speed up the rare classification rate, in the process of determining whether the first data segment belongs to the target category, if the first data segment belongs to the target category, the data segments near the first data segment are excluded with the second distance as the radius.
[0077] In this embodiment, the second distance is determined based on the local density of the first data segment, so that dynamic adaptive adjustment can be performed without causing a drop in accuracy due to short-term fluctuations of a rare category.
[0078] In fact, in some embodiments, when determining rare categories, the rare categories are used as iteration conditions to first determine whether each data segment to be classified belongs to a rare category, and then determine whether the non-classified data segment belongs to the next rare category, and so on.
[0079] (5) If the data segment has a second classification result, the second classification result is used as the final classification result; otherwise, the first classification result is used as the final classification result;
[0080] (6) Output monitoring results based on the final classification results.
[0081] This part can be combined with some established decision-making mechanisms to output corresponding logs and make decision responses based on the detection results.
[0082] The application of intelligent algorithms and dynamic adjustment capabilities enable the system to use computing resources more efficiently when processing large-scale data, reducing energy costs. For example, compared with traditional methods, this system can save about 30% of computing resource consumption when processing unit data volume. The application of rare classification technology improves the system's adaptability to new attacks, reduces false positives and false negatives, and enables security managers to manage and respond to complex network security incidents more easily.
[0083] In summary, this technical solution significantly improves the effectiveness and efficiency of the network traffic anomaly detection system by introducing innovations in dynamic adaptability, rare classification technology and intelligent algorithms, brings about significant productivity improvements and resource savings, and improves the convenience of operation and user experience.
[0084] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.
Claims
1. A multi-dimensional network anomaly monitoring method, characterized in that: include: Get network traffic data; Based on network traffic data collection, feature vectors of each data segment are extracted; Inputting the extracted feature vector into the first model to obtain a first classification result; Based on the extracted feature vectors, determine whether the data segments belong to any rare category one by one. If the data segments belong to any rare category, the rare category is used as the second classification result; If the data segment has a second classification result, the second classification result is used as the final classification result; otherwise, the first classification result is used as the final classification result; Output monitoring results based on the final classification results.
2. A multi-dimensional network anomaly monitoring method according to claim 1, characterized in that: The process of determining whether the data segment belongs to any rare category includes: Select a rare category i as the target category and obtain the number of known data segments Ki of the target category; Calculate the distance between the data segment to be classified and the Ki-th nearest neighbor as the first distance, and calculate the local density of the data segment to be classified using the first distance as the radius; Neighbor search step: searching neighbor data segments near the data segment to be classified as first data segments with an initial neighbor radius, and calculating the local density of each first data segment respectively; A first data segment is selected based on the local density of the data segment to be classified and all the first data segments, and it is determined whether the first data segment belongs to the target category. If so, the data segment to be classified is determined to be the target category. Otherwise, the first neighbor radius is increased and the process returns to the neighbor search step until the neighborhood radius increases to a preconfigured radius threshold, and then it is determined that the data segment to be classified is not the target category.
3. A multi-dimensional network anomaly monitoring method according to claim 2, characterized in that: The selecting a first data segment based on the local densities of the data segment to be classified and all the first data segments specifically includes: selecting a first data segment having a local density closest to the data segment to be classified from among all the first data segments.
4. A multi-dimensional network anomaly monitoring method according to claim 2, characterized in that: When it is determined that the data segment to be classified is not the target category, another rare category that has not been traversed is selected as the target category, and the data segment is re-determined whether it belongs to the target category until the data segment to be classified belongs to any rare category or the traversal of all rare categories is completed.
5. A multi-dimensional network anomaly monitoring method according to claim 2, characterized in that: In the process of determining whether the first data segment belongs to the target category, if the first data segment belongs to the target category, the data segment near the first data segment is excluded with the second distance as the radius.
6. A multi-dimensional network anomaly monitoring method according to claim 5, characterized in that: The second distance is determined based on a local density of the first data segment.
7. A multi-dimensional network anomaly monitoring method according to claim 1, characterized in that: The feature vector is extracted from at least the following information: Basic features, including source and destination IP addresses, source and destination ports, protocol type, transport layer data size, and timestamp; Traffic statistics, including the number of packets, number of bytes, and size of packets; Transport layer characteristics, including packet latency, packet transmission rate, and connection establishment and closing times; Application layer characteristics, including HTTP request method, HTTP response status code, and DNS query type and response time; The time series characteristics of network traffic, including the volatility of traffic within a certain period of time and the periodicity or regularity of time series data; Structural characteristics of network traffic, including the data structure of packets and the payload content of packets; Traffic statistics, including packet arrival rate, packet sending rate, and packet transmission rate; Network behavior characteristics, including the duration of network connections and the frequency and pattern of service requests.
8. A multi-dimensional network anomaly monitoring method according to claim 1, characterized in that: The first model is a neural network model.
9. A multi-dimensional network anomaly monitoring device, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.
10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 8 is implemented.