A method for detecting abnormal traffic in closed networks under resource-constrained scenarios

Abstract

The present invention discloses a method for detecting abnormal traffic in a closed network under a resource-constrained scenario, and belongs to the field of cyberspace security technology. The method of the present invention comprises: pre-processing network traffic to obtain traffic samples; calculating local sensitive hash values ​​for samples in sequence to obtain sample summaries; setting and maintaining a sliding window of fixed size for storing K sample summaries of the latest benign network traffic; pre-processing the network traffic to obtain a sample to be tested, calculating the summary of the sample to be tested, calculating the similarity with the K sample summaries in the sliding window, using the k-nearest neighbor strategy to calculate the abnormal score of the sample to be tested, and determining whether it is abnormal traffic. The method of the present invention does not rely on artificial intelligence models, does not require data labeling and feature extraction, can achieve good detection effects with lower computing resource overhead, and significantly reduces the false alarm rate and missed alarm rate compared to existing detection technologies based on local sensitive hash functions.

Description

Technical Field

[0001] The present invention belongs to the field of cyberspace security technology, and specifically relates to a method for detecting abnormal traffic in a closed network under resource-constrained scenarios. Background Art

[0002] In today's rapidly developing information technology world, networks have become deeply integrated into every aspect of social production. Technological developments, such as the Internet of Things (IoT), have made specialized cyber-physical systems, such as networked medical devices and intelligent electricity metering terminals, closely connected to cyberspace. Cybersecurity issues are particularly prominent in specialized systems. Abnormal traffic, such as cyberattacks, can severely impact the stability of network systems and the integrity of data. Researchers have proposed various methods for detecting abnormal traffic, including rule-based matching, supervised learning models, and unsupervised learning models. However, rule-based matching and supervised learning models lack the ability to detect unknown types of anomalies. Given that the characteristics of benign traffic are relatively stable and unlike the diverse types of abnormal traffic, unsupervised methods based solely on benign traffic have attracted considerable research attention. Unsupervised anomaly detection techniques detect anomalies based on the degree to which the sample under test deviates from a baseline of benign traffic. Their training process does not rely on labeled malicious samples, resulting in strong generalization capabilities.

[0003] Existing typical unsupervised anomaly detection technologies include: reconstruction-based method Autoencoder (Reference 1: J.Zhai, S.Zhang, J.Chen and Q.He, "Autoencoder and Its Various Variants," 2018IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan, 2018, pp.415-419, doi:10.1109 / SMC.2018.00080.), distance metric-based method KNN (Reference 2: Song S M. Research on Computer Software Security Detection Technology Based on KNN Algorithm [J]. Journal of Engineering and Technology Management (JETM), 2024, 1 (1).), and empirical cumulative distribution-based method ECOD (Reference 3: Z.Li, Y.Zhao, X.Hu, N.Botta, C.Ionescu and GHChen, "ECOD: Unsupervised Outlier Detection Using Empirical Cumulative Distribution Functions," in IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 12, pp. 12181-12193, 1 Dec. 2023, doi: 10.1109 / TKDE.2022.3159580.); the partition-based method IForest (reference 4: FT Liu, KMTing and Z.-H. Zhou, "Isolation Forest," 2008 Eighth IEEE International Conference on Data Mining, Pisa, Italy, 2008, pp. 413-422, doi: 10.1109 / ICDM.2008.17.), the single-class hyperplane-based method OCSVM (reference 5: A. Bounsiar and M.G.Madden, "One-Class Support Vector Machines Revisited," 2014 International Conference on Information Science & Applications (ICISA), Seoul, Korea (South), 2014, pp. 1-4, doi:10.1109 / ICISA.2014.6847442.); and the clustering-based method CBLOF (Reference 6: He Z, Xu X, Deng S. Discovering cluster-based local outliers [J]. Pattern recognition letters, 2003, 24(9-10): 1641-1650.). However, these existing unsupervised detection techniques all have limitations. Reconstruction-based methods are implemented using deep neural networks and are computationally expensive. Furthermore, when training data is insufficient or of low quality, they are prone to overfitting, which affects detection performance. Distance-based methods require calculating distances between large amounts of data, which consumes a lot of computing resources and is not suitable for high-dimensional data, potentially leading to the "curse of dimensionality." Methods based on the empirical cumulative distribution make prior assumptions about the distribution of benign samples. Noise in the data can lead to unstable results and affect detection accuracy. Methods based on partitioning, single-class hyperplanes, and clustering are computationally intensive and perform poorly in high-dimensional spaces due to factors such as data sparsity.

[0004] However, a greater limitation lies in its reliance on hardware resources. Existing unsupervised technologies are mostly based on machine learning or deep learning models, which place certain demands on computing resources such as processors and memory. However, in specialized IoT environments, such as the Industrial IoT and telemedicine, limited hardware resources must be prioritized to ensure the stable operation of business functions. Business activities are constrained by limited hardware resources, and are even forced to use simple communication protocols and minimize message payloads. In this scenario, the processing and memory resources available for anomaly traffic detection models are very limited, which limits their deployment and operation. Some researchers have proposed using methods based on locality-sensitive hashing functions to detect anomalies in resource-constrained scenarios. However, the performance of these methods in real-world IoT traffic has not lived up to their claims, and there is still considerable room for improvement.

[0005] In general, the shortcomings of existing unsupervised anomaly detection methods and existing abnormal traffic detection technologies based on locality-sensitive hash functions in resource-constrained scenarios are mainly reflected in the following aspects:

[0006] (1) High deployment overhead. Some typical unsupervised learning abnormal traffic detection methods rely on artificial intelligence models and require specialized software and hardware support, such as graphics cards and CUDA parallel computing libraries. In resource-constrained scenarios, the conditions for deploying these software and hardware may be insufficient. At the same time, the introduction of these software and hardware is not conducive to the security management and control of closed networks.

[0007] (2) Limited scalability. After training, many anomaly detection algorithms have fixed model parameters and structures, making it difficult to adapt to emerging anomaly patterns or attack methods in a timely manner, which is the "concept drift" problem. Existing methods often require significant adjustments to the model structure when introducing new data features, improving algorithms, or integrating other technologies, increasing the complexity of implementation and maintenance. The particularity of closed networks also increases the cost of updating models.

[0008] (3) High false positive and false negative rates. Some researchers have noted the effectiveness of locality-sensitive hash functions in detecting abnormal traffic in resource-constrained scenarios. However, the proposed method still suffers from high false positive and false negative rates for abnormal samples.

[0009] (4) Poor real-time performance. Since the models relied on by existing unsupervised detection technologies are relatively complex, the decision-making process takes a long time when hardware resources are limited, making it difficult to provide real-time detection capabilities in dedicated IoT scenarios. Summary of the Invention

[0010] In response to the problems that existing unsupervised anomaly detection models based on machine learning and deep learning are limited in deployment in resource-constrained scenarios, have high computational complexity, and cannot meet real-time detection needs, the present invention proposes a lightweight anomaly traffic detection method for closed networks in resource-constrained scenarios. The method introduces a locality-sensitive hash function to calculate the traffic anomaly score. It does not rely on artificial intelligence models, does not require data labeling and feature extraction, and can achieve good detection effects with low computing resource overhead. In addition, compared with existing detection technologies based on locality-sensitive hash functions, the false alarm rate and missed alarm rate of the method of the present invention are significantly reduced.

[0011] The present invention provides a method for detecting abnormal traffic in a closed network under resource-constrained scenarios, comprising the following steps:

[0012] Step 1: Preprocessing the acquired network traffic, including: extracting valid data fields from network data packets, detecting whether the valid data fields extracted from a single data packet meet a preset length, and if the length is insufficient, sequentially concatenating the valid data fields extracted from multiple data packets to form a sample; the valid data field includes a data packet ID, a flag bit, and data;

[0013] Step 2: Calculate the local sensitive hash value for each sample in sequence to obtain the sample summary;

[0014] Step 3: Set and maintain a sliding window W of fixed size n, which stores the summary of the latest n samples of benign network traffic; n is a positive integer greater than 2;

[0015] Step 4: In the detection phase, obtain each sample to be tested according to step 1 for the network traffic to be tested, obtain the summary of the sample to be tested according to step 2, calculate the similarity of the summary of each sample to be tested with the n samples in the sliding window W, and use the k-nearest neighbor strategy to calculate the anomaly score g of the sample to be tested;

[0016] Step 5: Compare the anomaly score g of the sample to be tested with the preset threshold θ. If g is greater than θ, the sample to be tested is judged as benign traffic, otherwise the sample to be tested is judged as abnormal traffic; update the summary of the sample to be tested that is judged as benign traffic to the sliding window W; when it is judged as abnormal traffic, trigger the alarm mechanism.

[0017] In step 4, a heap of fixed length k is set, in which the first k maximum values ​​of summary similarities between the sample to be tested and the n samples in the sliding window W are stored, and the average value of the k similarities in the heap is calculated as the anomaly score of the sample to be tested.

[0018] Compared with the existing technology, the advantages of the abnormal traffic detection method for closed networks in resource-constrained scenarios of the present invention are:

[0019] (1) The method of the present invention significantly reduces the running overhead. In step 2, the present invention uses the local sensitive hashing method to calculate the sample summary. The calculation only involves simple addition, array sequential search, and integer variable averaging. In step 3, the sliding window method and the k-nearest neighbor strategy used in step 4 are used to calculate the similarity by storing an auxiliary array and the k-nearest neighbor strategy. The similarity calculation involves bit calculation, which is lightweight and fast. At the same time, the time complexity of the heap algorithm is O(nlog k). The present invention reduces the space complexity from O(n) to O(k), and the complexity of calculating the average value is also reduced from O(n) to O(k), usually k = √n. Compared with other methods that require saving floating-point matrices and performing matrix calculations, the method of the present invention is more lightweight.

[0020] (2) The method of the present invention reduces overhead while taking into account detection accuracy. The similarity score calculated based on the local sensitive hash function is used as a good indicator for detecting network traffic anomalies. At the same time, the sliding window and k-nearest neighbor strategy meet the needs of real-time monitoring and detection, avoid the interference of outdated or irrelevant data on the current detection, and improve the reliability of detection.

[0021] (3) In some resource-constrained scenarios, due to the low bandwidth, the traffic data packets sent may have simple frame formats and short data, so the difference between benign traffic and abnormal traffic may not be obvious. For such scenarios, the method of the present invention aggregates multiple messages and then calculates the local sensitive hash summary. The number of messages is dynamically adjusted based on the current network bandwidth.

[0022] (4) The method of the present invention introduces a local sensitive hash function method to detect network traffic anomalies. It does not rely on artificial intelligence models. Compared with traditional unsupervised methods, it does not require feature extraction and model training, is immune to concept drift, has low computing and storage overhead, and is more suitable for deployment in closed networks with limited resources.

[0023] (5) Compared with the traditional anomaly detection method based on local sensitive hashing, the method of the present invention introduces a sliding window method based on the dynamic characteristics of network traffic behavior patterns that change over time, so that the network traffic to be detected is only compared with the latest benign traffic, reducing the detection error; at the same time, since the network traffic contains some redundant and invalid information for error correction and truncation, this part of information may affect the difference between benign traffic and abnormal traffic. The present invention retains the data fields that are effective for detection in the preprocessing stage, thereby improving the effect of the local sensitive hash function in detecting network traffic anomalies; when performing similarity comparison, the k-nearest neighbor strategy is also introduced to further reduce storage overhead and improve the detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flowchart of an implementation of the lightweight abnormal traffic detection method of the present invention;

[0025] Figure 2 This is a flow chart of network traffic preprocessing in step 1 of the method of the present invention;

[0026] Figure 3 This is a flowchart of calculating the local sensitive hash digest using the Nilsimsa algorithm as an example in step 2 of the method of the present invention;

[0027] Figure 4 Flow chart of flow detection in step 5 of the method of the present invention;

[0028] Figure 5 This is a performance comparison chart of different abnormal traffic detection methods in anomaly detection experiments;

[0029] Figure 6 Comparison of storage overhead of different abnormal traffic detection methods. DETAILED DESCRIPTION

[0030] The implementation of the technical solution of the present invention is described below with reference to the accompanying drawings and embodiments.

[0031] The present invention provides a lightweight abnormal traffic detection method for a closed network in a resource-constrained scenario. The overall implementation process is as follows: Figure 1 As shown. The method of the present invention introduces a local sensitive hash function to detect network traffic anomalies. The local sensitive hash summary of the latest set of benign traffic is stored in a sliding window. During the detection phase, the average similarity between the sample to be tested and the top k largest traffic in the sliding window is calculated as the anomaly score. Compared with other unsupervised learning anomaly detection methods, the method proposed in the present invention significantly reduces the operating overhead while ensuring a certain detection accuracy, and is more suitable for resource-constrained network scenarios. The implementation steps of the method of the present invention are described in detail below.

[0032] Step 1: Pre-process the received network traffic. The process is as follows: Figure 2 shown.

[0033] For the acquired network traffic data packets, their valid data fields should be extracted, including: retaining the packet ID, flag bit and data part, and discarding unnecessary or invalid field information such as checksum. In some scenarios with small data packets, since the differences between individual data packets are not obvious, the valid parts of multiple data packets can be spliced ​​together for processing. Check whether the valid data field extracted from the current data packet meets the preset length. If the length is insufficient, continue to extract the valid field for the next data packet, and splice the valid data fields extracted from multiple data packets in sequence as a sample. The sample that meets the preset length is used as the input of the local sensitive hash.

[0034] Step 2: Calculate the locality-sensitive hash value based on the sample data obtained in step 1. The locality-sensitive hash function used by the method of the present invention by default is the Nilsimsa algorithm.

[0035] like Figure 3The figure below illustrates the computational process of a locality-sensitive hash function, using the Nilsimsa algorithm as an example. It provides a 32-byte output representing the distribution of triplets in the data. The Nilsimsa algorithm processes each input sample character by character using a 5-character sliding window. Each time a new character enters the window, eight different trigrams are generated, and each trigram is processed using the hash function h(*). This hash function outputs an integer value i = h(trigram) in the range [0, 256). After analysis, the results are stored in a 256-length array acc. The number of occurrences of each hash value is counted. For each hash function output value i, acc[i] is incremented by 1. Initially, all elements in acc are 0. The total number of processed triplets is divided by 256 to obtain the average number of triplets at each position in acc. This average number is used as the threshold. If the number of occurrences of a triple's hash value exceeds the threshold, the triple is considered significant. The digest array is obtained based on the frequency of the triples at each position in the acc array. Specifically, the element values ​​in the acc array are compared with the threshold threshold. Positions with values ​​greater than the threshold are marked as 1 in the digest digest, and positions with values ​​greater than the threshold are marked as 0. That is, if acc[i] > threshold, digest[i] = 1; otherwise, digest[i] = 0, resulting in a 256-bit array digest. Finally, the digest array is flipped to obtain the final digest, which is a 256-bit array. Figure 3 In the example, the digest array and the final digest are expressed in hexadecimal format, with each digit corresponding to 4 bits. For example, the first digit 8 in the digest array corresponds to 1000, which is flipped to 0001, corresponding to the last digit of the digest, which is 1.

[0036] The method of the present invention introduces a locality-sensitive hash function method to detect network traffic anomalies. Traditional hash functions map different data to different hash buckets, and non-uniformly distributed data is approximately evenly distributed in the hash bucket. However, the locality-sensitive hash function retains the similarity relationship between data, and the locality-sensitive hash functions generated by similar data are also similar or even identical. Generally speaking, the similarity of benign traffic is high, while the similarity between abnormal traffic and benign traffic is low. This method does not rely on artificial intelligence models, does not require feature extraction and model training, is immune to concept drift, has low operating overhead, and is more suitable for deployment in closed networks with limited resources.

[0037] Step 3: Maintain a sliding window to store the latest locally sensitive hash values. Step 1 preprocesses the network traffic to obtain a chronological sample stream, and Step 2 obtains the digest of each sample. This step pre-sets a sliding window of fixed size K and maintains it to store the digests of the latest K samples of benign network traffic. This data is used to detect abnormal network traffic. In this embodiment of the present invention, K is set to a positive integer greater than 2.

[0038] The method of the present invention uses a sliding window to maintain the benign traffic characteristics of the latest network traffic. During detection, the sample summary of the tested traffic is calculated with the sample summary of the benign traffic in the sliding window. The average of the top k maximum values ​​is taken as the anomaly score. The sliding window method can quickly update and process data, which helps to achieve real-time anomaly detection and timely response to potential threats. The fixed-size sliding window limits memory usage and is suitable for running on devices with limited storage space. In addition, the sliding window improves the model's adaptability to dynamic data changes, thereby improving the effectiveness of detection.

[0039] Step 4: During the detection phase, the similarity between the summary of each traffic sample to be tested and the sample summaries in the sliding window is calculated. The method maintains a stack containing the top k maximum similarities and takes the average of the similarities in the stack as the anomaly score of the traffic sample to be tested.

[0040] In this embodiment of the present invention, the Nilsimsa algorithm is used to obtain sample digests. The similarity of the sample digests is calculated by counting the number of bits with equal values ​​in the two sample digests and subtracting 128 from the count to obtain the similarity s. A larger similarity s indicates more similarity between the two sample digests. Therefore, the range of similarity between two sample hash digests is -128 to 128.

[0041] This step employs a k-nearest-neighbor anomaly detection strategy. Directly calculating and averaging all similarities is susceptible to noise or outliers, fails to consider local characteristics of the data, and can lead to errors if the sample similarity distribution is uneven. The proposed method, by selecting an appropriate k, effectively mitigates the impact of noise on the detection results, resulting in smoother results and reducing the impact of single-point anomalies on the overall results. Furthermore, by maintaining a smaller heap array, memory usage is significantly reduced, while also lowering computational overhead.

[0042] Locally sensitive hashing can map inputs of arbitrary length to outputs of fixed length while preserving similarity between data points. The locally sensitive hash value of benign traffic will be similar to previous benign traffic, while the opposite is true for abnormal traffic. Therefore, the locally sensitive hash value of network traffic can be calculated and its similarity calculated with the hash value of normal data as a basis for determining whether the data is abnormal. Since the behavior patterns of normal traffic may change in different time periods, their locally sensitive hash values ​​may also be dissimilar. In addition, in scenarios with limited resources, saving the locally sensitive hash values ​​of all normal data would take up too much storage space. Therefore, a fixed-size window can be maintained to only save the locally sensitive hash values ​​calculated from the latest network traffic, reducing storage overhead while improving detection effectiveness.

[0043] Step 5: Compare the anomaly score of the network traffic sample to be detected with the preset threshold θ to determine whether it is abnormal traffic.

[0044] The process of the detection phase is as follows Figure 4 As shown. In an embodiment of the present invention, the network traffic to be tested is preprocessed according to step 1 to obtain a sample to be tested, the sample summary is extracted according to step 2, and then the similarity between each sample summary to be tested and the sample summary in the sliding window currently maintained in step 3 is calculated, and then the average value of the top k maximum similarities is obtained as the abnormality score g of the sample to be tested. If the Nilsimsa algorithm is selected, the larger the value of g, the more similar it is, so traffic greater than the threshold θ is identified as benign traffic, and traffic less than or equal to the threshold is identified as abnormal traffic. When the current sample is determined to be benign traffic, the content of the sliding window W described in step 3 will be updated. When the current sample is determined to be abnormal traffic, an alarm mechanism will be triggered.

[0045] The method of the present invention was compared with the existing method. The comparison results are as follows: Figure 5 and 6 shown.

[0046] like Figure 5As shown in the figure, the horizontal axis represents three indicators selected to evaluate detection capabilities. Accuracy is a basic indicator for evaluating the overall effectiveness of a classification model, representing the proportion of samples correctly classified by the model to the total number of samples. The false alarm rate is a metric used to measure the performance of a classification model, referring to the proportion of samples that are actually negative that the model incorrectly predicts as positive. A lower false alarm rate means that the model is more accurate in predicting negative samples and fewer false positives. The area under the receiver operating characteristic (AUROC) curve measures the model's ability to distinguish between classes and is independent of any specific decision threshold. Therefore, it provides a comprehensive assessment of model performance that is unaffected by the choice of threshold. The vertical axis represents the scores of different evaluation indicators tested in a resource-constrained network scenario, using a controller area network as an example. Different anomaly traffic detection methods are represented by different colored bars. As can be seen from the figure, the method of the present invention demonstrates superior performance across various evaluation indicators compared to existing typical methods. The accuracy and area under the receiver operating characteristic (ROC) curve reach over 95%, and the false alarm rate is less than 5%. Compared with the direct use of the Nilsimsa algorithm for anomaly detection, the performance improvement is significant. This shows that the method proposed in the present invention not only has lower operating costs, but also has stronger detection capabilities, achieving a balance between detection accuracy and operating costs.

[0047] like Figure 6 The figure shows a comparison of the storage overhead of the proposed method and a partitioning-based method. The horizontal axis shows the different methods, and the vertical axis shows the logarithmic overhead in kB. The sliding window size used in the experiment was 100, and the k-nearest neighbor parameter was k=5. As can be seen from the figure, the storage overhead of the proposed method is much lower than that of the partitioning-based method.

[0048] In general, the method of the present invention extracts the valid fields of network traffic, calculates the local sensitive hash value, makes anomaly judgment based on the similarity of hash summaries, introduces traffic aggregation, sliding window and k-nearest neighbor strategies, avoids the interference of irrelevant and noisy data, and improves the anomaly detection effect.

[0049] Except for the technical features described in the specification, all other technical features are known to those skilled in the art. The present invention omits descriptions of well-known components and well-known technologies to avoid redundancy and unnecessary limitation of the present invention. The implementation methods described in the above embodiments do not represent all implementation methods consistent with the present application. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the present invention.

Claims

1. A method for detecting abnormal traffic in a closed network under resource-constrained scenarios, characterized in that: The steps include: Step 1: Preprocess the acquired network traffic in sequence to generate a sample stream. The preprocessing includes: extracting valid data fields from network data packets, checking whether the valid data fields extracted from a single data packet meet a preset length, and if the length is insufficient, sequentially concatenating the valid data fields extracted from n data packets to form a sample; the valid data field includes a data packet ID, a flag bit, and data; n is a positive integer; Step 2: Calculate the local sensitive hash value for each sample in sequence to obtain the sample summary; Step 3: Set and maintain a sliding window W of fixed size K, which stores the summary of the latest K samples of benign network traffic; K is a positive integer greater than 2; Step 4: In the detection phase, obtain each sample to be tested according to step 1 for the network traffic to be tested, obtain the summary of the sample to be tested according to step 2, calculate the similarity between each sample to be tested and the summary of the K samples in the sliding window W, and take the average of the top k maximum similarities as the anomaly score g of the sample to be tested; Step 5: Compare the anomaly score g of the sample to be tested with the preset threshold θ. If g is greater than θ, the sample to be tested is judged as benign traffic, otherwise the sample to be tested is judged as abnormal traffic; update the summary of the sample to be tested that is judged as benign traffic to the sliding window W; when it is judged as abnormal traffic, trigger the alarm mechanism.

2. The method according to claim 1, characterized in that In the step 2, the input sample is calculated using the Nilsimsa algorithm to calculate a locally sensitive hash value and generate a sample digest, including: using a 5-character sliding window to scan the data of the input sample character by character, generating 8 different triplets for each sliding window, calculating the hash value of the triplet using the same hash function, and counting the 256-length array acc, wherein the i-th element of acc counts the number of occurrences of the hash value i; dividing the total number of triplets processed by the input sample by 256 to obtain a threshold threshold, and if the number of occurrences of the hash value of a triple exceeds the threshold, the triple is considered to be significant; comparing each element value in the array acc with the threshold threshold to obtain a digest array digest, if the i-th element value of acc is greater than the threshold, setting the i-th element value of digest to 1, otherwise setting the i-th element value of digest to 0, and finally flipping the obtained digest array digest to obtain the digest of the input sample.

3. The method according to claim 1, characterized in that In step 4, a heap of fixed length k is set, in which the first k maximum values ​​of summary similarities between the sample to be tested and the K samples in the sliding window W are stored, and the average value of the k similarities in the heap is calculated as the anomaly score of the sample to be tested.

4. The method according to claim 1 or 3, characterized in that In step 4, the similarity between the two sample summaries is calculated by counting the number of bits with equal values ​​in the two sample summaries, and subtracting 128 from the statistical value to obtain the similarity s. The larger the value of the similarity s, the more similar the two sample summaries are.

Images