A method for detecting APT malicious traffic

Abstract

The application provides a kind of APT malicious traffic detection method, it is related to network security technical field.The method includes: copying flow data to detection system, processing the characteristics of sample therein and converting into two-dimensional image, according to attribute splicing multiple samples and then inputting neural network model containing seed network.The neural network model is subjected to depth remapping, width remapping and internal remapping;Then through network expansion, architecture adaptation and parameter adaptation, neural network architecture search is completed, target network is obtained, and the two-dimensional image is detected using the target network.The method can improve the real-time performance, accuracy and comprehensiveness of malicious traffic detection.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to a method for detecting APT (Advanced Persistent Threat) malicious traffic. Background Technology

[0002] In recent years, the network environment has expanded rapidly with the advancement of information and communication technologies, while facing an increasing number of network threats. Currently, Advanced Persistent Threat (APT) attacks have become a focus of concern for countries and organizations worldwide. Attackers typically employ various strategies to carry out APT attacks, including zero-day exploits, where attackers discover new vulnerabilities in target systems or networks and exploit them immediately. Because APT attacks are persistent and their high-volume nature poses a risk of system crashes, detecting such attacks is quite challenging. Traditional security measures, such as email security, web server security, and antivirus software, while capable of effectively distinguishing general attacks, are no longer adequate against more complex APT attacks. Furthermore, existing firewalls have significant limitations, unable to achieve real-time detection of APT attacks while maintaining high accuracy.

[0003] In current technologies, using machine learning techniques to solve anomaly detection problems has become a trend. However, due to the complexity of normal behavior and the problem of data imbalance in machine learning model training—that is, the number of samples in some categories is significantly less than that in other categories—the model may perform poorly on a few categories, resulting in a lack of comprehensive performance. Summary of the Invention

[0004] To address the technical problems existing in the prior art, this invention proposes a method for detecting APT malicious traffic, specifically an adaptive fast neural network-based AutoASD (Automated Architecture Search for Detecting) method based on spatiotemporal features. This method adapts the architecture and parameters of the seed network to achieve networks with different depths, widths, or task requirements. Furthermore, this method introduces a large number of possible sub-paths through implicit embedding to limit the complexity of the search space, and utilizes parameter remapping techniques to adjust the kernel, effectively leveraging NAS (Network Architecture Search) for detection and segmentation tasks. Specifically, the method includes:

[0005] Copy the traffic data to be detected via port mirroring;

[0006] The data packets in the traffic data to be detected are used as samples. Missing values ​​and outliers are handled, and the features of the samples are divided into numeric features and character features. The character features are then one-hot encoded.

[0007] All features are normalized by mean, and then the attribute values ​​of each feature's category are made to be in the range [0, 255) by min-max normalization. The result is then converted into a two-dimensional image.

[0008] For any category of malicious traffic, stitch together a two-dimensional image of samples; where, in the stitched two-dimensional image, corresponding parts of the same attribute are aligned in the same column, and 'a' is the value that maximizes the probability of the traffic data to be detected being classified as that category;

[0009] Input the 2D image into the seeded network p u The neural network model is then remapped using a deep method: for the seed network p u In a stage of n layers, any layer has parameters. Correspondingly, the new network p q The corresponding stage of the middle S layer has parameters Seed network p u The parameters of the last layer in the network are copied to the new network p. q In the network layer,

[0010]

[0011] F(c) = min(c,n)

[0012] In the formula, c represents the new network p in the neural network model. q The number of layer order in the network. For the new network p q Parameters of the new layer, For seed network p u The parameters of the last layer;

[0013] Width remapping of a neural network model: given a seed network p u A convolutional layer with parameters y u ∈T e ×f×j×k For the new network p q The corresponding convolutional layer has parameters y q ∈T a×b×j×k In the width dimension, the seed network p u The first a or first b channels of the intermediate parameters are mapped to the new network p. q The passage in the middle,

[0014]

[0015] Where e and f are seed networks pu The input and output dimensions of the parameters, where a and b are the parameters of the new network p. q The input and output dimensions of the parameters are defined, where j and k are the width and height of the spatial dimension, and a << e, b << f; ":" indicates the range of values ​​for the dimension. For the new network p q All weights of the convolution kernel determined by the h-th and d-th output channels, For seed network p u All weights of the convolution kernel determined by the h-th and d-th output channels;

[0016] Kernel remapping is performed on the neural network model: the values ​​of the original 3×3 kernel parameters are assigned to a 3×3 region at the center of a large kernel with dimensions ω×ω, and the values ​​in other regions are assigned to 0.

[0017]

[0018] In the formula, For a large range of parameters in terms of kernel width and height; This represents the original 3×3 kernel width and height parameter range;

[0019] Multiple feasible sub-paths are introduced through implicit embedding, and the seed network p u Expanding into a super network, and extending the seed network p u The parameters are mapped to the supernet;

[0020] In the malicious traffic detection and classification task, a differentiable NAS process is initiated using a super network: the operation weights of the super network in certain periods on the training dataset are fine-tuned and trained accordingly, and then the operation weights and architecture parameters are alternately iterated to optimize the operation weights and architecture parameters; wherein, in each alternate iteration, the operation weights are fine-tuned according to the path sampled based on the architecture parameter distribution.

[0021] After the differentiable NAS process terminates, the target architecture is derived using the architecture parameters;

[0022] Seed network p u The parameters are mapped to the target architecture, and the target architecture is fine-tuned in the malicious traffic detection and classification task to obtain the target network;

[0023] The target network is used to detect the two-dimensional image corresponding to the traffic data to be tested.

[0024] Furthermore, the method also includes preprocessing the two-dimensional image:

[0025] Use gamma correction to adjust the pixel values ​​of a 2D image.

[0026]

[0027] In the formula, P is the new pixel value, P0 is the original pixel value, and γ is the gamma value;

[0028] Perform bit depth reduction and color balance adjustment on a 2D image.

[0029] Preferably, multiple feasible sub-paths are introduced through implicit embedding to expand the seed network into a super network, including:

[0030] The malicious traffic detection category can be broadened to an operable softmax.

[0031]

[0032] In the formula, Φ represents the operation set corresponding to malicious traffic detection and classification, and x represents the input tensor. k-th level operation Architecture parameters, For the k-th level operation Architecture parameters, This is a classification for detecting malicious traffic targeting x.

[0033] As can be seen, compared with the prior art, the technical solution proposed in this invention can realize networks with different depths, widths or different task requirements by adapting to the architecture and parameters of the seed network; by adjusting the kernel using parameter remapping technology, it can more effectively utilize NAS for detection and segmentation tasks; by introducing a large number of feasible sub-paths through implicit embedding, it can limit the complexity of the search space; furthermore, the combination of the above technical features can improve the real-time performance and accuracy of malicious traffic detection.

[0034] In addition to achieving the aforementioned beneficial effects, each preferred solution also achieves the following beneficial effects: brightness adjustment can finely adjust the brightness of the image while preserving details in both bright and dark areas; depth reduction helps reduce the storage space occupied by the image and maintains the visual quality of the image to a certain extent; color balance adjustment helps maintain the authenticity of different colors in the image, better highlighting the key color characteristics in the image and improving the image's recognizability; relaxing specific operations and introducing multiple feasible sub-paths can make the space continuous, which is conducive to the neural network model adaptively adjusting the feature weights, thereby more comprehensively distinguishing different attack categories. Attached Figure Description

[0035] Figure 1 A schematic diagram of the AutoASD automation architecture for applying the method proposed in this invention.

[0036] Figure 2 This is a schematic diagram illustrating the feature classification of the traffic dataset used in one embodiment of the present invention.

[0037] Figure 3This is a schematic diagram of the extraction length of various traffic features in one embodiment of the present invention.

[0038] Figure 4 This is a schematic diagram of a neural network self-adaptive search framework in one embodiment of the present invention.

[0039] Figure 5 This is a schematic diagram illustrating the performance of the AutoASD model on the UNSW-NB15 image dataset in one embodiment of the present invention.

[0040] Figure 6 This is a schematic diagram comparing the results of the AutoASD model and the traditional machine learning algorithm in one embodiment of the present invention. Detailed Implementation

[0041] One of the key challenges in malicious traffic classification tasks in existing technologies is the real-time and high-precision detection. Furthermore, attackers may use structured models to inspect target machines unnoticed, thus remaining within the system for extended periods. In recent years, to address these challenges, research in this field has focused on using machine learning techniques to solve the anomaly detection problem. Anomaly detection methods are approaches that detect anomalous behavior by modeling normal behavior. However, due to inherent limitations, such as the complexity of normal behavior and the problem of imbalanced data often encountered when training machine learning models (where the number of samples in some categories is significantly less than in others), the model performs poorly on a few categories, failing to achieve comprehensive and excellent functionality.

[0042] To overcome the aforementioned limitations, techniques such as Neural Architecture Search (NAS) and Transfer Learning (TL) have been widely applied. NAS is a technique for automatically finding the optimal neural network architecture to meet the needs of a specific task. In the past, NAS methods mainly searched for network architectures in fixed tasks, especially image classification. NAS methods consume a lot of computational resources, and early methods often used nested approaches to search for architectures. More recently, attempts have been made to reduce computational intensity through weight sharing. However, existing NAS methods still have high technical barriers in terms of parameter settings and computational requirements.

[0043] On the other hand, transfer learning, as a key technique, can adapt models to new tasks by leveraging knowledge from pre-trained models. In the field of computer vision, many state-of-the-art (SOTA) methods employ neural network architectures designed for image classification tasks, such as networks pre-trained on ImageNet (a large visualization database for research on visual object recognition software), and then fine-tuned for other tasks. It is worth noting that network architectures specifically designed for particular tasks can lead to performance improvements.

[0044] The inventors will now provide a more detailed description of the invention in conjunction with the accompanying drawings and specific embodiments.

[0045] Example 1

[0046] The AutoASD automation system corresponding to the technical solution proposed in this invention is divided into four major modules.

[0047] like Figure 1 As shown, Module 1 performs traffic capture and signature processing. At the start of traffic capture detection, the copied traffic is forwarded to the detection system via port mirroring, such as SPAN (Switched Port Analyzer). SPAN is a workstation that runs packet capture applications or remote monitoring probes, aiding in performance management and troubleshooting. It can copy network traffic from a specific VLAN or a group of ports to a single port, typically connected to a network analyzer such as a switchProb device. SPAN does not affect the network traffic exchange of the source port or VLAN. AutoASD does not interfere with normal network traffic forwarding. If the detection system has no prior knowledge of the threat, it can handle common C2 traffic attacks in APTs, and existing malicious traffic defenses can be deployed to suppress detected traffic. Furthermore, a detection system developed based on the AutoASD automation architecture can determine the benign or malicious nature of traffic passing through the middlebox by monitoring ongoing traffic.

[0048] Module 2 preprocesses the one-dimensional traffic data captured in Module 1, converting it into a two-dimensional image and aggregating information and relationships from multiple samples to facilitate subsequent neural network processing. Module 3 performs a network architecture search on the preprocessed image data from Module 2, obtaining the optimal neural network architecture and parameters that perform best on the validation set. Module 4 uses the optimal neural network model obtained in Module 3 to train on the training set. The convolutional neural network can extract image features and use them for classification tasks. Finally, the model's classification performance is validated on the test set.

[0049] In recent years, deep learning and image classification technologies have achieved significant success in the field of computer vision. In this invention, traffic data captured by the detection system is preprocessed and converted into two-dimensional images. This allows for the application of existing mature algorithms and models in image classification, such as Convolutional Neural Networks (CNNs), and transfer learning is performed using a pre-trained deep learning model. Compared to traditional one-dimensional data, two-dimensional images carry richer spatial information. The position of each pixel and its relationship to surrounding pixels provides more key features. Thus, spatial information helps to better capture the patterns and structures of malicious traffic data, thereby improving classification accuracy. Simultaneously, data augmentation techniques commonly used in image data processing help to expand the dataset, increase sample diversity, and improve the model's generalization ability. These methods are particularly important when malicious traffic datasets are typically small and imbalanced.

[0050] In this invention, the inventors used the UNSW-NB15 dataset. The effectiveness of malicious traffic classification and detection is evaluated based on its performance in identifying attacks, requiring a comprehensive dataset containing both normal and anomalous behavior. Older benchmark datasets are KDDCUP 99 and NSLKDD, which have been widely used. However, the KDDCUP 99 dataset contains a large number of redundant records in its training set, which can affect the bias of detection results towards frequent records. Therefore, using the aforementioned dataset for evaluation does not reflect the actual output performance. Secondly, multiple missing records can also change the nature of the data. The NSLKDD dataset is an improved version of KDDCUP99, which addresses issues such as data imbalance between normal / anomalous records and missing values. However, this dataset does not fully represent the modern low-footprint attack environment. To address more severe challenges, the UNSW-NB15 dataset was developed. The original network packets of the UNSW-NB15 dataset were created using the IXIAPerfectStorm tool from the Canberra Cyber ​​Range Lab at the University of New South Wales to generate a mixture of realistic modern normal activity and synthetic contemporary attack behavior. The tcpdump tool is used to capture raw traffic in 100GB Pcap files. This dataset contains normal traffic and nine types of attacks, the quantity and characteristics of which are shown in Table 1.

[0051] Table 1. Distribution of records in the UNSW-NB15 dataset

[0052]

[0053] The dataset was collected using the Bro-IDS (Bro Intrusion Detection System) tool, and 12 algorithms were developed to generate a total of 49 features with category labels. The 49 features were then classified as follows: Figure 2 As shown.

[0054] In APT attacks, the amount of data varies greatly across different categories. Therefore, the data is first normalized, including handling missing values ​​and outliers. Then, the 49 features are divided into numeric and character features, and the character features (such as protocol, service, attack category, etc.) are one-hot encoded. Next, all features are mean-normalized, mapping the data mean to 0. Min-Max Normalization is then used to ensure that the values ​​of each attribute are within the range [0, 255) to facilitate image transformation.

[0055] In the field of machine learning, standardization and normalization are both common data preprocessing methods used to adjust the scale of variables for more effective use in machine learning algorithms. However, their methods and purposes differ. Standardization aims to transform data into a distribution with a mean of 0 and a standard deviation of 1. It is suitable for data distributions that approximate a normal distribution and has good robustness in handling outliers. This method is also known as Z-score standardization, achieved by subtracting the mean and then dividing by the standard deviation. Normalization aims to scale data to a fixed range, typically 0 to 1 (0-255 in this invention). It is suitable for situations where no data distribution is assumed, especially for algorithms like neural networks that require data to be within a fixed range. This method is also known as Min-Max scaling, achieved by subtracting the minimum value and then dividing by the difference between the maximum and minimum values.

[0056] Finally, the algorithm converts the one-dimensional data into a two-dimensional image. The algorithm groups data according to the attack type (attack_cat), further distinguishes traffic from the same data stream within each group, concatenates n traffic data points from the same group and data stream into a longer one-dimensional array, uses Python's `to_2d_matrix()` function to convert the one-dimensional array into a two-dimensional matrix, and then uses the `to_image()` function to convert the two-dimensional matrix into an image. The attribute values ​​are the grayscale values. The details of the algorithm are as follows... Figure 3 As shown.

[0057] After feature processing, the traffic data to be detected is processed individually. Various behaviors in the network consist of a series of requests, responses, and data transmission operations, forming specific behavioral characteristics. From a traffic perspective, network behavior exhibits distributional characteristics in combinations of data packets. In the task of classifying malicious traffic data, traditional methods typically treat individual data packets as independent detection samples. However, in real-world scenarios, many types of attacks are not accomplished solely through single data packets. For example, DoS attacks (denial of service attacks) often involve sending a large number of malicious data packets, rendering servers or network resources unavailable to users. For this type of attack, considering only a single data packet will affect the accuracy of detecting anomalous behavior.

[0058] Therefore, to better capture the patterns and contextual information of attack behavior, this embodiment employs a novel method: grouping data based on attack type and the same type of traffic, and then concatenating multiple data samples from the same group. Different attack types may exhibit clear temporal and contextual relationships over a period of time. Therefore, aggregating behavior patterns with these characteristics effectively combines information from multiple samples within a single image, thereby capturing richer and more diverse features and better distinguishing different attack types. Furthermore, this approach can mitigate the impact of individual noise on the overall classification, improving the model's stability and robustness. This graphical representation contains more feature information, contributing to improved performance in malicious traffic classification tasks.

[0059] After preprocessing, the labels and various traffic attributes of each sample are removed, and the final dimension is A = {α1, α2, ..., α}. n}, where n is the number of feature dimensions. In this embodiment, the value 'a' that maximizes the accuracy of each attack category can be adaptively selected. The value 'a' means concatenating the attributes of 'a' samples from the same data stream into a two-dimensional image. For example, concatenating 3 samples into a (3, 44) two-dimensional array, with identical attribute values ​​in the same column, ultimately generates an image with a width × height of 11 × 12. Similarly, 5, 8, and 10 samples are concatenated into two-dimensional arrays with dimensions of (5, 44), (8, 44), and (10, 44), respectively, generating images with width × height of 11 × 20, 16 × 22, and 20 × 22, respectively. To facilitate model processing, all input images are resized to 30 × 30. Figure 3 The partial samples of each dataset shown are images generated by stitching together the samples, and it can be seen that the distinction between each category is obvious.

[0060] After the above processing, in some embodiments, in order to improve the recognizability of the image and the clarity of the contour, it is necessary to further process the image using image enhancement and contour recognition preprocessing methods.

[0061] Firstly, to improve image brightness, a non-linear brightness adjustment method, gamma correction, is used to adjust the image's pixel values.

[0062]

[0063] In the formula, P is the new pixel value, P0 is the original pixel value, and γ is the gamma value.

[0064] The gamma value can be adjusted according to the characteristics of the image to achieve optimal brightness. By applying gamma correction, the brightness of the image can be finely adjusted while preserving details in both bright and dark areas.

[0065] Secondly, to reduce the amount of image data, this embodiment reduces the number of bits in each color channel, i.e., bit depth reduction. The color values ​​of the original image are uniformly quantized to a specified bit depth range, such as 8 bits. This process helps reduce the storage space occupied by the image while maintaining the visual quality of the image to a certain extent.

[0066] Thirdly, to correct color deviations in the image, this embodiment uses a color balance adjustment method, which achieves color balance by adjusting the weights of the red, green, and blue channels. This method helps maintain the authenticity of different colors in the image and better highlights the key color characteristics of the image, thereby improving the image's recognizability.

[0067] The preprocessing methods described above effectively enhance image brightness, reduce data volume, and adjust color balance, facilitating subsequent analysis and recognition tasks. The combination of these steps demonstrates good performance when processing images with varying brightness, color, and shape characteristics.

[0068] After the above preprocessing, label images with clear feature distinctions between each category can be obtained. These label images are then directly input into a trained neural network model to learn features and complete the classification task.

[0069] In this field, fine-tuning deep convolutional neural networks (CNNs) using pre-trained models helps transfer knowledge learned from large datasets to a target task. While deep transfer learning generally improves performance through better accuracy and faster convergence, pre-trained CNN models have tens of thousands of weights, layers, and other parameters, and transfer learning results are often closely related to parameter settings. Therefore, transferring weights from an inappropriate network can impair the training process and potentially lead to lower accuracy.

[0070] In this embodiment, the AutoASD method is employed, utilizing a core algorithm that combines a unique parameter remapping paradigm with architecture search to enable the neural network architecture to efficiently and effectively adapt to new tasks. This method aims to adapt the architecture and parameters of a pre-trained network to new tasks while minimizing computational costs. As the most commonly used network for designing the search space in NAS methods, MobileNetV2 is selected as the seed network in this embodiment. To adapt the network to segmentation and detection tasks, architectural elements are adjusted at three levels: convolutional kernel size, network depth, and width.

[0071] In this embodiment, the inventors define parameter remapping as a paradigm aimed at mapping the parameters of one seed network to another. Specifically, the seed network is represented as p. u The target network is represented as p q Their parameters are y u and y q The remapping paradigm first performs depth-level remapping, and then simultaneously performs kernel-level and width-level remapping.

[0072] Deep remapping: Introducing more depth settings during architecture adaptation. If the seed network p u One stage has n layers, and the parameters of each layer can be expressed as: Similarly, if the target network p q The corresponding stage of the middle S layer has parameters p q It also exists in p u The parameters of the layer in the middle, and are p u This was copied from [source]. q The parameters of the network layers are all derived from p u The final layer of replication in this stage is obtained. Specifically:

[0073] F(c) = min(c,n)

[0074]

[0075] In the formula, c represents p in the neural network model. q The number of network layers, For p q Parameters of the new layer, For p u The parameters of the last layer.

[0076] Width remapping: for p u A convolution with parameters y u ∈T e×f×j×k ; for p q There is a parameter y q ∈Ta×b×j×k At the width level, p will be... u The first a or b channel of the medium parameter is mapped to the narrower medium channel.

[0077]

[0078] Where e and f are p u The input and output dimensions of the parameters are p, where a and b are p. q The input and output dimensions of the parameters are defined, where j and k are the width and height of the spatial dimension, and a << e, b << f; ":" indicates the range of values ​​for the dimension. For the new network p q All weights of the convolution kernel determined by the h-th and d-th output channels, For seed network p u The weights of the convolution kernel are determined by the h-th and d-th output channels.

[0079] Here, "narrower" refers to layers in the new network with fewer channels compared to the seed network. During the remapping process at the width level, if the width (i.e., number of channels) of a layer in the new network is smaller than the width of the corresponding layer in the seed network, that layer is considered "narrower." Therefore, only the parameters of the first 'a' or 'b' channels in the seed network are directly mapped to this narrower layer in the new network. Other channels exceeding the width of the new network layer are not mapped. In short, the goal is to preserve as much parameter information as possible from the seed network without changing the width of the new network layers.

[0080] Kernel Remapping: In most manually designed networks, the kernel size is typically set to 3×3. To expand the receptive field and capture rich features in segmentation or detection tasks, this embodiment introduces a larger kernel size setting during the adaptation process. To expand the 3×3 kernel to a larger kernel, the inventors assign the parameters of the central 3×3 region of the larger kernel to the values ​​of the original 3×3 kernel, and assign the values ​​of other regions around the central part to 0. The inventors represent the parameters of the original 3×3 kernel as y. 3×3 Representing a larger ω×ω kernel as y ω×ω The original 3×3 kernel parameter values ​​are assigned to a 3×3 region at the center of a large kernel with kernel dimensions of ω×ω, and the values ​​in other regions are assigned to 0.

[0081]

[0082] In the formula, The parameter range of the large kernel in the j and k dimensions; The parameter range of the original 3×3 kernel in the j and k dimensions.

[0083] In deep neural networks within machine vision, there's a concept called the receptive field, which represents the size of the area of ​​the original image that neurons at different locations within the network can perceive. Neurons cannot perceive all the information in the original image because these network structures commonly use convolutional and pooling layers, with each layer locally connected by sliding filters. A larger receptive field value indicates a wider range of the original image it can access, potentially containing more global and semantically higher-level features; conversely, a smaller value indicates that the features it contains are more local and detailed. Therefore, the receptive field value can be used to determine the approximate level of abstraction for each layer.

[0084] Subsequently, the neural network is adapted, completing the search for its self-adaptive architecture. This embodiment divides neural network adaptation into three steps. For example... Figure 4 As shown, firstly, the seed network is expanded into a super network, which represents the search space during the later architecture adaptation process. Secondly, a differentiable NAS method is executed to achieve network adaptation at the architecture level and obtain the target architecture. Finally, the parameters of the target architecture are adjusted to obtain the target network. It should be noted that the above parameter remapping mechanism is deployed before the architecture adaptation and parameter adaptation phases.

[0085] Network expansion: Multiple feasible sub-paths are introduced through implicit embedding, expanding the seed network into a super network, and the parameters of the seed network are mapped to the super network. Specifically, to use a differentiable NAS algorithm, Φ represents the set of candidate discrete operations (e.g., convolution, pooling, zero-padding, etc., i.e., the set of operations corresponding to malicious traffic detection and classification). To maintain spatial continuity, the classification selection of a specific operation is relaxed to the softmax of all possible operations, i.e.

[0086]

[0087] In the formula, x is the input tensor. k-th level operation Architecture parameters, For the k-th level operation Architecture parameters, This is a classification for detecting malicious traffic targeting x.

[0088] Architecture Adaptation: An extended supernetwork is used to initiate a differentiable NAS process on a malicious traffic detection and classification task. During the initial search phase, the architecture parameters cannot be updated in a clear and correct direction because the operation weights are not sufficiently trained. Therefore, the operation weights of the supernetwork are first fine-tuned on the training dataset for certain periods. After the weights are sufficiently trained, the operation weights and architecture parameters are alternately optimized. To accelerate the search process and decouple the parameters of different subnetworks, only one path is sampled based on the distribution of the architecture parameters for operation weight updates in each iteration. When the search process terminates, the architecture parameters are used to deduce the target architecture.

[0089] Parameter Adaptation: Obtaining the target architecture from architecture adaptation. To adapt to new segmentation or detection tasks, the target architecture becomes different from the seed network originally designed for image classification tasks. Unlike traditional training strategies, this invention does not require a cumbersome pre-training process on ImageNet. This embodiment utilizes a parameter mapping method to remap the parameters of the seed network to the target architecture. Finally, fine-tuning is performed directly on the target task to obtain the final target network, which is then applied to detect the two-dimensional images corresponding to the traffic data samples to be tested.

[0090] In this embodiment, the inventors also set up detection and evaluation indicators.

[0091] In classification tasks, commonly used evaluation metrics include accuracy, recall, precision, and F1 score. A true positive (TP) represents an attack that occurred and an alert was issued; a false positive (FP) represents an attack that did not occur but an alert was issued; a true negative (TN) represents an attack that occurred but no alert was issued; and a false negative (FN) represents an attack that occurred but no alert was issued. Accuracy, precision, recall, and F1 score are defined as follows.

[0092]

[0093]

[0094]

[0095]

[0096] TP, FP, TN, and FN refer to true positive, false positive, true negative, and false negative, respectively. Due to the severe imbalance in the dataset, ACC (Accuracy) cannot reflect the true performance of the experiment. For example, even if anomalous samples are easily identified as normal samples, the accuracy remains quite high in an imbalanced dataset because the data for normal samples is biased, which is meaningless for a network traffic anomaly detection system. Therefore, as mentioned in the preprocessing section, resampling should be performed for each category to maintain a balanced sample size, making the above evaluation metrics meaningful.

[0097] This embodiment uses the above detection evaluation indicators to evaluate the detection results of this embodiment, and conducts an in-depth comparison with traditional machine learning algorithms.

[0098] First, we demonstrate the classification performance of the AutoASD model on the dataset, such as... Figure 5 As shown, the AutoASD model achieved high accuracy, recall, and F1 score across different categories. In the `analysis` category, the AutoASD model exhibited high accuracy (78.7%), but relatively low recall and F1 score (35% and 49%, respectively). This suggests the possibility of some false negatives in this category. In the `backdoor` category, the AutoASD model achieved 100% accuracy, but lower recall and F1 score (24.0% and 38.0%, respectively). This indicates a potential for more false negatives in the `backdoor` category. In other categories, the AutoASD model achieved relatively good classification results, demonstrating high accuracy, recall, and F1 score. These results show that the AutoASD model exhibits different performance across different categories. This overall performance makes the AutoASD model quite outstanding in malicious traffic classification tasks.

[0099] To more comprehensively compare the performance of the models, the inventors compared the AutoASD model with traditional machine learning algorithms, including SVM (Support Vector Machine), decision trees, KNN (K-Nearest Neighbor), and Naive Bayes, on different categories. Figure 6 As shown, the AutoASD model exhibits a clear advantage in the vast majority of categories.

[0100] from Figure 6 In the results, it is clear that the AutoASD model exhibits a significant advantage in most categories. However, in certain categories, such as shellcode, traditional machine learning algorithms SVM and DT appear to perform better. This performance difference may stem from two aspects.

[0101] First, the AutoASD model performs end-to-end learning on the dataset, automatically extracting deep feature representations to better capture the inherent structure of the data. Traditional machine learning algorithms, on the other hand, require manual feature engineering, which may not fully uncover the latent features of the data. Second, the AutoASD model integrates multiple convolutional layers and attention mechanisms during training, adaptively adjusting the weights of different features to more effectively distinguish between different categories. Traditional machine learning algorithms, however, may be limited in their ability to adjust feature weights and cannot flexibly handle complex data relationships.

[0102] In addition to comparing accuracy, the inventors also focused on the model's robustness and generalization ability. Figure 6 As can be seen, in some categories (such as normal and reconnaissance), the performance of traditional machine learning algorithms is relatively low, while the AutoASD model shows more stability. This indicates that the AutoASD model has stronger robustness and can better cope with the differences in data distribution between different categories. Furthermore, in a few categories (such as shellcode), traditional machine learning algorithms perform well, but this does not mean that the corresponding model will perform equally well in other categories. This suggests that traditional machine learning algorithms may suffer from overfitting problems in a few sample categories, while the AutoASD model is more adaptable to different data distributions across categories globally.

[0103] The inventors also compared the performance of the AutoASD model and traditional machine learning algorithms under different data dimensions and sample sizes. As the data dimension and sample size increased, the AutoASD model maintained a relatively stable performance improvement trend across most categories. In contrast, the performance of traditional machine learning algorithms may reach a bottleneck in some cases, unable to improve further. This indicates that compared to the limitations of traditional machine learning algorithms in handling high-dimensional and large-sample-size data, which may prevent them from fully utilizing the potential features of the data, the AutoASD model can better mine the deeper information of the data under high-dimensional and large-sample-size conditions, thus achieving stronger generalization ability.

[0104] As can be seen, compared with the prior art, the technical solution proposed in this invention can realize networks with different depths, widths or different task requirements by adapting to the architecture and parameters of the seed network; by adjusting the kernel using parameter remapping technology, it can more effectively utilize NAS for detection and segmentation tasks; by introducing a large number of feasible sub-paths through implicit embedding, it can limit the complexity of the search space; furthermore, the combination of the above technical features can improve the real-time performance and accuracy of malicious traffic detection.

[0105] In addition to achieving the aforementioned beneficial effects, each preferred solution also achieves the following beneficial effects: brightness adjustment can finely adjust the brightness of the image while preserving details in both bright and dark areas; depth reduction helps reduce the storage space occupied by the image and maintains the visual quality of the image to a certain extent; color balance adjustment helps maintain the authenticity of different colors in the image, better highlighting the key color characteristics in the image and improving the image's recognizability; relaxing specific operations and introducing multiple feasible sub-paths can make the space continuous, which is conducive to the neural network model adaptively adjusting the feature weights, thereby more comprehensively distinguishing different attack categories.

[0106] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting APT malicious traffic, characterized in that, include: Copy the traffic data to be detected via port mirroring; The data packets in the traffic data to be detected are used as samples. Missing values ​​and outliers are handled, and the features of the samples are divided into numeric features and character features. The character features are then one-hot encoded. All features are normalized by mean, and then the attribute values ​​of each feature's category are made to be in the range [0, 255] by min-max normalization. For any category of malicious traffic, the processed 'a' samples are concatenated into a one-dimensional array, and then successively converted into a two-dimensional matrix and a two-dimensional image; where, in the two-dimensional image, corresponding parts of the same attribute are aligned in the same column, and 'a' is the value that maximizes the probability of the traffic data to be detected being classified as that category; Input the 2D image into a seeded network The neural network model is then remapped using deep mapping: for the seed network In a stage of n layers, any layer has parameters. Correspondingly, the new network The corresponding stage of the middle S layer has parameters Seed network Copy the parameters from the original network to the new network. In the network layer, ; ; In the formula, c represents the new network in the neural network model. The number of layer order in the network. For the new network Parameters of the new layer, For seed networks The parameters of the last layer; Width remapping of neural network models: given a seed network A convolutional layer has parameters For the new network The corresponding convolutional layer has parameters. In the width dimension, the seed network The first a or first b channels of the intermediate parameters are mapped to the new network. The passage in the middle, ; Where e and f are seed networks The input and output dimensions of the parameters are: a and b, representing the new network. The input and output dimensions of the parameters are given, where j and k are the width and height of the spatial dimension. ;:” represents the range of values ​​for the dimension. For the new network All weights of the convolution kernel determined by the h-th and d-th output channels, For seed networks All weights of the convolution kernel determined by the h-th and d-th output channels; Kernel remapping of the neural network model: Reassigning the values ​​of the original 3×3 kernel parameters to kernels with dimensions of 3×3. The core is a 3×3 region, and the values ​​in other regions are assigned to 0. ; In the formula, For a large range of parameters in terms of kernel width and height; This represents the original 3×3 kernel width and height parameter range; Multiple feasible sub-paths are introduced through implicit embedding to enhance the seed network. Expanding into a super network and seed network The parameters are mapped to the supernet; In the malicious traffic detection and classification task, a differentiable NAS process is initiated using a super network: the operation weights of the super network in certain periods on the training dataset are fine-tuned and trained accordingly, and then the operation weights and architecture parameters are alternately iterated and optimized; wherein, in each alternate iteration, the operation weights are fine-tuned according to the path sampled based on the architecture parameter distribution. After the differentiable NAS process terminates, the target architecture is derived using the architecture parameters; Seed network The parameters are mapped to the target architecture, and the target architecture is fine-tuned in the malicious traffic detection and classification task to obtain the target network; The target network is used to detect the two-dimensional image corresponding to the traffic data to be tested.

2. The method for detecting APT malicious traffic as described in claim 1, characterized in that, The process of inputting the two-dimensional image into the neural network model includes preprocessing the two-dimensional image beforehand. Use gamma correction to adjust the pixel values ​​of a 2D image. ; In the formula, P is the new pixel value, P0 is the original pixel value, and γ is the gamma value; Perform bit depth reduction and color balance adjustment on a 2D image.

3. The method for detecting APT malicious traffic as described in claim 1, characterized in that, The method of implicitly embedding multiple feasible sub-paths to expand the seed network into a super network includes: The malicious traffic detection category can be broadened to an operable softmax. ; In the formula, Φ represents the operation set corresponding to malicious traffic detection and classification, and x represents the input tensor. k-th level operation Architecture parameters, For the k-th level operation Architecture parameters, Classification for malicious traffic detection targeting x.

Images