A network covert communication detection method and system based on neighborhood transfer learning

Abstract

The application discloses a network covert communication detection method and system based on neighborhood transfer learning. The method first collects abnormal traffic and normal traffic from an open source data set as source domain data, generates a unified traffic representation vector after grouping, parsing and feature extraction, and pre-trains a detection model based on a recurrent neural network using the representation vector. Then, a small amount of covert communication traffic and background traffic is captured in the actual target network, and micro-call feature vectors are generated according to the same processing method. Next, the pre-trained model is loaded, and the model is fine-tuned using a small amount of target domain samples to obtain a special detection model adapted to the target environment. Finally, the fine-tuned model is deployed to the real network environment for real-time detection of covert communication behavior. The application can quickly build a high-precision detection model under small sample conditions, meeting the timeliness requirements of security emergency response.

Description

Technical Field

[0001] This invention relates to a network covert communication detection method based on neighborhood transfer learning, and also to a corresponding network covert communication detection system, belonging to the field of network security communication technology. Background Technology

[0002] Covert network communication has become a significant evasion tactic for cyberattacks, espionage, and illegal network activities. Attackers commonly employ techniques such as encryption obfuscation, tunneling, and traffic masquerading to conceal malicious communication traffic within normal network operations, bypassing traditional firewalls, WAFs, and intrusion prevention systems. Recent real-world attack and network security operation cases demonstrate that criminals utilize video conferencing tunnels, multi-layered nested encrypted tunnels, and cloud function interface masquerading to conduct covert infiltration and espionage. Traditional detection methods based on signature string matching are completely ineffective, leaving only artificial intelligence and traffic analysis techniques for identification. However, existing intelligent detection models generally suffer from strong data dependency, requiring thousands or even tens of thousands of labeled traffic samples for training. Sample collection, cleaning, and labeling can take weeks, making it difficult to adapt to the rapidly evolving nature of network threats.

[0003] In cybersecurity emergency response scenarios, the industry strictly adheres to the "golden hour" principle, requiring the establishment and deployment of covert communication detection capabilities within one hour. However, in real-world attack and defense scenarios, the number of malicious samples available is extremely limited, typically only a single digit. Existing deep learning and machine learning models are constrained by this small sample size and cannot converge effectively. Furthermore, mainstream transfer learning solutions often convert network traffic into natural language modalities for pre-training and fine-tuning. A significant domain gap exists between traffic data and text semantics, requiring hundreds of samples to achieve acceptable detection accuracy, which fails to meet the critical need for small samples and rapid deployment in emergency responses.

[0004] Chinese invention patent CN110336806B discloses a covert communication detection method that combines session behavior and communication relationships. This method uses a session flow aggregation algorithm to extract features, enabling the obtained feature vectors to describe the characteristics of a single session flow while also characterizing the similarity between session flows, thus providing a better description of the detection target from the perspective of session behavior. By combining communication relationship data under a single time window and communication data under multiple time windows, the evaluation metrics can overcome the limitations of attack feature description under a single time window, thus providing a better description of the detection target from the perspective of communication relationships. However, the above technical solution still suffers from reliance on massive labeled samples, low accuracy in small sample emergency scenarios, and inability to meet the rapid deployment requirements of the critical one-hour window for security emergency response.

[0005] To address the prominent shortcomings of existing technologies, such as reliance on massive amounts of labeled data, long training cycles, and low accuracy in small sample scenarios, which make them difficult to adapt to the timeliness requirements of security emergency response, there is an urgent need to propose a brand-new migration detection scheme to break through the bottleneck of data scarcity and achieve high-precision identification of various network covert communication behaviors in a short time. Summary of the Invention

[0006] The primary technical problem to be solved by this invention is to provide a method for detecting covert network communication based on neighborhood transfer learning.

[0007] Another technical problem to be solved by the present invention is to provide a network covert communication detection system based on neighborhood transfer learning.

[0008] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution: According to a first aspect of the present invention, a method for detecting covert network communication based on neighborhood transfer learning is provided, comprising the following steps: S1: Collect abnormal traffic and normal traffic from open source datasets as source domain pre-training data; the abnormal traffic is used to enable the pre-trained model to learn the basic discriminative ability to distinguish between abnormal network behavior and normal network behavior; S2: Perform packet grouping based on five-tuples, parsing traffic element attributes group by group, and multi-attribute fusion processing on the collected source domain traffic to generate a traffic representation vector with a unified dimension. S3: Use the traffic representation vector to divide the training set, validation set and test set, construct a lightweight pre-trained model based on recurrent neural network, determine the optimal hyperparameters through grid search, and save the pre-trained model and its parameters after completing the model pre-training on the source domain data. S4: Capture covert communication traffic and background traffic in the target network environment as fine-tuning data for the target domain; S5: The captured target domain traffic is processed in the same way as in step S2, including grouping, parsing, and feature extraction, to generate a micro-call feature vector with the same format as the traffic representation vector. S6: Load the pre-trained model saved in step S3, use the parameters of the pre-trained model as the initialization parameters of the target domain model, use the micro-call feature vector generated in step S5 to fine-tune the target domain model with a small sample, and save the fine-tuned target domain model and its parameters; wherein, the source domain and the target domain are both network traffic binary classification tasks, and the source domain is an anomaly detection task and the target domain is a covert communication detection task. The two tasks have inherent similarity in traffic discrimination features and constitute a neighborhood migration relationship; S7: Deploy the fine-tuned target domain model to the target network environment to detect and identify covert communication behaviors in real-time traffic.

[0009] Preferably, step S2 includes the following sub-steps: S21: Perform packet preprocessing on the original network traffic based on the five-tuple rule, wherein the five-tuple includes the source IP address, destination IP address, source port, destination port, and transport layer protocol; S22: Analyze and extract traffic element attribute features group by group; S23: The extracted multiple traffic element attributes are fused to construct a standardized traffic representation vector.

[0010] Preferably, in step S22, packet length, packet time, and packet direction information of the data stream are extracted in units of groups; Packet length refers to the length of the payload of each data packet in the session, which reflects the load characteristics of data transmission; packet time refers to the arrival time of each data packet or the time interval between adjacent data packets, which reflects the timing characteristics of traffic; packet direction refers to the transmission direction of each data packet, where the value sent from the client to the server is uplink, and the value sent from the server to the client is downlink.

[0011] Preferably, in step S23, it is assumed that a session has a total of For the data packet, for the Data packets :make Indicates the first The payload length of each data packet; let Indicates the first The transmission direction of each data packet, among which Indicates upward movement. Indicates downward movement; command Indicates the first The arrival time of the first data packet or the time of arrival of the second data packet The time interval of each data packet, and ; The signed packet-length sequence is defined as: Define the time series of the package as: Finally, the long sequence of signed packets and the packet time series are concatenated to form a complete representation vector of the traffic:

[0012] in, It is a signed packet-long sequence; It is a time series; Representing vectors belong 3D real space.

[0013] Preferably, step S3 includes the following sub-steps: S31: Divide the traffic representation vector dataset generated in step S2 into three disjoint subsets: training set, validation set, and test set. The training set is used for learning and updating model parameters, the validation set is used for selecting hyperparameters and monitoring overfitting, and the test set is used for unbiased evaluation of model performance. S32: A lightweight pre-trained model structure is built based on a gated recurrent unit network. The pre-trained model is used to learn the basic discriminative features between abnormal traffic and normal traffic. S33: Define the search space for hyperparameters required during model pre-training; S34: During the pre-training process, save the best-performing model and its parameters on the validation set, and output a general format file for loading in the subsequent fine-tuning stage.

[0014] Preferably, step S6 includes the following sub-steps: S61: Divide the micro-call feature vector generated in step S5 into a training set and a test set according to a preset ratio; S62: Load the pre-trained model and its parameters saved in step S3, and use the parameters of the pre-trained model as the initial parameters of the target domain model; S63: Based on the hyperparameters of the pre-trained model, the cross-entropy loss function is used to fine-tune the target domain model; wherein, the source domain is an anomaly detection task, and the target domain is a covert communication detection task. The two tasks have inherent similarity in traffic discrimination features, forming a neighborhood transfer relationship, thereby enabling the target domain model to converge quickly under small sample conditions and preventing overfitting. S64: Save the fine-tuned target domain model and its parameters, as well as the model's performance metrics on the test set, for subsequent actual detection deployment.

[0015] Preferably, in step S63, the fine-tuning objective is to optimize the following objective function:

[0016] in, This is the loss function used during model fine-tuning. For target domain model parameters; The number of training samples for the target domain; The prediction function of the target domain model is... Use the parameters to predict the input sample; For the first The flow representation vector of each target domain training sample; For the first The true labels of the training samples in the target domain.

[0017] According to a second aspect of the present invention, a network covert communication detection system based on neighborhood transfer learning is provided, comprising a processor and a memory; wherein the memory is coupled to the processor and is used to store a computer program, which, when executed by the processor, enables the processor to implement the network covert communication detection method described above.

[0018] Compared with existing technologies, this invention utilizes a pioneering neighborhood transfer learning technique to effectively transfer accumulated knowledge in the field of anomaly detection to the task of detecting covert network communication, significantly reducing the dependence on the number of labeled samples in the target domain. Based on the general anomaly discrimination capability obtained through pre-training on open-source datasets, only a very small number of covert communication samples are needed for fine-tuning to quickly build a dedicated model with high detection accuracy. This fundamentally overcomes the inherent limitation of traditional methods, which rely on massive training data and cannot adapt to small-sample emergency scenarios. Furthermore, because the pre-trained model provides good parameter initialization, the fine-tuning process converges rapidly. The complete process from capturing real-world network traffic to deploying the model can be completed in a very short time, fully meeting the practical timeliness requirements of the "golden hour" of network security emergency response, and maintaining high detection accuracy even under small sample conditions. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a network covert communication detection method based on neighborhood transfer learning in the first embodiment of the present invention. Figure 2 This is a schematic diagram of a network covert communication detection system based on neighborhood transfer learning, as shown in the second embodiment of the present invention. Detailed Implementation

[0020] The technical content of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0021] First Embodiment like Figure 1 As shown, the first embodiment of the present invention provides a network covert communication detection method based on neighborhood transfer learning, which includes at least the following steps: S1: Collect abnormal traffic and normal traffic from open source datasets as source domain pre-training data; the abnormal traffic is used to enable the pre-trained model to learn the basic discriminative ability to distinguish between abnormal network behavior and normal network behavior; S2: Perform packet grouping based on five-tuples, parsing traffic element attributes group by group, and multi-attribute fusion processing on the collected source domain traffic to generate a traffic representation vector with a unified dimension. S3: Use the traffic representation vector to divide the training set, validation set and test set, construct a lightweight pre-trained model based on recurrent neural network, determine the optimal hyperparameters through grid search, and save the pre-trained model and its parameters after completing the model pre-training on the source domain data. S4: Capture covert communication traffic and background traffic in the target network environment as fine-tuning data for the target domain; S5: The captured target domain traffic is processed in the same way as in step S2, including grouping, parsing, and feature extraction, to generate a micro-call feature vector with the same format as the traffic representation vector. S6: Load the pre-trained model saved in step S3, use the parameters of the pre-trained model as the initialization parameters of the target domain model, use the micro-call feature vector generated in step S5 to fine-tune the target domain model with a small sample, and save the fine-tuned target domain model and its parameters; wherein, the source domain and the target domain are both network traffic binary classification tasks, and the source domain is an anomaly detection task and the target domain is a covert communication detection task. The two tasks have inherent similarity in traffic discrimination features and constitute a neighborhood migration relationship; S7: Deploy the fine-tuned target domain model to the target network environment to detect and identify covert communication behaviors in real-time traffic.

[0022] In one embodiment of the present invention, step S2 includes the following sub-steps: S21: Perform packet preprocessing on the original network traffic based on the five-tuple rule, wherein the five-tuple includes the source IP address, destination IP address, source port, destination port, and transport layer protocol; S22: Analyze and extract traffic element attribute features group by group; S23: The extracted multiple traffic element attributes are fused to construct a standardized traffic representation vector.

[0023] In one embodiment of the present invention, step S3 includes the following sub-steps: S31: Divide the traffic representation vector dataset generated in step S2 into three disjoint subsets: training set, validation set, and test set. The training set is used for learning and updating model parameters, the validation set is used for selecting hyperparameters and monitoring overfitting, and the test set is used for unbiased evaluation of model performance. S32: A lightweight pre-trained model structure is built based on a gated recurrent unit network. The pre-trained model is used to learn the basic discriminative features between abnormal traffic and normal traffic. S33: Define the search space for hyperparameters required during model pre-training; S34: During the pre-training process, save the best-performing model and its parameters on the validation set, and output a general format file for loading in the subsequent fine-tuning stage.

[0024] In one embodiment of the present invention, step S6 includes the following sub-steps: S61: Divide the micro-call feature vector generated in step S5 into a training set and a test set according to a preset ratio; S62: Load the pre-trained model and its parameters saved in step S3, and use the parameters of the pre-trained model as the initial parameters of the target domain model; S63: Based on the hyperparameters of the pre-trained model, the cross-entropy loss function is used to fine-tune the target domain model; wherein, the source domain is an anomaly detection task, and the target domain is a covert communication detection task. The two tasks have inherent similarity in traffic discrimination features, forming a neighborhood transfer relationship, thereby enabling the target domain model to converge quickly under small sample conditions and preventing overfitting. S64: Save the fine-tuned target domain model and its parameters, as well as the model's performance metrics on the test set, for subsequent actual detection deployment.

[0025] The following is a detailed explanation of the specific implementation steps of the above-mentioned method for detecting covert network communication: In step S1, two types of data are mainly collected: abnormal traffic and normal traffic.

[0026] This invention systematically reviewed publicly available datasets in the field of abnormal traffic detection through web searches and academic search engines. The search results show that there is a large amount of open-source data in this field that can be used to pre-train transfer learning models. For example, the ISCXTor2016 dataset contains approximately 20GB of dark web traffic, the CTU-13 dataset contains over 50GB of malicious traffic, and the FlowPrint-Andrubis dataset contains approximately 120GB of normal traffic.

[0027] In step S2, the raw network traffic is grouped, parsed, and its features are extracted to generate a unified traffic feature vector. Specifically: In step S21, the collected raw network traffic is typically an offline pcap file, which contains disorganized data packets. To effectively extract the information contained in the traffic and further form a representation vector that can be used for model learning, this invention first performs grouping processing on the traffic. Specifically, all data packets with the same 5-tuple are grouped into the same group. The 5-tuple includes the source IP address, destination IP address, source port, destination port, and transport layer protocol.

[0028] In step S22, metadata of the data stream is extracted on a packet-by-packet basis. Specifically, this invention extracts the following three traffic metadata attributes: packet length, packet time, and packet direction. Packet length refers to the length of the payload of each data packet in a session, measured in bytes, and is used to reflect the load characteristics of data transmission; packet time refers to the arrival time of each data packet or the time interval between adjacent data packets, measured in milliseconds, and is used to reflect the temporal characteristics of the traffic; packet direction refers to the transmission direction of each data packet, where the value from the client to the server is uplink, and the value from the server to the client is downlink.

[0029] In step S23, the packet length and packet direction are merged to form a signed integer sequence; the packet time is processed into a positive floating-point sequence. The specific construction method is as follows: Suppose there are a total of For the data packet, for the Data packets :make Indicates the first The payload length of each data packet (in bytes); let Indicates the first The transmission direction of each data packet, among which Indicates upward movement. Indicates downward movement; command Indicates the first The arrival time of the first data packet or the time of arrival of the second data packet The time interval between data packets, in milliseconds, and .

[0030] The signed packet-length sequence is defined as: Define the time series of the package as: Finally, the two sequences are concatenated to form a complete representation vector of the flow:

[0031] in, It is a signed packet-long sequence; It is a time series; Representing vectors belong 3D real space.

[0032] In step S3, the processed feature vectors are used to partition the dataset, design the model structure and hyperparameter search space, pre-train the intermediate model, and save it. Specifically: In step S31, the traffic representation vector dataset constructed in step S2 is divided into three non-overlapping subsets: training set, validation set, and test set.

[0033] The training set is used for learning and updating model parameters; the validation set is used for selecting model hyperparameters and monitoring overfitting; and the test set is used for unbiased evaluation of the final model performance.

[0034] In one embodiment of the present invention, the three subsets are divided in the following proportions: 70% for the training set, 20% for the validation set, and 10% for the test set.

[0035] In step S32, a recurrent neural network model structure based on gated recurrent units (GRUs) is adopted, with the following specific configuration: the input layer dimension matches the representation vector dimension 2n; taking n=50 as an example, the input dimension is 100. The first GRU layer has a hidden state dimension of 128 and outputs the complete sequence; the second GRU layer has a hidden state dimension of 64 and outputs only the last time step. Dropout layers with a ratio of 0.5 are added between the two GRU layers and after the output to prevent overfitting.

[0036] The model then connects two fully connected layers: the first layer maps the 64-dimensional features to 32 dimensions (ReLU activation function), and the output layer maps the 32-dimensional features to 2 dimensions (Softmax activation function), outputting the probability distributions of anomalies and normal features. The model has approximately 127,000 parameters, making it a lightweight structure suitable for rapid pre-training and fine-tuning.

[0037] In step S33, the hyperparameter search space required during model training is defined. Specifically, this includes: the learning rate, which uses a log-uniform distribution, and the search range is... Batch size, candidate values ​​are 16, 32, 64 or 128; number of training epochs, ranging from 10 to 100; optimizer type, selectable are Adam, SGD or RMSprop; regularization coefficient, mainly refers to L1 or L2 weight decay coefficient; dropout ratio, ranging from 0.2 to 0.5.

[0038] The aforementioned hyperparameter space will serve as the basis for subsequent grid search hyperparameter tuning strategies.

[0039] In step S34, the model with the best performance on the validation set is saved. Specifically, after each training epoch, the model's performance metrics on the validation set are evaluated, such as the F1 score or accuracy. If the current epoch's performance is better than the historical best, the current model parameters are saved. After training, the saved content includes at least: model weight parameters, model architecture configuration, hyperparameter settings, and input feature dimension information. The model is saved in the common HDF5 (.h5) format for easy loading and fine-tuning in subsequent step S6.

[0040] Step S4 belongs to the security emergency response phase. During the emergency response process, this invention captures traffic data for fine-tuning the pre-trained model in a real network environment, including two categories: covert communication traffic and background traffic. Covert communication traffic refers to the traffic generated by covert communication tools. Specifically, this invention runs malicious tools in a controlled environment to collect covert communication traffic as positive samples in the fine-tuning phase. In the target network environment where the detector is to be deployed, this invention collects background traffic, i.e., normal traffic, as negative samples in the fine-tuning phase.

[0041] In step S5, the traffic captured from the real network is grouped, parsed, and its meta-attributes are extracted to generate representation vectors for model fine-tuning. The specific processing procedure of step S5 is exactly the same as that of step S2, and will not be repeated here.

[0042] It is particularly important to emphasize that the flow representation vectors generated in steps S2 and S5 are in the same format to ensure that the input data of the pre-trained model and the fine-tuned model maintain the same distribution, thereby ensuring that the model can be fine-tuned normally and work effectively.

[0043] In step S6, the source domain model pre-trained in step S3 is loaded, and the target domain representation vector generated in step S5 is used for model fine-tuning. Hyperparameters are adjusted, and the final model is saved. Based on the fundamental principles of neighborhood transfer learning, the source domain (anomaly detection domain) and the target domain (covert communication detection domain) have inherent similarities in traffic classification tasks, enabling the feature representations learned in the source domain to be effectively transferred to the target domain. Therefore, only a small number of target domain samples are needed to complete model adaptation. Specifically: In step S61, the representation vector dataset generated in step S5 is divided into two disjoint subsets: a training set and a test set. The training set is used for learning and updating model parameters; the test set is used for evaluating the final model performance. The partition ratio is 90% for the training set and 10% for the test set. Due to the total number of samples in the target domain... The number of training samples is extremely limited, usually only a few to dozens. This method of partitioning ensures that the training samples are as numerous as possible while still effectively verifying the model's generalization ability.

[0044] In step S62, the source domain model pre-trained and saved in step S3 is loaded. , its parameters As the target domain model The initial parameters.

[0045] According to the neighborhood transfer learning theory, the feature representations between the source domain and the target domain are approximately equivalent. Therefore, the source domain model already has the basic ability to distinguish between abnormal and normal traffic, and can be directly used as the starting point for the target domain model without having to start training from scratch.

[0046] In step S63, fine-tuning is performed using the target domain training set based on the loaded source domain model. The hyperparameter space during fine-tuning remains consistent with the hyperparameter space defined in step S33. According to the principle of neighborhood transfer learning, the goal of fine-tuning is to optimize the following objective function:

[0047] in, This is the loss function used for model fine-tuning; specifically, the present invention uses the cross-entropy loss function. For target domain model parameters; The number of training samples for the target domain; The prediction function of the target domain model is... Use the parameters to predict the input sample; For the first The flow representation vector of each target domain training sample; For the first The true labels of the training samples in the target domain.

[0048] Because the source domain model provides good parameter initialization ,in, These are the initial parameters for the target domain model; To fine-tune the learned parameter updates, the target domain model only needs a small number of iterations to converge.

[0049] Furthermore, the knowledge learned from the source domain also acts as a regularization constraint on the target domain, effectively preventing overfitting under small sample conditions.

[0050] In step S64, after fine-tuning, the present invention saves the final target domain model. and its parameters The saved content includes at least: model weight parameters, model architecture configuration, hyperparameter settings, and performance metrics on the test set. This model already possesses the ability to detect covert communication traffic in the target network environment and will be used for actual deployment and detection in subsequent step S7.

[0051] In step S7, the finely tuned detection model is deployed to detect and identify covert communication behaviors in a real-world network environment. Specifically, during long-term security incident response, steps S4 to S6 only need to be repeated to quickly update detection capabilities against new covert communication tools. Over time, a series of detectors targeting different covert communication tools will be gradually deployed in the network environment. These detectors work together to ultimately form a comprehensive defense system against covert network communication.

[0052] The following is a comparative analysis of relevant experiments in the embodiments of the present invention: In this embodiment of the invention, DoHC2 was selected as the detection target in the experiment. The reason for choosing this software is that DoHC2 uses DoH (DNS over HTTPS) covert channel construction technology to achieve covert C2 communication, which has been widely used in network attack simulation, red team penetration testing, and mainstream penetration testing software Cobalt Strike. Its implementation process has important reference value for detecting other covert network communication software.

[0053] In step S1 of this embodiment of the invention, two types of data, abnormal traffic and normal traffic, are collected for model pre-training.

[0054] In this embodiment of the invention, relevant traffic was downloaded from an open-source database and cleaned. The final pre-training data is shown in Table 1.

[0055] Table 1 Summary of Pre-training Traffic Data

[0056] To effectively learn the general characteristics of abnormal traffic, this embodiment of the invention maintains a balanced ratio between abnormal traffic and normal traffic during data processing, with each containing 50,000 traffic records.

[0057] For the selection of normal traffic, this embodiment of the invention uses website access, file transfer, email, and other related traffic from the ISCXTor2016 and FlowPrint open-source datasets as representative data. For the selection of abnormal traffic, this embodiment of the invention focuses on its similarity to covert network channels, ultimately selecting covert communication tool (SSR and Vmess) traffic from the LFETT2021 dataset, malware traffic from the CTU-13 dataset, and Tor traffic from the ISCXTor2016 dataset as sources of abnormal traffic.

[0058] The processing of the above four open-source libraries in this embodiment of the invention took about three weeks, with the processing of the FlowPrint dataset being the most time-consuming, taking nearly two weeks on its own.

[0059] The main reason is that the FlowPrint dataset contains over 2 million compressed files, a small portion of which only contain traffic dump files, while most of the compressed files are nested within deep folders, increasing the processing difficulty. Only after decompressing all these compressed files can the required traffic data be obtained.

[0060] After cleaning, this embodiment of the invention ultimately used one-quarter of the FlowPrint data for pre-training.

[0061] Subsequent experiments showed that the amount of data was sufficient for the model to achieve the desired detection accuracy.

[0062] In step S2 of this embodiment of the invention, the normal traffic and abnormal traffic collected in step S1 are processed.

[0063] In this embodiment of the invention, all data packets with the same 5-tuple are grouped into the same group, where the 5-tuple includes the source IP address, destination IP address, source port, destination port, and transport layer protocol. Each group represents a complete network session. A group must contain 40 or more data packets to be considered a valid group. This is because only when a group contains a sufficient number of data packets can the network behavior characteristics inherent in the traffic be fully reflected. Groups with fewer than 40 data packets are discarded; for groups with more than 40 data packets, the first 40 data packets are truncated and processed uniformly.

[0064] For each valid packet, this embodiment of the invention fuses the packet length and packet direction to form a signed integer sequence; the packet time is processed into a positive floating-point sequence. Specifically, the packet direction is +1 (uplink) or -1 (downlink), multiplied by the packet length to obtain the signed packet length; the packet time is the arrival time interval of adjacent data packets or an absolute timestamp, both of which are positive numbers. This embodiment of the invention concatenates the above signed packet length sequence and packet time sequence to form a complete representation vector of the traffic. This embodiment of the invention stores the representation vector in CSV format. In the CSV table, each row corresponds to the representation result of one data packet.

[0065] Specifically, in this embodiment of the invention, the first 40 data packets are selected for representation for each session. Each data packet corresponds to two features (packet length and packet time), therefore, the dimension of the final representation vector generated for each session is:

[0066] The first 40 dimensions are signed packet-length sequences. The last 40 dimensions are time series data. This 80-dimensional vector serves as the input feature for subsequent model training.

[0067] In step S3, this embodiment of the invention uses the CSV format traffic characterization data generated in step S2 to pre-train the detection model.

[0068] In this embodiment of the invention, normal traffic is set as negative samples and abnormal traffic is set as positive samples. The model's task is binary classification, that is, to determine whether the input traffic belongs to the normal category or the abnormal category. The deep learning model in this embodiment of the invention is developed based on the TensorFlow framework and uses Keras API as the high-level interface to facilitate rapid prototyping and model iteration.

[0069] Through grid search and experimental verification, the hyperparameters determined in this embodiment of the invention are as follows: input dimension 80, two-layer GRU (the first layer outputs the complete sequence in 128 dimensions, and the second layer outputs only the last time step in 64 dimensions), dropout ratio 0.5, fully connected layer mapping 64 dimensions to 32 dimensions (ReLU), output layer mapping to 2 dimensions (Softmax), batch size 32, number of training epochs 50, learning rate 0.001, optimizer Adam, and loss function is cross-entropy.

[0070] After pre-training, the model in this embodiment of the invention achieved a detection accuracy of approximately 98% on the test set, indicating that the model has fully learned the distinguishing features between abnormal and normal traffic. After training, this embodiment of the invention saves the pre-trained model and its parameters for subsequent model fine-tuning during the security emergency response phase.

[0071] In step S4, this embodiment of the invention collects DoHC2 traffic and background traffic in the deployment environment based on the detection target DoHC2 set above, for model fine-tuning.

[0072] To collect the covert communication traffic generated by DoHC2, this embodiment of the invention runs a DoHC2 client to establish C2 communication with the CobaltStrike server. Specifically, this embodiment of the invention starts a DoHC2 tunnel in a controlled environment, encapsulates the C2 traffic into HTTPS requests using the DoH protocol, and sends them to the DoH server.

[0073] To avoid excessive load on the DoH server or trigger rate limits due to frequent requests, this embodiment of the invention adopts a method of manually controlling the runtime and actively pausing data collection when connection anomalies or a significant increase in response latency are observed.

[0074] During the data collection process, this embodiment of the invention sets the filter condition to tcpport443 in Wireshark to save only HTTPS traffic, because DoHC2's C2 communication is based on the HTTPS protocol.

[0075] Meanwhile, the embodiments of the present invention record the traffic generated during the DoHC2 tunnel establishment, heartbeat maintenance and command issuance stages as positive samples for the fine-tuning stage.

[0076] To collect background traffic, i.e. normal traffic, in the deployment environment, this embodiment of the invention runs mainstream browsers, including Chrome, Firefox, and Edge, in the target network environment where the detector is to be deployed.

[0077] This invention collects the following types of traffic: web chat, online shopping, online meetings, streaming media, and remote desktop traffic.

[0078] After the data collection is completed, this embodiment of the invention stores the two types of traffic as PCAP files, and names and archives them according to the traffic type to facilitate the extraction of traffic attributes in the subsequent step S5.

[0079] After the above data collection process, this embodiment of the invention ultimately collected 1,132 DoHC2 traffic entries and 4,576 background traffic entries.

[0080] In step S5, the present invention grouped, parsed and extracted meta-attributes of the DoHC2 traffic captured in the real network and the background traffic to generate feature vectors for fine-tuning.

[0081] The processing flow is completely consistent with the method for handling normal and abnormal traffic in step S2 above, including sub-steps such as traffic grouping, effective group filtering, feature extraction, sequence concatenation, and CSV format storage. To avoid repetition, it will not be described in detail here.

[0082] After processing in step S5, including effective packet filtering and data cleaning, this embodiment of the invention obtained 1,100 DoHC2 covert communication traffic as positive samples and approximately 4,300 background HTTPS traffic as negative samples. This number is slightly less than the original collection amount, mainly because some packets had less than 40 data packets and were therefore removed.

[0083] In step S6, this embodiment of the invention randomly selects samples from the data processed in step S5 for model fine-tuning.

[0084] In the fine-tuning data, normal traffic, i.e., negative samples, is always maintained at 500. The number of covert communication traffic, i.e., positive samples, is determined based on the set. The different values ​​of are varied to simulate different levels of sample shortage scenarios.

[0085] To avoid randomness in the experimental results, the detection method was trained 10 times under different positive sample sizes. The data used for each fine-tuning was randomly sampled from the data processed by S5.

[0086] This embodiment of the invention saves the model after each training iteration and uses dedicated test data to evaluate detection accuracy. This test data contains 1,000 traffic samples, 500 positive and 500 negative. The test data and fine-tuning data are completely independent and do not overlap. The data collection process is the same as described in step S4, with data collected separately. This embodiment of the invention uses the average test accuracy of 10 experiments as the basis for comparing detection performance.

[0087] Furthermore, this embodiment of the invention selects the covert communication detection method (hereinafter referred to as DomEye) described in the paper "DomEye: Detecting network covert channel of domain fronting with throughput fluctuation" ((2024) Computers & Security Vol.144, DOI: 10.1016 / j.cose.2024.103976) as the comparison object. DomEye uses a random forest model, and the traffic features extracted have strong discriminative power, thus having a low dependence on training data. Therefore, this embodiment of the invention selects DomEye as the comparison object, which is technically reasonable.

[0088] As shown in Table 2, the embodiments of the present invention compare the accuracy of DomEye and the present invention in detecting DoHC2 covert communication under different positive sample numbers.

[0089] Table 2 Comparison of the accuracy of DoHC2 detection

[0090] Experimental results show that the detection accuracy of this invention is significantly better than DomEye under various sample size conditions. Specifically, when the number of positive samples is 1, DomEye's accuracy is 50.00%, while this invention reaches 88.40%, which is 38.40 percentage points higher. As the number of positive samples increases, the accuracy of both methods shows an upward trend, but the improvement of this invention is more significant. When the number of positive samples is 5, DomEye's accuracy is 67.00%, while this invention reaches 98.40%; when the number of positive samples increases to 10, DomEye's accuracy is 77.80%, while this invention's accuracy reaches as high as 99.60%.

[0091] The above results show that the present invention has significant advantages under small sample conditions and can effectively address the problem of scarce malicious samples in security emergency response. In contrast, the comparison method is less effective when the sample is sparse and cannot meet the needs of practical applications.

[0092] In step S7, this embodiment of the invention deploys the fine-tuned model and evaluates whether the time consumption of the entire process meets the "golden hour" principle for security emergency response. Specifically, the time statistics cover the entire process from real-world network traffic capture, traffic processing, model fine-tuning to model deployment.

[0093] The time-testing environment was built on a personal computer. This computer's configuration is similar to that of typical security operations personnel, equipped with 16GB of RAM, an Intel i7 CPU, and the Python-based TensorFlow deep learning framework. During testing, this embodiment of the invention primarily focuses on the DoHC2 traffic collection time, model fine-tuning time, and detection accuracy. The evaluation process for detection accuracy is consistent with the previous description, and the calculation methods for collection time and fine-tuning time are as follows: The calculation of traffic collection time starts from searching the internet for the DoHC2 tool and Cobalt Strike configuration tutorials using a search engine, and ends with storing the DoHC2 traffic as a pcap file. Since the DoHC2 tool is open-source on platforms like GitHub, and Cobalt Strike, as a mainstream penetration testing framework, has comprehensive documentation, this embodiment of the invention successfully completed tool download, environment configuration, and DoH server parameter settings in approximately 20 minutes. While security operations personnel may require more time, considering the popularity of the tools, this embodiment of the invention believes the process will not exceed 35 minutes. Subsequently, this embodiment of the invention starts Wireshark packet capture software, sets traffic filtering conditions, runs the DoHC2 client to establish C2 communication with the Cobalt Strike server, and saves the captured traffic as a pcap file. This process takes approximately 1.5 minutes, with DoHC2 tunnel establishment and heartbeat transmission taking approximately 5 seconds. Although the initial configuration and packet capture environment initialization may be time-consuming, once the environment is ready, the time for a single DoHC2 traffic capture can be reduced to approximately 40 seconds. When the positive sample size n=1, the total time for flow collection is approximately 36.5 minutes. The time required to further increase the sample size can be expressed as follows: minute.

[0094] The calculation of model fine-tuning time begins with opening the Python development tool. In this embodiment of the invention, the entire process, from starting the development tool to processing the DoHC2 traffic and background traffic in the pcap file into an 80-dimensional representation vector, takes approximately 3 minutes.

[0095] After obtaining the representation vector, this embodiment of the invention fine-tunes the model pre-trained in step S3. Although with the increase in the number of positive samples... With the increase in sample size, the training time increases slightly, but the increase is controlled within 1 second, meaning that the model fine-tuning time is basically unaffected by the sample size. The total fine-tuning time calculated in this embodiment of the invention is approximately 7 minutes, of which traffic processing takes 3 minutes and model training takes 4 minutes.

[0096] The final security emergency response time equals the sum of the DoHC2 traffic collection time and the model fine-tuning time. The time statistics cover the entire process from real-world traffic capture (step S4), traffic processing (step S5), model fine-tuning (step S6) to model deployment (step S7). Specific response times for different sample sizes are shown in Table 3.

[0097] Table 3 Security Response Time for DoHC2 Tools

[0098] Experimental results show that the present invention maintains excellent detection accuracy under different positive sample quantities. When the number of positive samples reaches 10, the detection accuracy for DoHC2 covert communication exceeds 99.5%. Regarding response time, although the overall processing time increases slightly with the increase in sample size, the overall response time can be controlled within 45 minutes in most cases. According to the "golden hour" emergency response principle commonly used in the cybersecurity industry, the detection system must form an effective detection capability against new covert communication tools within one hour. Based on the experimental data on both detection accuracy and response time, the network covert communication detection method provided by this invention fully meets the dual requirements of detection performance and rapid deployment in security emergency response scenarios.

[0099] Second Embodiment Based on the aforementioned methods for detecting covert network communication, the second embodiment of this invention provides a network covert communication detection system based on neighborhood transfer learning. For example... Figure 2 As shown, the network covert communication detection system includes one or more processors and a memory. The memory is coupled to the processor and is used to store computer programs. When the computer program is executed by the processor, the network covert communication detection method described in the above embodiments can be implemented.

[0100] The processor controls the overall operation of the network covert communication detection system to complete all or part of the steps described above. The processor can be a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processing (DSP) chip, etc. The memory stores various types of data to support the operation of the network covert communication detection system. This data may include, for example, instructions for any application or method operating on the network covert communication detection system, as well as application-related data. The memory can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, etc.

[0101] In one exemplary embodiment, the network covert communication detection system can be implemented by a computer chip or physical entity, or by a product with certain functions, to perform the above-described methods and achieve the same technical effects. A typical embodiment is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interface device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0102] In another exemplary embodiment, the present invention also provides a computer-readable storage medium including program instructions that, when executed by a processor, implement the steps of the method in any of the above embodiments. For example, the computer-readable storage medium may be the memory including the program instructions described above, which can be executed by a processor to complete the above-described network covert communication detection method and achieve the same technical effects as the above-described network covert communication detection method.

[0103] It should be noted that the above embodiments are merely illustrative examples, and the technical solutions of each embodiment can be combined, all of which are within the protection scope of this invention.

[0104] The network covert communication detection method and system based on neighborhood transfer learning provided by this invention have been described in detail above. Any obvious modifications made by those skilled in the art without departing from the essence of this invention will constitute an infringement of the patent rights of this invention and will incur corresponding legal liability.

Claims

1. A method for detecting covert network communication based on neighborhood transfer learning, characterized in that... Includes the following steps: S1: Collect abnormal traffic and normal traffic from open source datasets as source domain pre-training data; the abnormal traffic is used to enable the pre-trained model to learn the basic discriminative ability to distinguish between abnormal network behavior and normal network behavior; S2: Perform packet grouping based on five-tuples, parsing traffic element attributes group by group, and multi-attribute fusion processing on the collected source domain traffic to generate a traffic representation vector with a unified dimension. S3: Use the traffic representation vector to divide the training set, validation set and test set, construct a lightweight pre-trained model based on recurrent neural network, determine the optimal hyperparameters through grid search, and save the pre-trained model and its parameters after completing the model pre-training on the source domain data. S4: Capture covert communication traffic and background traffic in the target network environment as fine-tuning data for the target domain; S5: The captured target domain traffic is processed in the same way as in step S2, including grouping, parsing, and feature extraction, to generate a micro-call feature vector with the same format as the traffic representation vector. S6: Load the pre-trained model saved in step S3, use the parameters of the pre-trained model as the initialization parameters of the target domain model, use the micro-call feature vector generated in step S5 to fine-tune the target domain model with a small sample, and save the fine-tuned target domain model and its parameters; wherein, the source domain and the target domain are both network traffic binary classification tasks, and the source domain is an anomaly detection task and the target domain is a covert communication detection task. The two tasks have inherent similarity in traffic discrimination features and constitute a neighborhood migration relationship; S7: Deploy the fine-tuned target domain model to the target network environment to detect and identify covert communication behaviors in real-time traffic.

2. The network covert communication detection method as described in claim 1, characterized in that... Step S2 includes the following sub-steps: S21: Perform packet preprocessing on the original network traffic based on the five-tuple rule, wherein the five-tuple includes the source IP address, destination IP address, source port, destination port, and transport layer protocol; S22: Analyze and extract traffic element attribute features group by group; S23: The extracted multiple traffic element attributes are fused to construct a standardized traffic representation vector.

3. The network covert communication detection method as described in claim 2, characterized in that... In step S22, packet length, packet time, and packet direction information of the data stream are extracted in units of groups. Packet length refers to the length of the payload of each data packet in the session, which reflects the load characteristics of data transmission; packet time refers to the arrival time of each data packet or the time interval between adjacent data packets, which reflects the timing characteristics of traffic; packet direction refers to the transmission direction of each data packet, where the value sent from the client to the server is uplink, and the value sent from the server to the client is downlink.

4. The network covert communication detection method as described in claim 2, characterized in that... In step S23, it is assumed that there are a total of [number] sessions. For the data packet, for the Data packets :make Indicates the first The payload length of each data packet; let Indicates the first The transmission direction of each data packet, among which Indicates upward movement. Indicates downward movement; make Indicates the first The arrival time of the first data packet or the time of arrival of the second data packet The time interval of each data packet, and ; The signed packet-length sequence is defined as: Define the time series of the package as: Finally, the long sequence of signed packets and the packet time series are concatenated to form a complete representation vector of the traffic: in, It is a signed packet-long sequence; It is a time series; Representing vectors belong 3D real space.

5. The network covert communication detection method as described in claim 1, characterized in that... Step S3 includes the following sub-steps: S31: Divide the traffic representation vector dataset generated in step S2 into three disjoint subsets: a training set, a validation set, and a test set. The training set is used for learning and updating model parameters, the validation set is used for selecting hyperparameters and monitoring overfitting, and the test set is used for unbiased evaluation of model performance. S32: A lightweight pre-trained model structure is built based on a gated recurrent unit network. The pre-trained model is used to learn the basic discriminative features between abnormal traffic and normal traffic. S33: Define the search space for hyperparameters required during model pre-training; S34: During the pre-training process, save the best-performing model and its parameters on the validation set, and output a general format file for loading in the subsequent fine-tuning stage.

6. The network covert communication detection method as described in claim 1, characterized in that... Step S6 includes the following sub-steps: S61: Divide the micro-call feature vector generated in step S5 into a training set and a test set according to a preset ratio; S62: Load the pre-trained model and its parameters saved in step S3, and use the parameters of the pre-trained model as the initial parameters of the target domain model; S63: Based on the hyperparameters of the pre-trained model, the cross-entropy loss function is used to fine-tune the target domain model; wherein, the source domain is an anomaly detection task, and the target domain is a covert communication detection task. The two tasks have inherent similarity in traffic discrimination features, forming a neighborhood transfer relationship, thereby enabling the target domain model to converge quickly under small sample conditions and preventing overfitting. S64: Save the fine-tuned target domain model and its parameters, as well as the model's performance metrics on the test set, for subsequent actual detection deployment.

7. The network covert communication detection method as described in claim 6, characterized in that... In sub-step S63, the goal of fine-tuning is to optimize the following objective function: in, This is the loss function used for model fine-tuning; For target domain model parameters; The number of training samples for the target domain; The prediction function of the target domain model is... Use the parameters to predict the input sample; For the first The flow representation vector of each target domain training sample; For the first The true labels of the training samples in the target domain.

8. A network covert communication detection system based on neighborhood transfer learning, characterized in that... It includes a processor and a memory; wherein the memory is coupled to the processor and is used to store a computer program, which, when executed by the processor, causes the processor to implement the network covert communication detection method according to any one of claims 1 to 7.

Images