Original message binary feature extraction method and system based on time sequence convolutional network

Abstract

The invention discloses an original message binary feature extraction method based on a time sequence convolutional network. The method comprises the following steps: generating an embedded vector sequence of a message; constructing a time sequence convolutional network architecture; pre-training and classifier training are carried out through self-supervised pre-training, supervised fine tuning and multi-task learning; model interpretation is carried out, and quantized behavior features are extracted based on interpretation results; converting the features into a Snort / Suricata rule format, and establishing a mapping relation from the features to original message segments; and integration to an IDS / IPS engine is realized. The invention also provides an original message binary feature extraction system based on the time sequence convolutional network. A closed loop from feature discovery to automatic rule deployment is constructed, and the detection response efficiency and accuracy of complex network threats are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to a method and system for extracting binary features of raw packets based on temporal convolutional networks. Background Technology

[0002] Binary feature extraction from raw network packets is fundamental to network security analysis. Current mainstream methods primarily rely on manually designed protocol parsers and static rule matching, such as IDS / IPS systems like Snort. However, this approach has significant limitations.

[0003] First, it heavily relies on the experience of security experts, and the writing and maintenance of the rule base is costly. Second, the rule-matching nature of the system makes it difficult to effectively detect unknown threats such as advanced persistent threats, or to identify complex temporal behavioral patterns across multiple packets. In recent years, although some research has attempted to introduce machine learning, it has mostly focused on traffic statistics, losing key byte-level information in the payload; or it has adopted "black box" models, whose decision-making process is not transparent, and the analysis results are difficult to translate into detection rules that can be executed by traditional security devices, resulting in a huge obstacle between AI capabilities and actual defense capabilities. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method and system for extracting binary features from raw messages based on temporal convolutional networks.

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or to describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for extracting binary features from raw messages based on temporal convolutional networks, comprising:

[0008] The acquired raw message is converted into a byte sequence. By constructing byte value embedding, position encoding and protocol field embedding, the message embedding vector sequence is generated and then standardized to form a fixed-length representation.

[0009] A temporal convolutional network architecture is constructed, which uses causal convolution, dilated convolution, residual connections and attention mechanisms to extract and fuse features from the standardized embedded vector sequence.

[0010] The extracted quantized behavioral features are converted into Snort / Suricata rule format, and a mapping relationship between the features and the original message bytes is established.

[0011] By developing standardized interfaces and plugins, the analytical capabilities of temporal convolutional network architectures and the generated rules are integrated into the IDS / IPS engine.

[0012] Furthermore, the process of converting the acquired raw message into a byte sequence includes: capturing a complete raw message containing Ethernet frame, network layer header, transport layer header and payload in real time; parsing the captured message into a sequentially arranged byte stream while maintaining the original binary format of the message; adding boundary markers to each complete raw message and recording the timestamp when the raw message was captured to construct a timing relationship.

[0013] Furthermore, the process of generating the embedding vector sequence of the message by constructing byte value embedding, position encoding, and protocol field embedding includes: mapping each byte value in the byte sequence to a dense vector to form a byte value embedding; adding position encoding according to the position of each byte in the original message to distinguish the protocol header and payload areas; constructing protocol field embedding for the known protocol fields in the original message; and concatenating the byte value embedding, position encoding, and protocol field embedding to form the final embedding vector sequence.

[0014] Furthermore, the standardization process includes: setting a fixed length L, truncating embedding vector sequences with a length exceeding L, and padding embedding vector sequences with a length less than L; generating a mask matrix corresponding to the fixed length L to identify the effective byte positions and padding positions in the embedding vector sequences.

[0015] Furthermore, the constructed temporal convolutional network architecture includes: causal convolutional layers, dilated convolutional layers, residual blocks, and feature fusion layers;

[0016] The causal convolutional layer performs one-dimensional convolution operations through unidirectional padding, ensuring that the output at time t depends only on the input at time t and before it.

[0017] The dilated convolutional layer uses an exponentially growing sequence of dilation factors to expand the receptive field in order to capture long-term dependencies in the embedding vector sequence of the message.

[0018] Each residual block contains layer normalization, a nonlinear activation function, dilated causal convolution, and a random deactivation operation, and the residual block introduces a skip connection to add the input to the output after convolution;

[0019] The feature fusion layer employs an attention mechanism to weightedly fuse features from each layer.

[0020] Furthermore, after constructing the temporal convolutional network architecture, the process further includes: pre-training and classifier training through self-supervised pre-training, supervised fine-tuning, and multi-task learning; interpreting the trained temporal convolutional network architecture to identify key temporal patterns, and extracting quantified behavioral features based on the interpretation results.

[0021] Furthermore, the process of converting the extracted quantized behavioral features into Snort / Suricata rule format and establishing a mapping relationship between the features and the original message bytes includes:

[0022] Byte pattern rule generation: Extract key byte sequence patterns from the features of the temporal convolutional network architecture, convert the binary pattern into a hexadecimal representation, establish a mapping relationship between features and message byte offsets, and generate detection rules based on payload content;

[0023] Protocol field feature rule generation: Based on the embedded features of protocol fields, extract abnormal patterns of protocol header fields and generate threshold rules based on field values;

[0024] Embedded vector feature rule generation: Analyze the clustering patterns of embedded vectors, generate detection rules based on vector similarity, and convert anomaly scores into threshold rules;

[0025] Timing binary pattern rule generation: Extract timing-related binary patterns to generate detection rules across multiple packets.

[0026] Furthermore, the process of converting the extracted quantized behavioral features into Snort / Suricata rule format and establishing a mapping relationship between the features and the original message bytes also includes:

[0027] Feature-to-byte position mapping: Establish a mapping table from the features of the temporal convolutional network architecture to the byte positions of the original message, calculate the byte offset and length corresponding to the features, generate byte masks to identify important byte positions, and verify the accuracy of the generation rules on the original message;

[0028] Protocol anomaly rule generation process: Detect abnormal behavior at the protocol level, generate TCP flag anomaly detection rules, and generate corresponding detection rules based on payload characteristics;

[0029] Packet field mapping: Establish the mapping relationship between rule conditions and packet field offsets, locate the positions of specific fields in the IP header and TCP header, calculate the byte offset of feature patterns in the payload, and verify the accuracy of the generated rules on actual packets.

[0030] Secondly, the present invention also provides a system for extracting binary features of raw messages based on temporal convolutional networks, comprising:

[0031] The data acquisition and preprocessing layer is used to convert the acquired raw messages into byte sequences. By constructing byte value embeddings, position encodings, and protocol field embeddings, it generates a sequence of message embedding vectors and performs standardization processing to form a fixed-length representation.

[0032] The model building layer is used to construct the temporal convolutional network architecture, which uses causal convolution, dilated convolution, residual connections and attention mechanisms to extract and fuse features from the standardized embedded vector sequence.

[0033] The rule generation layer is used to convert the extracted quantized behavioral features into Snort / Suricata rule format and establish a mapping relationship between features and original message bytes.

[0034] The engine integration layer is used to integrate the analysis capabilities of temporal convolutional network architectures and the generated rules into the IDS / IPS engine by developing standardized interfaces and plugins.

[0035] Furthermore, the original message binary feature extraction system based on temporal convolutional networks also includes: a model training layer, used for pre-training and classifier training through self-supervised pre-training, supervised fine-tuning and multi-task learning; and a behavior pattern analysis layer, used for interpreting the trained temporal convolutional network architecture to identify key temporal patterns and extract quantified behavior features based on the interpretation results.

[0036] The beneficial effects of this invention are as follows: it learns and extracts complex temporal behavioral features from the original binary sequence of packets, transforms the features into interpretable and traceable Snort / Suricata rules through model interpretation technology, and directly deploys them into existing IDS / IPS engines with the help of standardized interfaces and plugins, thus building a closed loop from feature discovery to automated rule deployment, which significantly improves the efficiency and accuracy of detection and response to complex network threats. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating a method for extracting binary features from raw messages based on a temporal convolutional network, according to the present invention. Detailed Implementation

[0039] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0040] like Figure 1 As shown in some illustrative embodiments, this application provides a method for extracting binary features of raw messages based on temporal convolutional networks, including the following steps:

[0041] 101: Raw message preprocessing and embedding vector construction.

[0042] Specifically, the original message is acquired and converted into a byte sequence. By constructing byte value embedding, position encoding and protocol field embedding, an embedding vector sequence of the message is generated. The embedding vector sequence is then standardized to form a fixed-length representation, which further forms a fixed-length message vector sequence.

[0043] The process of obtaining the original message and converting it into a byte sequence includes the following steps:

[0044] By setting up a listener on the network interface, complete raw network packets—that is, unprocessed network data packets—are captured in real time. The captured raw packets include Ethernet frames, network layer headers, transport layer headers, and payloads. The captured raw packets are parsed into a sequentially ordered byte stream, preserving the original binary format. For example, the first 14 bytes of an Ethernet frame are converted into 14 integer values ​​between 0 and 255. This process maintains the original binary format without any decoding or character set conversion, thus preserving all the original bit pattern information of the protocol fields. This facilitates the identification of attacks that exploit specific bit patterns.

[0045] Add boundary markers to each complete original message, for example, by inserting special markers before and after the byte stream, or by recording the start and end indices of each message in the data structure. This allows for the accurate segmentation of each independent message entity from a continuous byte stream. At the same time, record the timestamp when the original message is captured to build a temporal relationship, that is, to string isolated messages together into a stream with a temporal order, so that the model built later can learn the interaction patterns between messages.

[0046] The process of generating the message's embedding vector sequence by constructing byte value embeddings, position encodings, and protocol field embeddings includes the following steps:

[0047] Each byte value in a byte sequence is mapped to a high-dimensional dense vector, forming a byte value embedding. A network packet is essentially a sequence of bytes from 0 to 255. Byte value embedding uses an embedding matrix to map each of the 256 byte values ​​to a high-dimensional dense vector, such as 64-dimensional or 128-dimensional, allowing the model to learn the complex relationships between bytes in a continuous vector space. Position encoding is added based on the position of each byte in the original packet to distinguish between the protocol header and payload regions. The meaning of a byte is highly correlated with its position in the packet; therefore, this application generates a position encoding vector for each byte position. This position encoding vector can be added to or concatenated with the byte value embedding, so that the final embedding representation simultaneously contains both "what this byte is" and "where this byte is." Specifically, the position encoding of the protocol header and payload regions can be designed with different patterns to help the model distinguish between these two parts.

[0048] For known protocol fields in the original message, specialized protocol field embeddings are constructed. For known protocols, such as IP and TCP, their header fields have clear semantics, such as source IP, destination port, and TTL. If only byte values ​​and positional encoding are relied upon, the model would require a significant amount of computational power to parse the meaning of these fields. Therefore, this application constructs specialized protocol field embeddings for known protocol fields in the original message: First, key protocol fields in the message are identified, such as source IP address, destination IP address, source port, destination port, TCP flags, and TTL. Then, one or more dedicated protocol field embedding vectors are appended to the identified fields or their constituent bytes. For example, all four bytes belonging to the "source IP address" will have a semantic vector of "source IP" appended to them, in addition to their own byte value embedding and positional encoding. This enhances the model's semantic understanding ability, allowing it to focus on key information more quickly. The byte value embedding, positional encoding, and protocol field embedding vectors are concatenated to form the final embedding vector sequence. This embedding vector sequence has higher dimensionality and richer information, and can fully express the byte content, structural position, and protocol semantics of the original message.

[0049] The process of standardizing the embedded vector sequence to form a fixed-length representation includes:

[0050] Set a fixed length L, truncate embedding vector sequences with a length exceeding L, and fill embedding vector sequences with a length less than L with zero vectors until the length reaches L. By performing pruning or zero-padding operations on the embedding vector sequences, they are unified to a fixed length L, avoiding numerical deviations from affecting training.

[0051] A mask matrix corresponding to a fixed length L is generated to identify the positions of valid bytes and padding in the embedding vector sequence. Simple padding can introduce invalid information and potentially mislead the model; therefore, this application generates a binary vector. For example, a value of 1 indicates that the corresponding embedding vector contains valid packet data, while a value of 0 indicates that the corresponding embedding vector contains invalid padding data. Given the significant variation in network packet lengths, standardizing them to a fixed length facilitates subsequent batch processing.

[0052] In step 101, the raw, unstructured binary network packets of varying lengths are transformed into a structured, information-rich, and fixed-length numerical representation. This is achieved through three processes: extraction of the original packet byte sequence, construction of multi-dimensional embedding vectors, and standardization of the packet sequence. This process transforms the original network packets into a unified numerical representation, providing a data foundation for the model. By integrating byte values, location information, and semantic knowledge of key protocol fields, the model's feature representation capabilities are enhanced, and the learning difficulty is reduced. Furthermore, through standardization, computational efficiency is ensured, parallel processing of the model is supported, and the model's generalization ability is improved. Compared to traditional detection methods that rely on fixed rules and signatures, this method is more effective in dealing with complex scenarios such as unknown attacks, protocol variants, and encrypted traffic.

[0053] 102: Construct a temporal convolutional network architecture, that is, construct a temporal convolutional network model with causality and multi-scale receptive field to model the embedding vector sequence of messages.

[0054] Specifically, a temporal convolutional network architecture is constructed, which employs causal convolution, dilated convolution, residual connections, and attention mechanisms to extract and fuse features from the standardized message embedding vector sequence. The constructed temporal convolutional network architecture includes: causal convolutional layers, dilated convolutional layers, residual blocks, and feature fusion layers.

[0055] The causal convolutional layer performs one-dimensional convolution operations through unidirectional padding, ensuring that the output at time t depends only on the input at time t and earlier, avoiding the use of future information. For example, for a convolutional kernel of size 3, when processing data at position t, it only uses the data at t-2, t-1, and t, without involving the data at t+1. This ensures that the model can be deployed in online traffic monitoring systems requiring real-time responses without data leakage due to the use of future information. Simultaneously, the causal convolutional layer is designed to support multi-channel input, simultaneously receiving and processing the fused vector generated in step 101, which contains byte values, positional encoding, and protocol field embeddings. This allows the model to adaptively learn the relationships between different channels. Furthermore, the causal convolutional layer is designed to adaptively adjust the convolutional kernel size to adapt to different protocol modes.

[0056] Dilated convolutional layers employ an exponentially growing sequence of dilation factors to expand the receptive field, capturing long-term dependencies in the embedding vector sequence of a message. Specifically, dilated convolutional layers exponentially expand the receptive field by inserting holes into the convolutional kernel, enabling the network to capture long-term dependencies. The dilation rate increases layer by layer, such as 1, 2, 4, 8, etc., so that the receptive field expands exponentially with each layer of depth. The dilation rate can also be dynamically adjusted. Furthermore, different layers of dilated convolutional layers use different dilation rates, allowing shallower networks to focus on local, short-term patterns, while deeper networks capture macroscopic, long-term patterns. Dilated convolutional layers can model long-term dependencies across entire message sequences and even across message sequences with a very small number of layers and parameters, and can simultaneously identify anomalous behaviors across different time spans.

[0057] As network layers deepen, vanishing gradients and training difficulties become major technical problems. The design of residual blocks can create a pathway for information and gradient flow. Each residual block contains layer normalization, non-linear activation functions, dilated causal convolutions, and random deactivation operations to prevent overfitting. Furthermore, residual blocks introduce skip connections, adding the input to the output after convolution, ensuring that gradients are losslessly propagated to shallower layers during backpropagation, thus mitigating the vanishing gradient problem. Preferably, the residual block design incorporates a gating mechanism to control information flow, dynamically controlling how much original information can flow through.

[0058] The feature fusion layer employs an attention mechanism to weightedly fuse features from different levels. After processing through multiple convolutional layers and residual blocks, the network obtains feature maps at different levels and scales. The feature fusion layer uses an attention mechanism to calculate an attention weight for each feature vector. This weight represents the importance of the feature to the final judgment. For example, when detecting DDoS attacks, the model might assign a higher weight to "SYN packet frequency." Finally, all feature vectors are weighted and summed according to their weights, aggregating into a single global temporal feature vector. The attention mechanism enables the model to automatically learn and focus on the most discriminative part of the packet embedding vector sequence, ignoring irrelevant background information, thereby improving the accuracy of feature representation.

[0059] Residual connections are added to each convolutional block to alleviate the vanishing gradient problem and support deeper network structures. To capture features of the message across different semantic dimensions, the feature fusion layer designs parallel convolutional branches using convolutional kernels of different sizes and dilation rates to capture patterns at different time scales. The features from multiple branches are then fused and aggregated into a global temporal feature vector, which serves as input for subsequent tasks. Causal convolutional layers, dilated convolutional layers, residual blocks, and the feature fusion layer together construct a feature extraction engine capable of mining deep temporal patterns and anomalous behaviors from the binary sequence of the original message that are difficult to detect using traditional methods.

[0060] 103: Model pre-training and classifier training are specifically carried out through self-supervised pre-training, supervised fine-tuning, and multi-task learning.

[0061] The embedded vector sequence of the message generated in step 101 is input into the temporal convolutional network model constructed in step 102. Based on the following three-stage training strategy, the model can go from initially understanding traffic patterns to distinguishing between normal and abnormal traffic, and finally accurately identifying a variety of threats.

[0062] Phase 1: Self-supervised pre-training.

[0063] Self-supervised pre-training is performed on large-scale unlabeled traffic data, that is, the embedding vector sequence of the packets generated in step 101 is input into the model, and the following self-supervised task is designed:

[0064] The masking reconstruction task involves randomly masking a portion of the packet embedding vectors in an input sequence, requiring the model to predict the original packet embedding vector at the masked location based on the undressed context vector. To accomplish this, the model's dilated convolutions need to understand long-range dependencies; for example, to predict a TCP packet, the model might need to review the three-way handshake information from several previous packets. This forces the model to delve deeper into the temporal correlations of the traffic sequence, learning deeper features such as protocol state machines and session context.

[0065] Traffic prediction task: Take a continuous sequence of packets as input and the model is required to predict one or more packets that follow it. The model needs to output a prediction vector that is as close as possible to the actual packet embedding vector. This task can directly train the model's ability to model traffic.

[0066] Phase Two: Oversight-based fine-tuning.

[0067] Based on the pre-trained model, labeled data of normal / abnormal traffic is used to train the model specifically, enabling it to distinguish between normal and abnormal traffic and complete the binary classification task. Specifically, a binary classifier is added to the end of the model. The embedding vector sequence of labeled messages is input into the model, the cross-entropy loss between the model's output anomalous probability and the true label is calculated, and the model weights are updated using the backpropagation algorithm.

[0068] Phase 3: Multi-task learning.

[0069] The goal is to further expand the capabilities of the fine-tuned model, enabling it not only to determine anomalies but also to identify specific attack types and communication protocols, thereby improving the model's generalization ability. More refined labeled data than in the supervised fine-tuning phase is used; in addition to normal / anomaly labels, the data includes attack type labels such as DDoS, scanning, brute-force attacks, and Trojan communication, as well as protocol identification labels such as HTTP, HTTPS, DNS, SSH, and FTP. The training process involves paralleling multiple task-specific classification heads after the model's shared backbone network: Head 1, an anomaly detection binary classifier; Head 2, an attack type multi-classifier; and Head 3, a protocol identification multi-classifier. During training, for each input sample, the model simultaneously outputs predictions for all three tasks and calculates three independent loss functions. The final total loss is a weighted sum of these three losses, and the model optimizes by minimizing this total loss.

[0070] 104: Model interpretation and feature extraction, which involves interpreting the trained temporal convolutional network architecture to identify key temporal patterns and extracting quantified behavioral features based on the interpretation results, thereby transforming the model's decision-making process into traceable and verifiable logic.

[0071] Among them, interpreting the trained temporal convolutional network architecture includes at least one of the following methods: First, analyze the trained convolutional kernel weights to understand the temporal patterns learned by the temporal convolutional network architecture. By analyzing the weight matrix, the specific temporal patterns that the convolutional kernels are sensitive to can be identified.

[0072] The second method involves analyzing the weight distribution of the attention mechanism in the temporal dimension of the temporal convolutional network architecture to locate the key time period or key message that contributes the most to the classification decision. The third method involves calculating the gradient of the classification result with respect to the input temporal sequence to identify the time period and feature dimension that contributes the most to the detection result. The fourth method involves using the SHAP method to calculate the importance score of each time step and feature dimension.

[0073] Among them, the extracted and quantified behavioral features include at least one of the following: Burst pattern feature extraction: Identify burst behaviors based on key time periods and calculate statistical indicators for that time period, including: burst start timestamp, duration, average data transmission rate, and peak rate. The above features can provide a measure for quantifying DoS / DDoS attacks or large-scale data leakage behaviors.

[0074] Periodic pattern feature extraction: Based on the periodic patterns in the model interpretation results, periodic behaviors are detected, and key indicators such as period length, phase shift, and intensity of periodic events are extracted to detect covert threats such as C&C communication heartbeats and timed tasks.

[0075] Abnormal timing feature extraction: Based on the model's interpretation of the differences between normal and abnormal traffic, identify timing patterns that significantly deviate from the normal baseline, such as abnormal packet intervals or irregular transmission patterns; Protocol behavior feature extraction: Based on the model's interpretation of specific protocol fields, generate features describing abnormal protocol interactions, such as: abnormal flag combinations during TCP connection establishment / termination, abnormal User-Agent distribution or URL path patterns in HTTP requests, and abnormal domain name length or character distribution in DNS queries.

[0076] 105: Generation of binary feature mapping to Snort / Suricata rules, which involves converting the extracted quantized behavioral features into Snort / Suricata rule format and establishing a mapping relationship between features and original message bytes, specifically including:

[0077] Byte pattern rule generation: Key byte sequence patterns are extracted from the features of the temporal convolutional network architecture. The binary patterns are converted into hexadecimal representations, and a mapping relationship between features and message byte offsets is established to generate detection rules based on payload content. Specifically, based on the keyword byte sequence extracted in step 104 that contributes the most to the detection results, the binary patterns are directly converted into Snort's content field, solidifying the model's "discovery" into a precise payload signature.

[0078] Protocol field feature rule generation: Based on the protocol field embedding features, extract the abnormal patterns of the protocol header fields and generate threshold rules based on the field values. Specifically, based on the protocol field anomalies and statistical feature thresholds extracted in step 104, for protocol field anomalies, generate corresponding field detection rules; for statistical features, generate rules based on threshold or detection_filter according to the threshold calculated in step 104.

[0079] Embedded vector feature rule generation: Analyze the clustering patterns of embedded vectors, generate detection rules based on vector similarity, and convert anomaly scores into threshold rules.

[0080] Timing binary pattern rule generation: Extract timing-related binary patterns to generate detection rules across multiple packets. Specifically, based on the abnormal timing behaviors across packets and sessions identified in step 104, advanced options such as Snort's flowbits or event_filter are used to construct state detection rules.

[0081] Feature-to-byte position mapping: A mapping table is established between the features of the temporal convolutional network architecture and the byte positions of the original message. The byte offset and length corresponding to the features are calculated, and a byte mask is generated to identify important byte positions. The accuracy of the generated rules is then verified on the original message. Specifically, using the message parsing index established in step 101 and the attribution results interpreted by the model in step 104, a mapping table is constructed between the features of the temporal convolutional network architecture and the original byte offsets. For example, when the model indicates that "the 45th dimension of the embedding vector" is a key feature, this mapping table shows that this dimension corresponds to the 4 bytes starting from the 120th byte of the original message payload. Based on this, byte mask rules are generated for the Snort rules.

[0082] Protocol anomaly rule generation process: Detect abnormal behavior at the protocol level, generate TCP flag anomaly detection rules, and generate corresponding detection rules based on payload characteristics.

[0083] Packet Field Mapping: This involves establishing a mapping relationship between rule conditions and packet field offsets, locating specific fields in the IP and TCP headers, calculating the byte offsets of characteristic patterns in the payload, and verifying the accuracy of the generated rules on actual packets. Specifically, for protocol header fields, their offset positions in the IP and TCP headers can be located. The generated rules are then reapplied to the original packet sample that triggered the rule for simulated testing. If the rule matches successfully, the mapping is accurate; if it does not match, the rule parameters are adjusted or the packet is marked for manual review.

[0084] 106: IDS / IPS engine integration, which means integrating the analysis capabilities of temporal convolutional network architecture and the rules generated in step 106 into the IDS / IPS engine, i.e., the mainstream intrusion detection / prevention system, by developing standardized interfaces and plugins.

[0085] To achieve compatibility with IDS / IPS engines from different vendors and versions, a standardized interface needs to be designed. This interface is typically provided in the form of an API or SDK, defining standardized data input / output formats, function call methods, and error handling mechanisms. The input interface receives standardized message data from the IDS / IPS engine, while the output interface returns structured analysis results, such as anomaly scores, extracted features, or triggered alarm information.

[0086] A Snort plugin was developed to integrate analysis capabilities for temporal convolutional network architectures. Specifically, Snort captures network traffic and decodes it into an internal data structure. This traffic data is then passed to the Snort plugin, which integrates the temporal convolutional network architecture analysis functionality. The plugin internally calls the embedding vector generation logic in step 101 to convert the packets into a model-recognizable format. The plugin feeds the embedding vector sequence into a trained model for real-time inference. The model's anomaly score or classification result is returned to the plugin. Based on a preset threshold, if an anomaly is detected, the plugin generates a Snort alert event. This event can include rich contextual information such as the AI ​​model's confidence level and key features, and is recorded in the Snort log. This allows Snort to maintain its original rule-based detection capabilities while gaining AI detection capabilities for unknown threats and complex temporal patterns.

[0087] Suricata's integration strategy is to directly extend its rule engine to support time-series feature detection. For example, a keyword can be designed, and rules containing that keyword can be written. When Suricata processes HTTP traffic, it feeds the relevant message sequences into the model, which calculates anomaly scores in real time. If a threshold is exceeded, an alarm is triggered. By integrating AI capabilities into Suricata's native rule language, analysts can flexibly invoke AI models for detection, just like writing traditional rules.

[0088] Through steps 101-103, complex temporal behavioral features can be automatically learned and extracted from the original binary sequence of packets. Compared to the limitations of traditional methods that rely on manually defined rules, this application can capture deep patterns of unknown threats and advanced attacks. Through model interpretation in step 104, abstract AI features are transformed into interpretable and traceable Snort / Suricata rules, ensuring the transparency and credibility of the detection results. Finally, through step 106, standardized interfaces and plugins are used to directly deploy the AI ​​capabilities into existing IDS / IPS engines, achieving the integration of AI capabilities with existing defense infrastructure.

[0089] This application constructs a closed loop from feature discovery to automated rule deployment, which not only enables the defense system to respond quickly to new threats, but also realizes the mining of historical data through the established mapping relationship and batch analysis function, thereby improving the detection and response efficiency and accuracy of complex network threats. It upgrades the traditional IDS / IPS from a passive rule enforcer to a proactive, intelligent, and self-evolving security defense node.

[0090] This application also provides a system for extracting binary features from raw messages based on temporal convolutional networks, including:

[0091] The data acquisition and preprocessing layer is used to convert the acquired raw messages into byte sequences. By constructing byte value embeddings, position encodings, and protocol field embeddings, it generates a sequence of message embedding vectors and performs standardization processing to form a fixed-length representation.

[0092] The model building layer is used to construct a temporal convolutional network architecture. This temporal convolutional network architecture uses causal convolution, dilated convolution, residual connections, and attention mechanisms to extract and fuse features from the standardized embedded vector sequence.

[0093] The model training layer is used for pre-training and classifier training through self-supervised pre-training, supervised fine-tuning, and multi-task learning.

[0094] The behavior pattern analysis layer is used to interpret the trained temporal convolutional network architecture to identify key temporal patterns and extract quantified behavioral features based on the interpretation results.

[0095] The rule generation layer is used to convert the extracted quantized behavioral features into Snort / Suricata rule format and establish a mapping relationship between features and original message bytes.

[0096] The engine integration layer is used to integrate the analysis capabilities of temporal convolutional network architectures and the generated rules into the IDS / IPS engine by developing standardized interfaces and plugins.

[0097] The data acquisition and preprocessing layer includes:

[0098] The conversion module is used to capture complete raw packets containing Ethernet frames, network layer headers, transport layer headers, and payloads in real time. It parses the captured packets into a sequentially arranged byte stream, preserving the original binary format of the packets. Boundary markers are added to each complete raw packet, and a timestamp is recorded when the raw packets are captured, establishing a timing relationship. The generation module maps each byte value in the byte sequence to a dense vector, forming a byte value embedding. Position encoding is added based on the position of each byte in the raw packet to distinguish between protocol headers and payload areas. Protocol field embeddings are constructed for known protocol fields in the raw packet. The byte value embeddings, position encodings, and protocol field embeddings are concatenated to form the final embedding vector sequence.

[0099] A standard module is used to set a fixed length L, truncate embedding vector sequences with a length exceeding L, and pad embedding vector sequences with a length less than L; and generate a mask matrix corresponding to the fixed length L to identify the positions of valid bytes and padding positions in the embedding vector sequence.

[0100] The constructed temporal convolutional network architecture includes: causal convolutional layers, dilated convolutional layers, residual blocks, and feature fusion layers. The causal convolutional layers perform one-dimensional convolution operations through unidirectional padding, ensuring that the output at time t depends only on the input at time t and earlier. The dilated convolutional layers expand the receptive field using an exponentially growing sequence of dilation factors to capture long-term dependencies in the embedding vector sequence of the message. Each residual block contains layer normalization, a non-linear activation function, dilated causal convolution, and random deactivation operations. The residual blocks also introduce skip connections to add the input to the output after convolution. The feature fusion layer uses an attention mechanism to weightedly fuse features from each layer.

[0101] The rule generation layer includes:

[0102] The byte pattern rule generation module is used to extract key byte sequence patterns from the features of the self-temporal convolutional network architecture, convert the binary pattern into a hexadecimal representation, establish a mapping relationship between features and message byte offsets, and generate detection rules based on payload content.

[0103] The protocol field feature rule generation module is used to extract abnormal patterns in protocol header fields based on protocol field embedding features and generate threshold rules based on field values.

[0104] The embedded vector feature rule generation module is used to analyze the clustering patterns of embedded vectors, generate detection rules based on vector similarity, and convert anomaly scores into threshold rules.

[0105] The timing binary pattern rule generation module is used to extract timing-related binary patterns and generate detection rules across multiple packets.

[0106] The feature-to-byte position mapping module is used to establish a mapping table from the features of the temporal convolutional network architecture to the byte positions of the original message, calculate the byte offset and length corresponding to the features, generate a byte mask to identify important byte positions, and verify the accuracy of the generation rules on the original message.

[0107] The protocol anomaly rule generation module is used to detect abnormal behavior at the protocol level, generate TCP flag anomaly detection rules, and generate corresponding detection rules based on payload characteristics.

[0108] The packet field mapping module is used to establish the mapping relationship between rule conditions and packet field offsets, locate the position of specific fields in the IP header and TCP header, calculate the byte offset of feature patterns in the payload, and verify the accuracy of the generated rules on actual packets.

[0109] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for extracting binary features from original messages based on temporal convolutional networks, characterized in that, include: The acquired raw message is converted into a byte sequence. By constructing byte value embedding, position encoding and protocol field embedding, the message embedding vector sequence is generated and then standardized to form a fixed-length representation. A temporal convolutional network architecture is constructed, which uses causal convolution, dilated convolution, residual connections and attention mechanisms to extract and fuse features from the standardized embedded vector sequence. The extracted quantized behavioral features are converted into Snort / Suricata rule format, and a mapping relationship between the features and the original message bytes is established. By developing standardized interfaces and plugins, the analytical capabilities of temporal convolutional network architectures and the generated rules are integrated into the IDS / IPS engine.

2. The method for extracting binary features of original messages based on temporal convolutional networks according to claim 1, characterized in that, The process of converting the acquired raw message into a byte sequence includes: Capture complete raw packets containing Ethernet frames, network layer headers, transport layer headers, and payloads in real time; parse the captured packets into sequentially arranged byte streams while preserving the original binary format of the packets; add boundary markers to each complete raw packet and record the timestamp when the raw packet was captured to build a timing relationship.

3. The method for extracting binary features of original messages based on temporal convolutional networks according to claim 2, characterized in that, The process of generating the message embedding vector sequence by constructing byte value embedding, position encoding, and protocol field embedding includes: Each byte value in the byte sequence is mapped to a dense vector to form a byte value embedding; a position code is added according to the position of each byte in the original message to distinguish the protocol header and payload areas; protocol field embeddings are constructed for the known protocol fields in the original message; the byte value embeddings, position codes and protocol field embeddings are concatenated to form the final embedding vector sequence.

4. The method for extracting binary features of original messages based on temporal convolutional networks according to claim 3, characterized in that, The standardization process includes: setting a fixed length L, truncating embedding vector sequences with a length exceeding L, and padding embedding vector sequences with a length less than L; generating a mask matrix corresponding to the fixed length L to identify the positions of valid bytes and padding positions in the embedding vector sequence.

5. The method for extracting binary features of original messages based on temporal convolutional networks according to claim 4, characterized in that, The constructed temporal convolutional network architecture includes: causal convolutional layers, dilated convolutional layers, residual blocks, and feature fusion layers; The causal convolutional layer performs one-dimensional convolution operations through unidirectional padding, ensuring that the output at time t depends only on the input at time t and before it. The dilated convolutional layer uses an exponentially growing sequence of dilation factors to expand the receptive field in order to capture long-term dependencies in the embedding vector sequence of the message. Each residual block contains layer normalization, a nonlinear activation function, dilated causal convolution, and a random deactivation operation, and the residual block introduces a skip connection to add the input to the output after convolution; The feature fusion layer employs an attention mechanism to weightedly fuse features from each layer.

6. The method for extracting binary features of original messages based on temporal convolutional networks according to claim 5, characterized in that, Following the construction of the temporal convolutional network architecture, the following is also included: Pre-training and classifier training are performed through self-supervised pre-training, supervised fine-tuning, and multi-task learning. The trained temporal convolutional network architecture is interpreted to identify key temporal patterns, and quantified behavioral features are extracted based on the interpretation results.

7. The method for extracting binary features of original messages based on temporal convolutional networks according to claim 6, characterized in that, The process of converting the extracted quantized behavioral features into Snort / Suricata rule format and establishing a mapping relationship between the features and the original message bytes includes: Byte pattern rule generation: Extract key byte sequence patterns from the features of the temporal convolutional network architecture, convert the binary pattern into a hexadecimal representation, establish a mapping relationship between features and message byte offsets, and generate detection rules based on payload content; Protocol field feature rule generation: Based on the embedded features of protocol fields, extract abnormal patterns of protocol header fields and generate threshold rules based on field values; Embedded vector feature rule generation: Analyze the clustering patterns of embedded vectors, generate detection rules based on vector similarity, and convert anomaly scores into threshold rules; Timing binary pattern rule generation: Extract timing-related binary patterns to generate detection rules across multiple packets.

8. The method for extracting binary features of original messages based on temporal convolutional networks according to claim 7, characterized in that, The process of converting the extracted quantized behavioral features into the Snort / Suricata rule format and establishing a mapping relationship between the features and the original message bytes further includes: Feature-to-byte position mapping: Establish a mapping table from the features of the temporal convolutional network architecture to the byte positions of the original message, calculate the byte offset and length corresponding to the features, generate byte masks to identify important byte positions, and verify the accuracy of the generation rules on the original message; Protocol anomaly rule generation process: Detect abnormal behavior at the protocol level, generate TCP flag anomaly detection rules, and generate corresponding detection rules based on payload characteristics; Packet field mapping: Establish the mapping relationship between rule conditions and packet field offsets, locate the positions of specific fields in the IP header and TCP header, calculate the byte offset of feature patterns in the payload, and verify the accuracy of the generated rules on actual packets.

9. A system for extracting binary features from raw messages based on temporal convolutional networks, characterized in that, include: The data acquisition and preprocessing layer is used to convert the acquired raw messages into byte sequences. By constructing byte value embeddings, position encodings, and protocol field embeddings, it generates a sequence of message embedding vectors and performs standardization processing to form a fixed-length representation. The model building layer is used to construct the temporal convolutional network architecture, which uses causal convolution, dilated convolution, residual connections and attention mechanisms to extract and fuse features from the standardized embedded vector sequence. The rule generation layer is used to convert the extracted quantized behavioral features into Snort / Suricata rule format and establish a mapping relationship between features and original message bytes. The engine integration layer is used to integrate the analysis capabilities of temporal convolutional network architectures and the generated rules into the IDS / IPS engine by developing standardized interfaces and plugins.

10. The system for extracting binary features of raw messages based on temporal convolutional networks according to claim 9, characterized in that, Also includes: The model training layer is used for pre-training and classifier training through self-supervised pre-training, supervised fine-tuning, and multi-task learning. The behavior pattern analysis layer is used to interpret the trained temporal convolutional network architecture to identify key temporal patterns and extract quantified behavioral features based on the interpretation results.

Images