Traffic sequence processing method based on non-negative kernel function self-attention mechanism

Abstract

The application discloses a traffic sequence processing method based on a non-negative kernel function self-attention mechanism and belongs to the technical field of network traffic analysis. The method adopts a multi-head linear attention architecture, linearly approximates an exponential kernel function through a learnable non-negative mapping function, and optimizes a matrix calculation sequence to reduce the complexity from a quadratic order to a linear order. The effectiveness of attention weights is ensured through a non-negative kernel function construction, a multi-head parallel computing mechanism is adopted to capture diversified global dependency modes, and the memory consumption of explicitly calculating an N*N attention matrix is avoided. The application solves the calculation bottleneck and memory limitation problems of a traditional self-attention mechanism when processing large-scale network traffic sequences, significantly improves the processing efficiency and accuracy of a website identification task while maintaining the expression capability of a model, and is suitable for long sequence traffic data analysis containing a large number of session windows.

Description

Technical Field

[0001] This invention relates to the fields of computer network technology, deep learning and network security, and in particular to a traffic sequence processing method based on a non-negative kernel function self-attention mechanism. Background Technology

[0002] With the rapid development of internet technology, network traffic has experienced explosive growth. Accurate analysis and identification of network traffic is crucial for network management, security monitoring, and user experience optimization. Website fingerprinting, as an important traffic analysis technology, aims to identify specific websites visited by users by analyzing metadata (such as packet size and timing) in encrypted traffic, and it remains effective even when the traffic content is encrypted.

[0003] In recent years, deep learning-based sequence models, especially the Transformer architecture with self-attention mechanisms at its core, have demonstrated powerful capabilities in processing sequence data and have been attempted for application in traffic analysis. Standard multi-head attention mechanisms can capture global dependencies within a sequence, where each parallel computational unit is called an "attention head," which is crucial for understanding complex traffic patterns across multiple session windows. However, standard self-attention mechanisms suffer from a significant performance bottleneck: their computational complexity increases exponentially with the square of the sequence length (…). This quadratic complexity is directly proportional to the length of network traffic sequences. When processing large network traffic sequences (i.e., N is very large), this quadratic complexity leads to huge computational and memory overhead, making the model difficult to train and deploy, and severely limiting its application in resource-constrained environments or scenarios requiring real-time analysis.

[0004] Some existing linear attention methods attempt to reduce complexity through kernel function decomposition. The core idea is to treat the Softmax exponential similarity function (a normalized exponential function used to map multidimensional vectors to a probability distribution summing to 1) in standard attention mechanisms as an implicit kernel function, and then find an explicit feature mapping. ,Will The attention weight matrix is ​​decomposed into an inner product form after feature mapping, thus utilizing the associative law of matrix multiplication to reduce computational complexity from... Reduce to .

[0005] However, existing linear attention methods have significant limitations in constructing feature maps. On the one hand, traditional linearization methods typically use fixed, predefined kernel functions (such as polynomial kernels, radial basis function kernels, etc.). Theoretically, these functions have a significant "kernel space gap" with the target kernel function, making it difficult to accurately approximate the Softmax distribution in standard attention, resulting in severe information loss. On the other hand, fixed kernel functions lack flexibility and cannot adaptively adjust to the complex and variable characteristics of network traffic data, limiting the model's expressive ability in high-dimensional nonlinear feature extraction, thus leading to a decrease in recognition accuracy.

[0006] Therefore, there is an urgent need for a traffic sequence processing method that can maintain strong sequence modeling capabilities and achieve linear computational complexity, while also efficiently estimating Softmax and adapting to data characteristics through a learnable kernel function. The purpose of this invention is to overcome the high computational cost of standard self-attention and the estimation bias and lack of flexibility caused by the fixed kernel function in existing linear attention methods. Summary of the Invention

[0007] To address the aforementioned shortcomings of existing technologies, this invention provides a traffic sequence processing method based on a non-negative kernel function self-attention mechanism. The core of this invention lies in providing an attention mechanism with linear computational complexity. This mechanism utilizes a learnable non-negative mapping function to map queries and keys to a non-negative feature space, implicitly defining an effective kernel function. Furthermore, by cleverly rearranging the order of matrix multiplication, the computational complexity is reduced from linear to linear. Reduce to (in For mapping dimensions, and usually This ensures that while maintaining the model's expressive power, it achieves high computational efficiency and adaptive learning of data features in sequence modeling and processing.

[0008] The specific technical solution of the present invention is as follows:

[0009] This invention provides a traffic sequence processing method based on a non-negative kernel function self-attention mechanism, the method comprising the following steps:

[0010] Step S1: Monitor network traffic and preprocess the network traffic to obtain a network traffic feature sequence. ,in For the number of windows, For feature dimensions;

[0011] Step S2: The network traffic feature sequence Through respectively Projecting a set of independent linear projection matrices generates Each attention point has its own query matrix. Key matrix Sum matrix ,in For the index of the attention head, and ;

[0012] Step S3: Query matrix for each attention head Bond matrix Apply a learnable nonnegative mapping function Map query and key features to In the feature space, a nonnegative query matrix is ​​obtained. Non-negative bond matrix ,in The dimensions after mapping;

[0013] The learnable nonnegative mapping implicitly defines a kernel function. The kernel function is used to approximate the exponential kernel in standard attention, thus serving as an effective approximation of the exponential kernel. Furthermore, the learnable nonnegative mapping implicitly defines the kernel function in a way that satisfies Mercer's theorem, guaranteeing the effectiveness of the kernel function.

[0014] Step S4: Based on the nonnegative bond matrix Sum matrix Calculate the attention numerator for each attention head. Specifically ;

[0015] Step S5: Based on the nonnegative query matrix Non-negative bond matrix Calculate the normalization factor for each attention head. Specifically, it involves calculating the diagonal matrix. ,in It is a length of A vector of all 1s;

[0016] Step S6: According to the attention numerator described in step S4 and the normalization factor described in step S5 Calculate the output of each attention head. Specifically ;

[0017] Step S7, put all The output of each attention head The data is then concatenated to obtain the final attention representation. .

[0018] Preferably, in step S1, the preprocessing includes the following sub-steps:

[0019] S11: Extract at least one of the time-related statistical features and packet content-related features from network traffic;

[0020] S12: Divide and align the extracted features according to the session window to form the initial serialized data;

[0021] S13: Standardize or normalize the serialized data to eliminate the influence of dimensions.

[0022] Preferably, in step S2, the number of attention heads... The value of is a positive integer between 4 and 16.

[0023] Preferably, the nonnegative mapping function described in step S3 Adopt the following form:

[0024] ;

[0025] in, The weight matrix is ​​a learnable matrix. This is a learnable bias vector.

[0026] Preferably, the learnable weight matrix and bias vector At the start of model training, the Xavier initialization method is used to initialize the parameters;

[0027] The satisfy ,in The sequence length is given.

[0028] Preferably, in step S4, the attention numerator is calculated. The process, by changing the order of matrix multiplication, reduces the computational complexity from... Reduce to Specifically, it includes the following sub-steps:

[0029] S41: First calculate the intermediate cache matrix ,in , The computational complexity of this step is... ;

[0030] S42: Then calculate the attention numerator. ,in , The computational complexity of this step is... .

[0031] Preferably, the normalization factor is calculated in step S5. The process specifically includes the following sub-steps:

[0032] S51: First calculate the sum vector of keys. ,in , It is a length of The computational complexity of this step is O(n log n). ;

[0033] S52: Then calculate the normalized intermediate vector. ,in , The computational complexity of this step is... ;

[0034] S53: Finally from the vector Extract the diagonal elements and construct a normalized diagonal matrix. .

[0035] Preferably, after step S7, the method further includes representing the final attention. Step S8: Input to downstream tasks

[0036] S81: Represent the attention The input is fed into a fully connected classification layer;

[0037] S82: Based on the output of the fully connected classification layer, use the Softmax function to calculate the probability that the network traffic sequence belongs to each website in the predefined website set;

[0038] S83: Based on the calculated probability, determine the target website to which the traffic sequence most likely belongs.

[0039] Preferably, the model training for the downstream task employs a multi-class cross-entropy loss function, and the Adam optimizer is used for weight updates. The initial learning rate of the Adam optimizer is... The weight decay coefficient is .

[0040] Furthermore, the method is used to model network traffic data containing multiple session windows to capture dependencies between distant windows and perform website identification.

[0041] The beneficial technical effects of this invention are as follows:

[0042] 1. Extremely high computational efficiency: This invention reduces the computational complexity of the self-attention mechanism from quadratic by introducing nonnegative mapping and rearranging the computational order. Successfully reduced to linear This enables the invention to efficiently process extremely long network traffic sequences, making real-time or near real-time traffic monitoring and analysis possible, while significantly reducing computing resource consumption and energy consumption;

[0043] 2. Powerful Model Expressiveness and Flexibility: This invention employs a learnable non-negative mapping function instead of a fixed kernel function. This allows the model to automatically learn and adjust the optimal feature mapping based on the characteristics of actual network traffic data, thereby more accurately approximating the ideal attention distribution. This not only overcomes the poor flexibility of fixed kernel functions but also maintains strong model fitting ability and recognition accuracy while ensuring efficiency. Furthermore, although computational complexity is reduced, this invention retains the core capability of modeling the global context of the sequence. The output at each sequence position is influenced by information from all other positions in the sequence, effectively capturing complex traffic pattern dependencies across distant session windows, which is crucial for accurate website fingerprinting.

[0044] 3. Excellent end-to-end trainability and practicality: The entire model, including the core non-negative mapping function parameters, is differentiable, supporting end-to-end joint training using gradient descent, facilitating optimization and integration into larger deep learning pipelines. This ensures that the algorithm can be easily applied and tuned as an efficient and flexible module in practical traffic analysis tasks.

[0045] In summary, this invention, as a traffic sequence processing method, cleverly combines a learnable kernel function with a linear attention mechanism, achieving an excellent balance between computational efficiency, model performance, and practicality, and providing a powerful tool for processing large-scale, long-sequence network traffic data analysis. Attached Figure Description

[0046] Figure 1 This diagram presents the overall framework of a traffic sequence processing method based on a non-negative kernel function self-attention mechanism. From left to right, it depicts the complete processing flow: starting with the input of the original network traffic sequence, followed by preprocessing for feature extraction and data standardization; then, a multi-head projection module generates query, key, and value matrices; the core part demonstrates the transformation process of the query and key by a learnable non-negative mapping function, as well as the subsequent linear attention calculation module, showing the calculation paths for the numerator and normalization factor; finally, after multi-head fusion and classification decision-making, the website identification result is output. The entire diagram clearly illustrates the complete architecture of this invention from data input to result generation.

[0047] Figure 2This is a comparison chart of PR curves for various methods under different numbers of tags. The precision-recall curves visually compare the classification performance of each method under different tag scales. In the chart, "open-2Tab" to "open-5Tab" represent multi-target traffic mixed scenarios simulating users simultaneously opening 2 to 5 different browser tabs in an open-world environment.

[0048] Figure 3 This is a comparison of the area under the PR curve for various methods under different numbers of labels. Furthermore, the area under the curve of each method is compared in a quantitative way, which reflects the overall performance advantage of the model.

[0049] Figure 4 This chart compares the Top-k accuracy of various methods under different numbers of tabs, evaluating the recognition capabilities of different methods from the perspective of Top-k accuracy. In the chart, "open-2Tab" to "open-5Tab" represent multi-target traffic mixed scenarios simulating users simultaneously opening 2 to 5 different browser tabs in an open-world environment.

[0050] Figure 5 This graph illustrates the impact of different traffic lengths and loading times on accuracy, analyzing the influence of traffic sequence length and loading time on recognition accuracy. In the graph, "closed-world" represents a closed-set evaluation scenario, where the model only classifies within a known, fixed set of websites; "open-world" represents an open-world evaluation scenario, where the model needs to perform recognition in a mixed environment containing known monitored websites and a large amount of unknown background traffic.

[0051] Figure 6 This diagram illustrates the impact of different embedding dimensions and kernel estimation dimensions on accuracy, exploring the moderating effect of these key parameters on model performance. In the diagram, "closed-world" represents a closed-set evaluation scenario, where the model classifies only a known, fixed set of websites; "open-world" represents an open-world evaluation scenario, where the model must perform identification in a mixed environment containing both known monitored websites and a large amount of unknown background traffic.

[0052] Figure 7 This chart compares the runtime (left) and accuracy (right) of different attention mechanisms for long sequences. It comprehensively compares the efficiency and accuracy of different attention mechanisms when processing long sequences. The left half shows the computation time comparison, and the right half shows the accuracy comparison, jointly verifying the superiority of this invention in terms of efficiency and accuracy. In the chart, "Standard Attention" refers to a computational complexity of O(n log n). The standard self-attention mechanism serves as a performance benchmark; "TopM Attention" represents the existing technical path of optimizing computation by selecting the m highest attention values ​​as key features; "LINA" refers to the linearized attention mechanism based on non-negative kernel approximation proposed in this invention. Detailed Implementation

[0053] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0054] Example 1: Method Example

[0055] Reference Figure 1 The diagram illustrates a traffic sequence processing method based on a non-negative kernel function self-attention mechanism. This invention provides a complete solution for network traffic sequence processing. The system mainly includes a feature sequence input module, a multi-head linear attention calculation module, and a classification output module. The specific steps are as follows:

[0056] Step S1: Obtain the input network traffic feature sequence

[0057] The method proposed in this invention is applicable to real and complex physical network environments. Specifically, it captures raw network data packets in the transmission link in real time by deploying traffic acquisition probes at network ingress switches, core routers, or gateways. These data packets are real encrypted traffic generated when end users access the Internet or interact with specific network services through physical devices (such as personal computers, mobile phones, etc.). The acquisition process not only records the precise timestamps generated by physical transmission but also fully preserves the data packet size and direction information reflecting business logic, providing real and objective underlying data support for subsequent solutions to specific technical problems such as network security monitoring and traffic classification. Based on the above-collected physical network data, let the network traffic trajectory data be... The sequence consists of L data packets arranged in chronological order. For any data packet in the sequence... Its characteristics can be expressed as Includes timestamps and symbolic packet information , Symbols indicate the direction of data packet transmission. Represents outbound data packets. Represents inbound data packets. Indicates the size of the data packet;

[0058] Network traffic feature sequence It is usually obtained by preprocessing raw network traffic data. The preprocessing process may include:

[0059] ① Extract time-related statistical features from network traffic, including packet arrival time interval, session duration, and traffic rate change patterns; at the same time, extract features related to packet content, including packet size distribution, protocol type distribution, and port usage.

[0060] ② The extracted features are divided and aligned according to the session window to form the initial serialized data. The size of the session window can be adjusted according to the specific application scenario, and is usually set between 0.1 seconds and 1 second;

[0061] ③ Standardize or normalize the serialized data to eliminate the influence of different feature units and improve the stability and convergence speed of model training.

[0062] Obtain the preprocessed network traffic feature sequence ,in This indicates the number of windows, specifically the number of session windows. This represents the feature dimension. The feature sequence contains various statistical features extracted from network traffic, including time-related features and packet content-related features. In practical applications, the number of windows... The value can be adjusted from hundreds to thousands depending on the specific scenario.

[0063] Step S2: Multi-head projection generates query, key, and value matrices. In this embodiment, the feature dimensions of the input sequence are set. Number of attention heads And let the query / key projection dimension of each header be 1. The value projection dimension is (In this embodiment, and Equivalent value, but under different configurations The width of the output feature can be adjusted independently to control its width.

[0064] use A set of independent linear projection matrices for the input sequence Perform projection. Each projection matrix includes: query projection matrix. Key projection matrix Value projection matrix ,in, Indicates the index of the attention head. The size of the projection space determines the dimension of the final output vector of each attention head. The projection calculation process is as follows:

[0065] Query matrix: ;

[0066] Key matrix: ;

[0067] Value matrix: ;

[0068] During model training, the parameters of the projection matrix are optimized using gradient descent, and the projection matrices of different attention heads are independent to ensure that each head can capture complementary feature representations in the flow sequence. Typically, the number of attention heads... Take a positive integer between 4 and 16, preferably an even number in this embodiment (e.g., This is primarily due to considerations of industrial-grade computing efficiency and hardware architecture compatibility. Although in terms of algorithm principles... There is no parity limit to the value of , but when actually deployed in network traffic monitoring equipment, an even number of heads can more effectively utilize the single instruction multiple data stream parallel characteristics of high-performance computing units (such as GPUs or FPGAs) and achieve better data alignment and memory access modes in video memory, thereby significantly reducing the computational latency when splicing multi-head features and improving the throughput of large-scale concurrent data processing.

[0069] Step S3: Apply a learnable nonnegative mapping function

[0070] To overcome the problems caused by the use of the Softmax exponential kernel function in the standard attention mechanism In terms of computational complexity, this step uses explicit feature mapping to transform the high-dimensional kernel operations implicit in Softmax into low-dimensional inner product operations. Softmax is a normalized exponential function used to map multi-dimensional vectors to a probability distribution that sums to 1; Attention refers to the mechanism of assigning weights to values ​​based on the similarity between the query and the key.

[0071] Query matrix for each attention head Bond matrix Apply a learnable nonnegative mapping function Mapping the original features to The feature space is dimensional, thus yielding the nonnegative query matrix. Non-negative bond matrix The mapping function takes the following form:

[0072] ;

[0073] in, The weight matrix is ​​a learnable matrix. These are learnable bias vectors. In the early stages of training, the parameters... and The Xavier initialization (or Glorot initialization) method is used to keep the variance of activation values ​​and gradients consistent across layers, thus avoiding gradient vanishing or exploding.

[0074] Regarding the kernel approximation dimension The construction logic is explained as follows:

[0075] The nonnegative mapping function Used to construct features based on the original input. Explicit feature mapping. In this construction process, by introducing a dimension of... The transformation space will transform the original 3D feature mapping to A higher-dimensional space. Specifically, this manifests as:

[0076] ① Dimensional transformation: Weight matrix number of rows The feature dimension after linear transformation is determined, making The output result is dimensional vector;

[0077] ② Kernel function approximation: mapping function Acting on this Each element of the dimensional vector is ultimately generated Also A dimensional feature vector. The feature vector after this mapping is... The inner product in a high-dimensional kernel space can effectively approximate the kernel operation in the original high-dimensional kernel space;

[0078] ③Performance balance: parameters As a kernel approximation dimension, its value directly affects the balance between the approximation accuracy and computational complexity of the kernel function; increasing its value... It can improve the ability to express complex nonlinear distributions.

[0079] In this embodiment, parameters Defined as the kernel approximation dimension, preferably set to Introduction The core logic lies in: through matrices Will Projecting the original features to A 3D space is provided, allowing the kernel function to be represented as two... The inner product of two vectors, i.e. This construction method follows Mercer's theorem, ensuring the validity of the kernel function. The setting principle ensures... ( (where the sequence length is used), thus reducing the computational complexity from quadratic to linear. Furthermore, because... and It is learnable; the mapping function can adaptively approximate and estimate the target Softmax distribution based on the nonlinear characteristics of network traffic data.

[0080] After completing the above nonnegative mapping, the computational paradigm of the attention mechanism undergoes a fundamental change, transforming from the quadratic computational complexity of standard attention to linear computational complexity. The computational formula for standard attention is: Its bottleneck lies in the fact that it must first be calculated The similarity matrix. This invention, through kernel function approximation, transforms the formula into:

[0081] ;

[0082] Based on the associative law of matrix multiplication, the calculation order changes from the original "calculate first" to "calculate later". and The product of "first calculate" is rearranged to "the product of "first calculate". and The product of. and The intermediate results have only one dimension. This completely avoids the generation A high-complexity weight matrix of this size is an inevitable result of achieving linear complexity.

[0083] Step S4: Calculate the attention numerator with linear complexity.

[0084] Based on nonnegative bond matrix Sum matrix Calculate the attention numerator for each attention head. By changing the order of computation, linear complexity can be achieved:

[0085] ① First, calculate the intermediate cache matrix. :

[0086] ;

[0087] The computational complexity is... . This is called the global context feature matrix. Its physical meaning is the interactive aggregation feature between the key vector space and the value vector space. It compresses the information of the entire sequence into a matrix whose size is proportional to the sequence length. In an unrelated constant-order matrix.

[0088] ②Then calculate the attention numerator. :

[0089] ;

[0090] The computational complexity is... . This represents the unnormalized response result after global context weighting, expressed by the current query matrix. With aggregation features Multiplication directly yields the feature representation at each position, compared to traditional methods that directly compute... Matrix Compared to other methods, this approach significantly reduces computational overhead.

[0091] Step S5: Calculate the normalization factor for linear complexity.

[0092] Based on nonnegative query matrix Non-negative bond matrix Calculate the normalization factor for each attention head. (Corresponding to the denominator of Softmax in standard attention):

[0093] ① First, calculate the sum vector of the keys. :

[0094] ;

[0095] The computational complexity is... . This is the feature sum vector, representing the non-negative key features at all positions. Global energy distribution in each dimension of the dimensional space.

[0096] ②Then calculate the normalized intermediate vector. :

[0097] ;

[0098] The computational complexity is... . The normalized weights are summed for each position. This is achieved by projecting the query vector onto the global key energy vector. Above, we obtain the scalar scale used for normalization at each position.

[0099] ③ Finally, from the vector Extract elements from the matrix and construct a normalized diagonal matrix:

[0100] ;

[0101] in Represented by vector The elements are used to construct a diagonal matrix, which is used to adjust the numerator in subsequent steps. Perform line-by-line normalization.

[0102] Step S6: Calculate the attention head output

[0103] According to the attention numerator and normalization factor Calculate the output of each attention head:

[0104] ;

[0105] in Represents a diagonal matrix Find the inverse. Because... Given a diagonal matrix, its inversion operation has... It has linear complexity.

[0106] The linearized attention via nonnnegative kernel approximation (LINA) mechanism employed in this invention is based on the design of a nonnegative mapping function. This function has significant advantages in mathematical design: due to the constant within the square root... The existence of restricts the function's range to . That is, having The strict lower bound.

[0107] This design brings about two technological improvements:

[0108] ① Overcoming activation failure: It effectively avoids the "zero dead zone" phenomenon that is prone to occur in traditional neural networks ReLU (linear rectified function, a common activation method that returns negative values ​​to zero), ensuring that neurons remain activated during training and avoiding gradient vanishing;

[0109] ② Enhanced numerical stability: During the normalization process, this property ensures the stability of the normalized matrix. (where index) Representing the The diagonal elements of the (attention head) have extremely strong stability. Specifically, since each element after mapping is not less than , making The diagonal elements (i.e., the sum of the weights at each position) are at least equal to the number of sequence windows. This completely eliminates the risk of computational overflow and instability caused by a denominator that is zero or too small, from a mathematical perspective.

[0110] Step S7, Multi-head Fusion Output

[0111] All The output of each attention head By concatenating the features, we obtain the final attention representation that integrates global dependency information from multiple perspectives:

[0112] ;

[0113] This representation integrates diverse global dependency patterns captured by different attention heads, each focusing on learning different types of inter-window dependencies, and the number of attention heads... It is usually taken as a positive integer between 4 and 16.

[0114] Step S8, Optional downstream classification steps

[0115] After step S7, the final attention representation can be obtained. Input is sent to downstream tasks for processing:

[0116] ① Expressing attention The input is fed into a fully connected classification layer, which maps the high-dimensional attention representation to the website category space;

[0117] ② Based on the output of the fully connected classification layer, the Softmax function (a normalized exponential function used to map a multidimensional vector to a probability distribution with a sum of 1) is used to calculate the probability that the network traffic sequence belongs to each website in a predefined set of websites. This function, by exponentializing and normalizing the raw scores (Logits) output by the classification layer, clearly characterizes the model's confidence scores for different website categories, thereby achieving accurate traffic classification and identification.

[0118] ③ Based on the calculated probability, determine the target website to which the traffic sequence most likely belongs, and complete the website identification task.

[0119] The model training employs the multi-class cross-entropy loss function and uses the Adam optimizer for weight updates. In this embodiment, the hyperparameters of the Adam optimizer are set as follows: initial learning rate... The weight decay coefficient is The end-to-end joint training of the nonnegative mapping function parameters and the classification layer is achieved through the backpropagation algorithm.

[0120] Example 2: Experimental Verification

[0121] (I) Experimental Setup

[0122] 1. Experimental Environment

[0123] This experiment was conducted on a server equipped with an NVIDIA GeForce RTX 3090 GPU and 64GB of RAM, aiming to verify the performance of the method of this invention on real network traffic data. The operating system was Windows 10, and the programming environment was Python 3.8 with PyTorch version 2.1.2.

[0124] 2. Hyperparameter settings

[0125] During the model training phase, the batch size was uniformly set to 100, and the total number of epochs was 30. The learning rate is used to optimize the parameters. For the model structure proposed in this invention, the input dimension d=4, the number of attention heads h=8, and the projection dimension of each head is... Meanwhile, the feature dimension (i.e., the low-rank approximate rank) after kernel mapping takes the value r=4.

[0126] Through the specific experimental environment and hyperparameter configurations described above, it was ensured that the model could efficiently reproduce the linear complexity calculation of the LINA structure and achieve stable convergence under high-dimensional mapping.

[0127] (II) Experimental Dataset and Comparison Methods

[0128] This embodiment uses an open-world multi-tag traffic dataset for evaluation. This dataset covers 100 specific monitored websites, with websites randomly selected from the top 20,000 on Alexa (a ranking system that measures a website's global popularity and is often used to obtain the most representative background website traffic on the internet) as non-monitored background traffic. The dataset contains over 250,000 instances of multi-tag obfuscated traffic, simulating a real-world scenario where a user simultaneously browses 2 to 5 websites with different tags.

[0129] In the experimental processing, the total data packet length of the input sequence was uniformly processed to 10000. Based on the aforementioned window partitioning logic, the long sequence was converted into a window size. The input features are consistent with the aforementioned description of "the number of windows being adjusted between hundreds and thousands". The dataset is divided into training, validation, and test sets in an 8:1:1 ratio to ensure that the model can obtain sufficient training and objective reproducibility evaluation under evaluation scenarios of varying complexity, from open-2Tab to open-5Tab (i.e., in an open-world environment, simulating a multi-target traffic mixture scenario where a user simultaneously opens 2 to 5 different tabs). The baseline models for comparison include mainstream state-of-the-art traffic identification methods, including:

[0130] ① Deep learning-based methods: such as DF, VarCNN (based on convolutional neural networks), ARES (based on attention mechanisms), and TikTok (focusing on temporal pattern learning);

[0131] ② Methods based on statistical feature extraction: such as RF (aggregate matrix);

[0132] ③ New methods: such as CountMamba (based on a state-space model).

[0133] All comparison methods were reproduced using their official default parameters to ensure fairness in the comparison, such as setting the ARES network layer to 5 layers and the CountMamba window size to 44ms.

[0134] To ensure fairness in the comparison, all deep learning-based comparison methods were reimplemented within our experimental framework or using the official code, and the same training strategies (including optimizer, learning rate, batch size, and number of iterations) were employed.

[0135] (3) Evaluation metrics: Mean Precision (MAP), F1 score (harmonic mean of precision and recall), training time per epoch, and feature extraction speed;

[0136] In the key experimental steps, the raw flow load first undergoes standardization preprocessing, transforming the flow rate into a window value. A fixed sequence; then, in the feature sequence alignment stage, extract... Metadata (uplink and downlink traffic size and quantity) is normalized. In end-to-end classification training, the key is to use the AdamW optimizer with a weight decay strategy, and to employ a warm-up learning rate in the first 5 epochs to ensure... The high-dimensional projection matrix and A learnable kernel function of dimension can converge smoothly in the early stages of training, avoiding the model from getting stuck in local optima due to drastic gradient fluctuations.

[0137] (III) Performance Evaluation Results

[0138] 1. PR curve and area under the curve (AUC) analysis

[0139] Comparison of PR curves for various methods under different tag sizes, as follows: Figure 2 As shown in the figure, this method (LINA) maintains the highest precision at different recall levels. The AUC comparison results for each method are shown in the figure. Figure 3 As shown, in open-2Tab to open-5Tab scenarios, the AUC value of this method remains stable between 0.9168 and 0.9295, significantly outperforming traditional methods such as ARES. This advantage is attributed to the learnable nonnegative mapping function. It can adaptively approximate the target attention kernel distribution based on multi-flow characteristics, thereby extracting more discriminative feature representations in multi-label confusion environments.

[0140] 2. Top-k accuracy comparison

[0141] The Top-k accuracy comparison results of each method under different numbers of labels are as follows: Figure 4As shown, taking open-2Tab as an example, when k=1, the P@2 accuracy of this method is close to 100%; in the open-5Tab scenario, the P@5 accuracy when k=5 can still be maintained above 80%. This is mainly due to the linear complexity of LINA, which enables the model to efficiently process long sequence inputs, thereby capturing more global and cross-span flow dependencies in multi-label association modeling.

[0142] 3. The impact of different flow lengths and loading times on accuracy

[0143] The influence of different flow lengths and loading times on accuracy is as follows: Figure 5 As shown in the figure. Experiments revealed that when the maximum trajectory length reaches 10000, the accuracy of MAP@5 rapidly improves and stabilizes. This verifies the superiority of this method in processing ultra-long traffic sequences. The linear complexity breaks through the memory bottleneck, allowing the model to utilize traffic trajectory information over a longer range to compensate for the accuracy loss caused by multi-label confusion.

[0144] 4. The impact of different embedding dimensions and kernel estimation dimensions on accuracy

[0145] The effect of different parameters on accuracy, such as Figure 6 As shown, MAP@5 significantly improves when the embedding dimension increases from 32 to 256. Furthermore, as the kernel estimation dimension increases... As the rank of the low-rank approximation increases from 2 to 8, the accuracy shows a steady upward trend. This reflects the fact that this method increases the rank of the low-rank approximation. (in This approach can more accurately represent the detailed distribution of attention weights, achieving a good balance between model complexity and representational power.

[0146] 5. Comprehensive comparison of computational efficiency and accuracy

[0147] The comprehensive comparison results of different attention mechanisms in terms of runtime and accuracy are as follows: Figure 7 As shown. Considering the quadratic computational complexity of the standard self-attention mechanism, which demands extremely high computational resources, this embodiment uniformly sets the evaluation sequence length to 1800 to ensure that all comparative experiments can be conducted fairly under the same hardware environment. Experimental results show that at this length, the standard attention takes 18.21s, while this method only takes 0.34s, improving efficiency by approximately 53 times.

[0148] This significant performance leap stems primarily from the fundamental restructuring of computational logic in this invention: by rearranging the original "query-key" priority computation order to a "key-value" priority aggregation mode, redundant computation is greatly reduced; simultaneously, by utilizing learnable kernel functions to map features to a low-rank space, the computational complexity is successfully reduced from... Reduce to This allows for an exponential increase in computational efficiency while minimizing the loss of precision.

[0149] Among them: Standard Attention serves as a performance benchmark, demonstrating the highest computational complexity of the operating state; Top-M Attention represents the existing technical path of optimizing computation by selecting the m highest attention values ​​as key features; and the method of this invention (LINA) represents the linearization processing scheme based on non-negative kernel approximation proposed in this application.

[0150] (iv) Experimental Conclusions

[0151] The above experiments verify that the flow sequence processing method based on the non-negative kernel function self-attention mechanism provided by this invention has the following significant effects:

[0152] (1) Significantly improved computational efficiency: The traditional self-attention mechanism is significantly improved. The computational complexity was reduced to This makes it possible to process long-sequence traffic data;

[0153] (2) Enhanced long-distance dependency modeling capability: Through multi-head mechanism and global attention mechanism, it effectively captures complex dependency relationships between long-distance session windows;

[0154] (3) Improved website recognition accuracy: By combining linear attention mechanism and multi-head design, the accuracy of website recognition is improved while maintaining computational efficiency;

[0155] (4) Achieve adaptive learning of parameters: Through a learnable non-negative mapping function, the model can adaptively approximate the optimal attention kernel function.

[0156] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, and for those of ordinary skill in the art, various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. Therefore, the present invention is not limited to the specific details without departing from the general concept defined by the claims and their equivalents.

Claims

1. A flow sequence processing method based on a non-negative kernel function self-attention mechanism, characterized in that, The method includes the following steps: Step S1: Monitor network traffic and preprocess the network traffic to obtain a network traffic feature sequence. ,in For the number of windows, For feature dimensions; Step S2: The network traffic feature sequence Through respectively Projecting a set of independent linear projection matrices generates Each attention point has its own query matrix. Key matrix Sum matrix ,in For the index of the attention head, and ; Step S3: Query matrix for each attention head Bond matrix Apply a learnable nonnegative mapping function Map query features and key features to In the feature space, a nonnegative query matrix is ​​obtained. Non-negative bond matrix ,in The dimensions after mapping; The learnable nonnegative mapping implicitly defines a kernel function. The kernel function is used to approximate the exponential kernel in standard attention, thus serving as an effective approximation of the exponential kernel. Furthermore, the learnable nonnegative mapping implicitly defines the kernel function in a way that satisfies Mercer's theorem, guaranteeing the effectiveness of the kernel function. Step S4: Based on the nonnegative bond matrix Sum matrix Calculate the attention numerator for each attention head. Specifically ; Step S5: Based on the nonnegative query matrix Non-negative bond matrix Calculate the normalization factor for each attention head. Specifically, it involves calculating the diagonal matrix. ,in It is a length of A vector of all 1s; Step S6: According to the attention numerator described in step S4 and the normalization factor described in step S5 Calculate the output of each attention head. Specifically ; Step S7, put all The output of each attention head The data is then concatenated to obtain the final attention representation. ; In step S1, the preprocessing includes the following sub-steps: S11: Extract at least one of the time-related statistical features and packet content-related features from network traffic; S12: Divide and align the extracted features according to the session window to form the initial serialized data; S13: Standardize or normalize the serialized data to eliminate the influence of units; In step S5, the normalization factor is calculated. The process specifically includes the following sub-steps: S51: First calculate the sum vector of keys. ,in , It is a length of The computational complexity of this step is O(n log n). ; S52: Then calculate the normalized intermediate vector. ,in , The computational complexity of this step is... ; S53: Finally from the vector Extract the diagonal elements and construct a normalized diagonal matrix. .

2. The method according to claim 1, characterized in that, In step S2, the number of attention heads The value of is a positive integer between 4 and 16.

3. The method according to claim 1, characterized in that, The nonnegative mapping function described in step S3 Adopt the following form: in, The weight matrix is ​​a learnable matrix. It is a learnable bias vector.

4. The method according to claim 3, characterized in that, The learnable weight matrix and bias vector At the start of model training, the Xavier initialization method is used to initialize the parameters; The satisfy ,in The sequence length is given.

5. The method according to claim 1, characterized in that, In step S4, the attention numerator is calculated. The process, by changing the order of matrix multiplication, reduces the computational complexity from... Reduce to Specifically, it includes the following sub-steps: S41: First calculate the intermediate cache matrix ,in , The computational complexity of this step is... ; S42: Then calculate the attention numerator. ,in , The computational complexity of this step is... .

6. The method according to claim 1, characterized in that, Following step S7, the method further includes representing the final attention statement. Step S8: Input to downstream tasks S81: Represent the attention The input is fed into a fully connected classification layer; S82: Based on the output of the fully connected classification layer, use the Softmax function to calculate the probability that the network traffic sequence belongs to each website in the predefined website set; S83: Based on the calculated probability, determine the target website to which the traffic sequence most likely belongs.

7. The method according to claim 6, characterized in that, The downstream task model training employs a multi-class cross-entropy loss function and uses the Adam optimizer for weight updates. The initial learning rate of the Adam optimizer is... The weight decay coefficient is .

8. The method according to any one of claims 1-7, characterized in that, The method is used to model network traffic data containing multiple session windows in order to capture dependencies between distant windows and perform website identification.

Images