Post-training-based multi-level feature self-attention malicious URL (Uniform Resource Locator) detection method

Abstract

The invention discloses a multi-level feature self-attention malicious URL detection method based on post-training, belongs to the field of malicious URL detection, and overcomes the defects of a traditional detection method in domain adaptability and feature capture by constructing a pre-training language model based on post-training and combining multi-level feature extraction and a dynamic attention mechanism. By introducing three self-supervised learning tasks of a mask language model, a noise language model and domain discrimination, the understanding of the model on URL features is enhanced; by extracting multi-level features from a low order to a high order and dynamically adjusting the weight of a feature layer, malicious links can be identified more accurately; experimental results show that under the challenging conditions of small-scale data, class imbalance, attack resistance and the like, the method is remarkably superior to an existing deep learning model and a traditional method, more effective technical support is provided for network security, and meanwhile, the universality, the robustness, the expandability and the data change response capability of the model are evaluated through experiments.

Description

Technical Field

[0001] The present invention belongs to the field of malicious URL detection, and in particular relates to a malicious URL detection method based on post-training multi-level feature self-attention. Background Art

[0002] Malicious URLs are deceptive web links designed to convince users to reveal sensitive and personal information. Clicking on these deceptive links may lead to serious consequences, including personal data theft and cyber attacks.

[0003] Malicious URL detection is an important part of network security, aiming to identify and intercept potential network threats. In terms of user privacy protection, malicious URLs are often used for phishing and information theft. Research on malicious URL detection technology provides a solid theoretical foundation and technical support for the field of network security. Malicious URL detection technology initially relied on rule-based detection methods, which usually used blacklists and simple feature engineering and could not adapt to dynamically changing network threats. Later, with the development of deep learning technology, researchers began to try to use methods such as convolutional neural networks (CNN) and recurrent neural networks (RNN). Although there have been progress in feature extraction, manual processing of input data is still required. In recent years, pre-trained language models (such as BERT, GPT, etc.) have attracted widespread attention due to their powerful feature learning capabilities. Researchers began to explore the application of these models to URL detection, although direct application still faces challenges in domain adaptability and feature extraction.

[0004] Post-training strategy is a process of further optimizing and adjusting the model based on the pre-trained language model, with the aim of making it better suited to specific tasks or fields. The existing technology has the problem of insufficient domain adaptability. The existing malicious URL detection model often has poor adaptability in specific fields and cannot effectively identify new malicious URL patterns. Through the post-training strategy, the pre-trained language model can be adapted to specific tasks, thereby improving the model's ability to capture domain-specific features.

[0005] Multi-level feature extraction is a technical method that aims to extract features of different levels and granularity from input data to capture more comprehensive information. This method is particularly important in deep learning models, especially when dealing with complex data such as text, images or audio. The existing technology still has the problem of incomplete feature extraction. Traditional feature extraction methods usually only focus on local features or global features, and fail to fully reflect the complex structure and semantic information of URLs. The use of multi-level feature extraction, extracting features of different levels from low-order to high-order, can ensure that the model can pay attention to details and overall context at the same time.

[0006] The dynamic attention mechanism is an improved attention mechanism that aims to dynamically adjust the attention weight according to the characteristics and context of the input data to more effectively capture key information. The static attention mechanism is usually used in the prior art. The static attention mechanism has limitations. The static attention mechanism cannot dynamically adjust the focus when processing different contexts, which may lead to the neglect of important information. By introducing the dynamic attention mechanism, the weight is dynamically adjusted according to the importance of the feature layer, which can enhance the focus on key information.

[0007] In addition, there is a lack of self-supervised learning tasks in the existing technology. Many existing models rely on a large amount of labeled data, which is costly and time-consuming to obtain, resulting in poor performance of the model in new environments. By designing self-supervised learning tasks (such as masked language models, noisy language models, and domain discrimination tasks), the model can effectively learn in the absence of a large amount of labeled data. In practical applications, there is still the problem of insufficient robustness. Existing models usually perform poorly when faced with small-scale data and category imbalance in practical applications, affecting the accuracy and reliability of detection. Comprehensive evaluation in a variety of challenging scenarios can ensure the effectiveness and robustness of the model in different environments. Summary of the invention

[0008] In order to solve the shortcomings of the prior art and achieve the purpose of improving malicious URL detection performance and enhancing detection effect, the present invention adopts the following technical solutions:

[0009] The malicious URL detection method based on post-training multi-level feature self-attention includes the following steps:

[0010] Step 1: Build a dataset;

[0011] Step 2: Build a pre-trained language model, including a backbone network with a character embedding module and a heterogeneous interaction module;

[0012] Step 3: Pre-train the language model and extract features through the multi-order feature extraction module. The multi-layer Transformer encoder learns semantic knowledge, but the low- and medium-level features may be degraded during the processing. The multi-order feature extraction module uses the hierarchical information of the backbone network to extract features from low to high levels.

[0013] Step 4: Based on the backbone network, weights are assigned through the layer-aware attention module. Inspired by the channel attention principle, the layer-aware attention module independently assigns weights to feature maps of different layers to improve processing efficiency and accuracy.

[0014] Step 5: Spatial pyramid pooling: Innovatively apply the spatial pyramid pooling technology in computer vision to the Transformer model; for the weighted feature results extracted by the layer-aware attention module, multi-level pyramid pooling is used (using a sliding window with a size of win = [a / n] and a step length of str = [b / n], n×n is the pyramid level, the first level pooling operation is recorded as Pool1, the second level pooling operation is recorded as Pool2, and the third level pooling operation is recorded as Pool3);

[0015] The results after pooling are F1=Pool1(F att ), F2 = Pool2(F att ), F3=Pool3(F att );

[0016] The pooled feature map is divided into local spatial partitions from fine to coarse, and local features are aggregated; then, the connected feature map is average pooled along the fixed sequence length dimension:

[0017] F final =AvgPool f (Concat(F1,F2,F3))

[0018] Finally, the URL features are converted into binary class representations for prediction through standard Dropout layers and fully connected layer processing, which improves the model’s adaptability to features of different scales and enhances the overall prediction accuracy.

[0019] Assume that the Dropout operation is recorded as Dropout and the fully connected layer operation is recorded as FC, then the prediction result is:

[0020] y=FC(Dropout(F final ))

[0021] Step 6: Post-training: Masked LM and Noisy LM are used to learn the contextual semantics of URL subword and character-level representations, and domain discrimination tasks are used for domain adaptation learning.

[0022] Furthermore, the character embedding module in step 2 uses a two-layer bidirectional gated recurrent unit (BiGRU) network to encode the character sequence in the input token, processes the input through forward and reverse gated recurrent units GRU, and integrates the hidden layer state to generate context-aware character embedding; the specific calculation process is that for a sequence with an input window size of d, the forward GRU calculates:

[0023]

[0024] Reverse GRU calculation:

[0025]

[0026] Final output:

[0027]

[0028] When generating character embeddings, the input sequence is represented as a subword sequence, the embedding of the characters in each subword is calculated and processed through a bidirectional gated recurrent unit (BiGRU), and then the hidden states of the first and last characters of each subword are concatenated to obtain the subword-level embedding.

[0029] Furthermore, the heterogeneous interaction module in step 2 fuses and separates subword and character representations after each Transformer layer; first, the subword and character representations are converted separately through different fully connected layers, then concatenated and integrated using CNN layers, and then the fused features are restored to two channels using a fully connected layer with GELU activation, and residual connections are added to retain the original information, and finally layer normalization is performed. This process enriches the mutual representation of the two channels while retaining their respective characteristics. The pre-training task also enhances the distinguishing ability of the dual-channel framework.

[0030] The specific process of the heterogeneous interaction module is as follows: s , character representation is x c , first through different fully connected layers W s1 and W c1 Convert subword and character representations separately: y s =W s1 x s ,y c =W c1 x c .

[0031] Then use CNN layer to splice and integrate: y = CNN ([y s ;y c ])(where [;] represents a concatenation operation).

[0032] Then use a fully connected layer W with GELU activation s2 and W c2 Restore the fused features to two channels: z s =W s2 y, z c =W c2 y.

[0033] Adding residual connections preserves the original information:

[0034] Finally, perform layer normalization:

[0035] Furthermore, in step 3, the output of the word-level and character-level encoders of the backbone network at each layer (denoted as k1, k2, ..., k n and c1,c2,...,c m , each output has the dimension of (H, W, C)), first merge the two channels through one-dimensional convolution and restore the original dimension, then stack along the new dimension 0 to form a tensor X (dimension is (N, H, W, C), N is the number of layers), to optimize the representation, define the permutation matrix P to exchange dimensions 0 and 1 to obtain X' (dimension is (H, N, W, C)), X' is used as the input of the subsequent attention module, thereby effectively integrating the features of each layer and avoiding the limitation of relying only on the features of the last layer.

[0036] The specific calculation process is as follows: Assume that the outputs of the word-level and character-level encoders of the backbone network CharBERT at each layer are k1, k2, ..., k n and c1,c2,...,c m , each output has dimensions (H, W, C);

[0037] First, we merge the two channels through one-dimensional convolution and restore the original dimension. Let the one-dimensional convolution operation be denoted as Conv1D. The result after processing is

[0038] Then stack along the new dimension 0 to form a tensor X (with dimensions (N, H, W, C), where N is the number of layers):

[0039]

[0040] Define the permutation matrix P to swap dimensions 0 and 1 to get X' (with dimensions (H, N, W, C)):

[0041] X′=PX

[0042] X' serves as the input of the subsequent attention module.

[0043] Furthermore, in step 4, spatial context descriptors are generated by respectively performing average pooling and maximum pooling on the features. and They are then processed through a shared multilayer perceptron (MLP) network (containing hidden layers) to generate a channel attention map M∈R C×1×1 The specific calculation is:

[0044] M c (F)=σ(MLP(AvgPool(F)+MLP(MaxPool(F))))

[0045] Among them, σ represents the sigmoid function;

[0046] The attention map is multiplied with the input pyramid feature map to highlight important features, allowing the model to focus on more discriminative information and enhance the expressiveness of URL features.

[0047] Furthermore, in step 6, three unsupervised learning tasks, namely, a masked language model task, a noise language model task, and a domain discrimination task, are used to further train the pre-trained language model to adapt to the URL domain and learn context information at different levels.

[0048] Furthermore, the masked language model task is to integrate the masked language model in the token channel training to enhance the context interpretation ability by predicting the masked tokens in the URL sequence; the masked language model task is based on masking some tokens in the input URL sequence and then letting the model predict these masked tokens; let the URL sequence be x=[x1,x2,...,x n ], in this sequence, a certain proportion of positions will be randomly selected for masking; for the position i selected to be masked, its original token x i is replaced by a special mask marker to obtain the masked sequence The goal of the model is to predict the original token at the masked position based on the contextual information, that is, other tokens in the sequence that are not masked; the model adjusts its internal parameters by learning a large amount of URL data to maximize the accuracy of predicting the masked token.

[0049] From a mathematical point of view, the model learns a mapping function f MLM , so that:

[0050]

[0051] Among them, f MLM Represents the function determined by the model's parameter θ, that is:

[0052]

[0053] The model optimizes the parameter θ by minimizing the difference between the predicted value and the true value (such as cross entropy loss);

[0054] Its loss function L MLM It is expressed as:

[0055]

[0056] Among them, y i represents the real token, i.e., at the mask position, p i Represents the probability distribution of tokens at that position predicted by the model.

[0057] Furthermore, considering the semantic sparsity of URLs, the mask ratio is fine-tuned from the default 15% of BERT to 10%. This adjustment is to better adapt to the characteristics of URL data, so that the model can more effectively learn the semantic relationship between tokens in URLs during training.

[0058] Furthermore, the noise language model task is to capture the internal morphological characteristics of vocabulary terms by introducing character noise into words (such as deleting, inserting or scrambling characters), forcing the model to predict the original words from the perturbed character representations; for this purpose, a new word-level vocabulary is constructed as a prediction space, and the prediction probability is calculated by a specific formula.

[0059] Furthermore, the domain discrimination task: based on the concept of the next sentence prediction (NSP) task of BERT, the goal is transformed into distinguishing the URL domain from the general text domain; the model receives input in the format of [CLS]A[SEP]B[SEP], where A and B are randomly selected from the URL domain or the text domain in different cases; a domain discriminator is used to predict the sample domain according to the hidden state of [CLS], a gradient reversal layer (GRL) is added before the discriminator, and the discriminator parameters are optimized through back propagation, so that the model learns domain-independent features, enhances the recognition ability of URL-specific patterns, and maintains the transferability of language knowledge.

[0060] The advantages and beneficial effects of the present invention are:

[0061] The present invention combines post-training with multi-level feature extraction to propose a new malicious URL detection model, which significantly improves the detection performance. The present invention uses three self-supervised learning tasks (masked language model, noise language model and domain discrimination task) to enhance the model's learning ability of URL features by performing post-training on the basis of the pre-trained model. These tasks enable the model to better understand its structure and semantics when processing URLs; the present invention can extract different features of URLs from low-order to high-order by designing multiple feature processing modules. This multi-level feature extraction ensures that the model can pay attention to local details and global context information at the same time; the present invention introduces a hierarchical attention mechanism, which can dynamically adjust the weight according to the importance of the feature layer to strengthen the influence of key features, thereby improving the overall detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a flow chart of a method in an embodiment of the present invention.

[0063] Figure 2 Schematic diagram of the BiGRU in an embodiment of the present invention.

[0064] Figure 3 It is a diagram of a heterogeneous interaction architecture in an embodiment of the present invention.

[0065] Figure 4 Schematic diagram of three unsupervised post-training tasks in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0067] The present invention uses a post-trained pre-trained language model (such as CharBERT) combined with multi-level feature extraction and dynamic attention mechanism to process data. First, by collecting and organizing data sets containing malicious and normal URLs, data cleaning and standardization are performed; then, using character-level and word-level feature extraction modules, multi-level features are extracted from URLs to capture their complex structure and semantic information, and character-level and word-level features are fused through heterogeneous interaction modules; then, combined with the dynamic attention mechanism, the weights are dynamically adjusted according to the importance of the features to enhance the model's attention to key features; after post-training of masked language models, noisy language models, and domain discrimination self-supervised learning tasks, the model's adaptability in specific fields is improved; finally, the present invention applies spatial pyramid pooling for multi-scale downsampling to obtain local features and global context. The PMANet of the present invention achieves multi-faceted integration in URL feature extraction, including capturing information at the vocabulary and character levels, extracting features from low-order to high-order, and discerning patterns in global and local ranges.

[0068] like Figure 1 As shown in the figure, the malicious URL detection method based on post-trained multi-level feature self-attention of the present invention uses the post-trained CharBERT to process the URL to extract features at the character and subword levels; then, the multi-order feature extraction module derives the encoding representation from low-level to high-level; then, the layer-aware attention dynamically readjusts the weights of different features; then, spatial pyramid pooling is performed to highlight local nuances and consolidate global context, specifically including the following steps:

[0069] Step 1: Construct a data set, including data set selection and analysis, as well as data sampling and partitioning;

[0070] Among them, data set selection and analysis: Four public malicious URL data sets are selected, including GramBeddings, Mendeley, Kaggle 1 and Kaggle 2. The differences in these data sets in sample size, TLD distribution, etc. provide rich test scenarios for comprehensively evaluating the effectiveness of the model in different network environments.

[0071] Data sampling and partitioning: When evaluating the effectiveness of multi-layer features, 128,000 URLs were randomly selected from the training set of the GramBeddings dataset for training, and 32,000 URLs were randomly sampled for validation and testing. When studying the model's dependence on the scale of training data, training data was selected from the GramBeddings and Mendeley datasets in different proportions (such as 1%, 5%, 10%, 20%, 40%, 60%, 80%) to comprehensively examine the performance of the model under different data volumes. For each dataset, it is reasonably divided into training set, validation set, and test set according to experimental requirements to ensure the randomness and representativeness of the data, so as to accurately evaluate the generalization ability and stability of the model.

[0072] Step 2: Constructing the backbone network CharBERT: The pre-trained CharBERT is used as the backbone network, which has a dual-channel framework and can capture subword and character-level information. The backbone network includes a character embedding module and a heterogeneous interaction module.

[0073] 1) Character embedding module: such as Figure 2 As shown in the figure, a two-layer bidirectional gated recurrent unit (BiGRU) network is used to encode the character sequence in the input token, and the input is processed by the forward and reverse GRUs, and the hidden layer states are integrated to generate context-aware character embeddings. The specific calculation process is as follows: for a sequence with an input window size of d, the forward GRU calculates:

[0074]

[0075] Reverse GRU calculation:

[0076]

[0077] Final output:

[0078]

[0079] When generating character embeddings, the input sequence is represented as a subword sequence, the embedding of the characters in each subword is calculated and processed through BiGRU, and then the hidden states of the first and last characters of each subword are concatenated to obtain the subword-level embedding.

[0080] 2) Heterogeneous interaction modules: Figure 3As shown in the figure, subword and character representations are fused and separated after each Transformer layer. The subword and character representations are first transformed separately through different fully connected layers, then concatenated and integrated using CNN layers, and then the fused features are restored to two channels using a fully connected layer with GELU activation, and residual connections are added to retain the original information, and finally layer normalization is performed. This process enriches the mutual representation of the two channels while retaining their respective features. The pre-training task also enhances the discrimination ability of the dual-channel framework.

[0081] The specific process of the heterogeneous interaction module is as follows: s , character representation is x c , first through different fully connected layers W s1 and W c1 Convert subword and character representations separately: y s =W s1 x s ,y c =W c1 x c .

[0082] Then use CNN layer to splice and integrate: y = CNN ([y s ;y c ])(where [;] represents a concatenation operation).

[0083] Then use a fully connected layer W with GELU activation s2 and W c2 Restore the fused features to two channels: z s =W s2 y, z c =W c2 y.

[0084] Adding residual connections preserves the original information:

[0085] Finally, perform layer normalization:

[0086] Step 3: Feature extraction through multi-level feature extraction module: The pre-trained language model learns semantic knowledge through multi-layer Transformer encoders, but it may cause low- and medium-level feature degradation in the process. This module aims to extract features from low to high levels by using the hierarchical information of the backbone network CharBERT. The output of the word-level and character-level encoders of the backbone network CharBERT at each layer (denoted as k1, k2, ..., k n and c1,c2,...,c m, each output has the dimension of (H, W, C)), first merge the two channels through one-dimensional convolution and restore the original dimension, then stack along the new dimension 0 to form a tensor X (dimension is (N, H, W, C), N is the number of layers). To optimize the representation, the permutation matrix P is defined to exchange dimensions 0 and 1 to obtain X' (dimension is (H, N, W, C)), which is used as the input of the subsequent attention module, thereby effectively integrating the features of each layer and avoiding the limitation of relying only on the features of the last layer.

[0087] The specific calculation process is as follows: Assume that the outputs of CharBERT's word-level and character-level encoders at each layer are k1, k2, ..., k n and c1,c2,...,c m , each output has dimensions (H, W, C).

[0088] First, we merge the two channels through one-dimensional convolution and restore the original dimension. Let the one-dimensional convolution operation be denoted as Conv1D. The result after processing is

[0089] Then stack along the new dimension 0 to form a tensor X (with dimensions (N, H, W, C), where N is the number of layers):

[0090]

[0091] Define the permutation matrix P to swap dimensions 0 and 1 to get X' (with dimensions (H, N, W, C)):

[0092] X′=PX

[0093] X' serves as the input of the subsequent attention module.

[0094] Step 4: Assign weights through the layer-aware attention module: Inspired by the channel attention principle, this module independently assigns weights to feature maps of different layers to improve processing efficiency and accuracy. Spatial context descriptors are generated by average pooling and maximum pooling features respectively. and They are then processed through a shared multilayer perceptron (MLP) network (containing hidden layers) to generate a channel attention map M∈R C×1×1 The specific calculation is:

[0095] M c (F)=σ(MLP(AvgPool(F)+MLP(MaxPool(F))))

[0096] Where σ represents the sigmoid function.

[0097] The attention map is multiplied with the input pyramid feature map to highlight important features, allowing the model to focus on more discriminative information and enhance the ability to express URL features.

[0098] Step 5: Spatial pyramid pooling: The spatial pyramid pooling technology in computer vision is innovatively applied to the Transformer model. For the weighted feature results extracted by the layer-aware attention module, three-level pyramid pooling is used (using a sliding window of size win = [a / n] and stride str = [b / n], n×n is the pyramid level, the first level pooling operation is recorded as Pool1, the second level pooling operation is recorded as Pool2, and the third level pooling operation is recorded as Pool3).

[0099] The results after pooling are F1=Pool1(F att ), F2 = Pool2(F att ), F3=Pool3(F att ).

[0100] The feature map is divided into local spatial partitions from fine to coarse, and local features are aggregated. Then, the concatenated feature map is average pooled along the fixed sequence length dimension:

[0101] F final =AvgPool f (Concat(F1,F2,F3))

[0102] Finally, the URL features are converted into binary class representations for prediction through standard Dropout layers and fully connected layer processing, which improves the model’s adaptability to features of different scales and enhances the overall prediction accuracy.

[0103] Assume that the Dropout operation is recorded as Dropout and the fully connected layer operation is recorded as FC, then the prediction result is:

[0104] y=FC(Dropout(F final ))

[0105] Step 6: Post-training method: Propose a post-training method such as Figure 4 As shown, Masked LM and Noisy LM are used to learn the contextual semantics of URL subword and character-level representations, while the domain discrimination task is responsible for domain adaptation learning. The pre-trained language model (CharBERT) is further trained using three unsupervised learning tasks (masked language model, noisy language model, and domain discrimination task) to adapt to the URL domain and learn different levels of contextual information.

[0106] 1) Masked Language Model (MLM) Task: Integrate MLM in token channel training to enhance contextual interpretation capabilities by predicting masked tokens in URL sequences. Considering the semantic sparsity of URLs, the mask ratio is fine-tuned from BERT’s default 15% to 10%.

[0107] The masked language model task is based on masking some tokens in the input URL sequence and then letting the model predict these masked tokens. Let the URL sequence be x = [x1, x2, ..., x n ], in this sequence, a certain proportion of positions are randomly selected for masking operations.

[0108] For the position i selected to be masked, its original token x i is replaced by a special mask tag (such as [MASK]) to obtain the masked sequence The goal of the model is to predict the original token at the masked position based on the contextual information (i.e. other tokens in the sequence that are not masked).

[0109] The model learns from a large amount of URL data and adjusts its internal parameters to maximize the accuracy of predicting masked tokens. From a mathematical point of view, the model learns a mapping function f MLM , so that:

[0110]

[0111] where f MLM It is a function determined by the model's parameter θ, namely:

[0112]

[0113] The model optimizes the parameters θ by minimizing the difference between the predicted values ​​and the true values ​​(e.g., cross entropy loss).

[0114] Its loss function L MLM It can be expressed as:

[0115]

[0116] where y i is the real token (at the masked position), p i is the probability distribution of tokens at that position predicted by the model.

[0117] In this paper, considering the semantic sparsity of URLs, the mask ratio is fine-tuned from the default 15% of BERT to 10%. This adjustment is to better adapt to the characteristics of URL data, so that the model can more effectively learn the semantic relationship between tokens in URLs during training.

[0118] For example, when processing a URL containing multiple subwords and characters, by masking some tokens, the model is forced to use the surrounding contextual information to infer the content of the masked part, thereby enhancing the ability to understand the URL semantics. This helps the model more accurately capture the semantic features related to malicious intent in the URL in subsequent malicious URL detection tasks, thereby improving the accuracy of detection.

[0119] Take "https: / / google.com" as an example: For the URL "https: / / google.com", first perform word segmentation. Assume that the result after word segmentation is ["https",": / / ","google",".","com"] (the actual word segmentation may be more detailed, this is a simplified example). Then, randomly select the token to be masked according to a certain ratio (such as 15%). For example, the token "google" is selected and replaced with [MASK], and the masked URL is ["https",": / / ","[MASK]",".","com"].

[0120] 2) Noisy Language Model (Noisy LM) Task: The internal morphological features of vocabulary terms are captured by introducing character noise into words (such as deleting, inserting or scrambling characters), forcing the model to predict the original word from the perturbed character representation. For this purpose, a new word-level vocabulary is constructed as the prediction space, and the prediction probability is calculated by a specific formula.

[0121] 3) Domain Discrimination (DD) Task: Based on the concept of the next sentence prediction (NSP) task of BERT, the goal is changed to distinguish the URL domain from the general text domain. The model receives input in the format of [CLS]A[SEP]B[SEP], where A and B are randomly selected from the URL domain or the text domain in different cases. The domain discriminator is used to predict the sample domain based on the hidden state of [CLS]. A gradient reversal layer (GRL) is added before the discriminator. The discriminator parameters are optimized through back propagation, so that the model learns domain-independent features and enhances the recognition ability of URL-specific patterns while maintaining the transferability of language knowledge.

[0122] V. Effects of the invention:

[0123] Table 1

[0124]

[0125]

[0126] As can be seen from Table 1, with the smallest training subset of only 6,400 URLs (1%), the method of the present invention performs well, achieving an accuracy of 93.52%, which is significantly higher than the baseline range of 81.25% to 91.35%. In addition, the model of the present invention exhibits a high recall of 91.34%, indicating its sensitivity in identifying malicious website URLs, compared to the highest baseline recall of 90.83%. The F1 score is a comprehensive indicator of precision and recall, and it shows that the method of the present invention has a significant advantage of 93.38%, while the F1 scores of the baseline methods range from 79.47% to 91.53%. Even though the performance of the baseline model gradually improves with the increase of training samples, with the method of the present invention, using 80% of the training samples, the method of the present invention still achieves 98,40% accuracy and 98.39% F1 score dataset. This exceeds the best performing baseline model, which has an accuracy of 97.36% and an F1 score of 97.33%. Overall, these results on a balanced dataset highlight that our approach is able to maintain high precision and recall, effectively distinguishing between malicious and benign samples.

[0127] The present invention extracts effective features from historical data as training data through a post-trained pre-trained language model combined with multi-level feature extraction and an attention mechanism. With the application of deep learning technology in malicious URL detection, higher requirements are placed on the quality and diversity of training data to improve the accuracy and generalization of detection. The detection capability of the present invention can identify and block these threats in real time, protect the personal information security of users, and compare the user's access behavior with known malicious URLs by dynamically analyzing malicious links, thereby achieving a rapid response to potential threats. The present invention can be widely used in various aspects of network security, including real-time monitoring, user behavior analysis, and security policy formulation. At the same time, the performance of the present invention directly affects the effectiveness and reliability of subsequent security protection measures, so its optimization and improvement are the key to ensuring network security.

[0128] In a range of challenging scenarios, the proposed method outperforms methods based on previous leading pre-trained models and traditional DL techniques, achieving excellent versatility and robustness while also demonstrating efficient learnability, scalability to different data amounts, and adaptability to the changing threat landscape.

[0129] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. Malicious URL detection method based on post-training multi-level feature self-attention, characterized by The steps include: Step 1: Build a dataset; Step 2: Build a pre-trained language model, including a backbone network with a character embedding module and a heterogeneous interaction module; Step 3: Pre-train the language model and use the hierarchical information of the backbone network through a multi-order feature extraction module to extract features from low to high levels; Step 4: Based on the backbone network, the layer-aware attention module is used to independently assign weights to feature maps of different layers; Step 5: Spatial pyramid pooling: Multi-level pyramid pooling is used for the weighted feature results extracted by the layer-aware attention module; the pooled feature map is divided into local spatial partitions from fine to coarse, and local features are aggregated; then, the connected feature map is average pooled along the fixed sequence length dimension; finally, the URL features are converted and predicted; Step 6: Post-training; The contextual semantics of URL subword and character-level representations are learned, and domain discrimination tasks are used to perform domain adaptation learning.

2. The malicious URL detection method based on post-training multi-level feature self-attention according to claim 1 is characterized in that: The character embedding module in step 2 uses a two-layer bidirectional gated recurrent unit network to encode the character sequence in the input token, processes the input through forward and reverse gated recurrent units, and integrates the hidden layer states to generate context-aware character embeddings; when generating character embeddings, the input sequence is represented as a subword sequence, the embedding of the characters in each subword is calculated and processed through the bidirectional gated recurrent unit, and then the hidden states of the first and last characters of each subword are connected to obtain the subword-level embedding.

3. The malicious URL detection method based on post-training multi-level feature self-attention according to claim 1 is characterized in that: The heterogeneous interaction module in step 2 fuses and separates subword and character representations after each Transformer layer; first, the subword and character representations are converted separately through different fully connected layers, then concatenated and integrated using CNN layers, and then the fused features are restored to two channels using a fully connected layer with GELU activation, and residual connections are added to retain the original information, and finally layer normalization is performed.

4. The malicious URL detection method based on post-training multi-level feature self-attention according to claim 1 is characterized in that: In step 3, the outputs of the word-level and character-level encoders of the backbone network at each layer are first merged into two channels and the original dimensions are restored, and then stacked along the new dimension 0 to form a tensor X. The permutation matrix P is defined to exchange dimensions 0 and 1 to obtain X', which is used as the input of the subsequent attention module.

5. The malicious URL detection method based on post-training multi-level feature self-attention according to claim 1 is characterized in that: In step 4, spatial context descriptors are generated for the features after average pooling and maximum pooling respectively, and then they are processed through a shared multi-layer perceptron network to generate a channel attention map.

6. The malicious URL detection method based on post-training multi-level feature self-attention according to claim 1 is characterized in that: In step 6, the pre-trained language model is further trained using an unsupervised learning task to adapt to the URL domain and learn context information at different levels.

7. The method for detecting malicious URLs based on post-training multi-level feature self-attention according to claim 6, characterized in that: The unsupervised learning task includes a masked language model task, which integrates a masked language model in token channel training to enhance context interpretation capabilities by predicting masked tokens in URL sequences; the masked language model task is based on masking some tokens in the input URL sequence and then letting the model predict these masked tokens; For the position selected to be masked, its original token is replaced with a special masking mark to obtain a masked sequence; the goal of the model is to predict the original token of the masked position based on the context information, that is, other tokens in the sequence that are not masked; The model learns from a large amount of URL data and adjusts its internal parameters to maximize the accuracy of predicting masked tokens.

8. The method for detecting malicious URLs based on post-training multi-level feature self-attention according to claim 7, characterized in that: The mask ratio is fine-tuned from BERT’s default 15% to 10%.

9. The method for detecting malicious URLs based on post-training multi-level feature self-attention according to claim 6, characterized in that: The unsupervised learning task includes a noisy language model task, which is to capture the internal morphological characteristics of vocabulary terms by introducing character noise into words, forcing the model to predict the original words from the perturbed character representations; For this purpose, a new word-level vocabulary is constructed as the prediction space and the prediction probability is calculated.

10. The method for detecting malicious URLs based on post-training multi-level feature self-attention according to claim 6, characterized in that: The unsupervised learning task includes a domain discrimination task. Based on the concept of the next sentence prediction task of BERT, the goal is transformed into distinguishing the URL domain from the general text domain. The model receives input in the format of [CLS] A [SEP] B [SEP], where A and B are randomly selected from the URL domain or the text domain in different cases. The domain discriminator is used to predict the sample domain according to the hidden state of [CLS], a gradient reversal layer is added before the discriminator, and the discriminator parameters are optimized by back propagation.

Images