Malicious URL detection method based on BERT-CNN and hierarchical attention mechanism

Abstract

The invention discloses a malicious URL detection method based on BERT-CNN and a hierarchical attention mechanism, and the method comprises the following steps: S1, obtaining a URL data set, dividing the URL data set into a malicious URL data set and a normal URL data set, and sorting and analyzing data features; s2, sample data is preprocessed, a data set is refined into a training set, a test set and a verification set, and unbalanced data is processed; s3, performing feature extraction, and converting the text into numerical features by using a BERT model; s4, respectively deploying an algorithm for fusing the attention mechanism and the convolutional neural network, an algorithm for fusing the attention mechanism and the gating circulation unit, and an algorithm model for fusing the attention mechanism and the long-short term memory network, and inputting the processed sample data into the models for training; and S5, evaluating the model through the evaluation index, and judging whether the model can accurately identify the malicious URL and the normal URL. According to the method, the problem that important local features may be lost when a traditional model processes URL data can be solved, the importance of URL key sub-words is captured through word-level attention, the whole URL semantic structure is integrated through sentence-level attention, and the attention weight of a noise part is reduced through dynamic weight distribution.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence detection, and particularly to a malicious URL detection method based on BERT-CNN and hierarchical attention mechanism. Background Art

[0002] With the rapid development of the Internet, network security issues have become increasingly prominent, and malicious URL (Uniform Resource Locator) detection has become an important task to protect users from cyberattacks. Malicious URLs are usually used to spread malware, initiate phishing attacks or steal users' sensitive information, posing huge security risks to individuals and enterprises. Traditional malicious URL detection methods mainly rely on rule-based blacklists or machine learning models, but these methods often perform poorly when facing new types of malicious URLs and are difficult to cope with the continuous evolution of attack methods.

[0003] In recent years, deep learning technologies have made remarkable progress in the field of natural language processing. Especially the emergence of pre-trained language models has provided powerful tools for text classification and sequence modeling tasks. BERT captures the context information of text through a bidirectional Transformer architecture and can effectively understand the semantic structure of malicious URL text. However, the BERT model may have difficulty capturing local features when processing long URLs, while convolutional neural networks have significant advantages in extracting local features. Therefore, combining the global semantic understanding ability of BERT and the local feature extraction ability of CNN to build a hybrid model has become an important research direction for improving the performance of malicious URL detection.

[0004] Although existing deep learning methods have achieved certain results in the malicious URL detection task, there are still some challenges. First, traditional models have difficulty capturing both global and local features when processing long URLs, resulting in limited detection accuracy. Second, the diversity and concealment of malicious URLs make the model perform poorly in detecting new types of malicious URLs. In addition, existing methods usually ignore the importance differences of different parts in malicious URL text and fail to fully utilize the attention mechanism to enhance the model's ability to capture key information. With the complexity of cyberattacks, traditional rule-based malicious URL detection methods are gradually showing limitations. Machine learning-based malicious URL detection algorithms can more effectively identify new types of malicious URLs by learning patterns from a large amount of data. Malicious URLs come in various forms and are difficult to identify through a single rule. Attackers constantly update malicious URLs, making it difficult for traditional methods to cope; and malicious URLs are often disguised as normal URLs, making it difficult to distinguish. Summary of the Invention

[0005] To solve the problem that traditional models have little discrimination between normal and malicious URL data in complex scenarios, the purpose of the present invention is to provide a malicious URL detection method based on BERT-CNN and hierarchical attention mechanism, which captures the global semantic information of malicious URLs through the BERT module, extracts the local features of malicious URLs through the CNN module, and dynamically allocates attention weights through the hierarchical attention mechanism to enhance the model's ability to capture key information.

[0006] To achieve the above objectives, the present invention provides the following technical solutions:

[0007] The present invention provides a malicious URL detection method based on BERT-CNN and hierarchical attention mechanism, including the following steps:

[0008] S1. Obtain the URL dataset, divide the URL dataset into malicious URL and normal URL datasets, and organize and analyze the data features;

[0009] S2. Through preprocessing the sample data, refine the dataset into a training set, a test set, and a validation set, and process the imbalanced data;

[0010] S3. Perform feature extraction, and use the BERT model to convert the text into numerical features;

[0011] S4. Deploy algorithms that fuse the attention mechanism and the convolutional neural network, algorithms that fuse the attention mechanism and the gated recurrent unit, and algorithms that fuse the attention mechanism and the long short-term memory network model respectively, and input the processed sample data into the model for training;

[0012] S5. Evaluate the model through evaluation metrics to determine whether the model can accurately identify malicious URLs and normal URLs.

[0013] The core architecture of the present invention integrates a hierarchical attention mechanism and a spatial-channel reconstruction convolutional module. First, the model converts the original URL string into a context-related token embedding representation through a pre-trained BERT encoder, preserving the sequential semantic information of the URL. In the encoding stage, an improved BERT is used as the basic module. The encoder sequentially includes a 3×3 convolutional layer, batch normalization, and a ReLU activation function. The convolutional module CNN effectively extracts the local n-gram features of the URL and suppresses noise through separable convolution and dynamic channel weighting. To enhance the feature expression ability, a multi-stage channel attention mechanism is introduced in the deep layer of the encoder, and the local features are fused with the global context through skip connections. The word-level attention applies the attention mechanism to each token representation output by BERT, calculates the importance weights of each token in the sentence, and applies a second attention layer to the local features extracted by CNN to focus on the importance of different n-gram features. The CNN feature extraction layer uses multiple convolutional kernels of different sizes, and each convolutional kernel is followed by a ReLu activation function and max pooling. The outputs of convolutional kernels of different sizes are concatenated to form the final feature representation. The decoder part adopts a symmetric convolutional structure. The entire model uses an unsupervised training method to learn the decision boundary by reconstructing the semantic and structural features of normal URL samples, and finally realizes anomaly detection through the prediction error. The BERT-CNN network architecture introducing the hierarchical attention mechanism is as Figure 1 shown. This architecture makes full use of the semantic understanding ability of BERT and the local feature extraction advantage of CNN, and realizes the adaptive weighting of different-level features of the URL through the hierarchical attention mechanism, significantly improving the detection robustness of ambiguous URLs.

[0014] Preferably, in step S4, the fusion attention mechanism is specifically:

[0015] On the basis of adding the word attention mechanism, a sentence attention layer is added to allocate the text features and weights of the entire URL;

[0016] The word attention layer uses CNN to extract the context semantic information features about individual characters and conducts training. The word attention mechanism calculates the weights of the extracted key features; the sentence attention layer uses CNN to extract the context semantic features of the entire URL text for model training, and the sentence attention mechanism calculates the weights of the extracted key sentence features.

[0017] Furthermore, the word attention layer is specifically:

[0018] The context embedding output by BERT is a tensor with the shape of (batch_size, seq_len, hidden_size), where the batch size batch_size = 32; the input sequence length seq_len = 50; the hidden layer dimension hidden_size = 768; the word-level attention mechanism calculates the attention score for each word, and the word vector U is obtained through the activation function softmax it and the context vector U w The relationship between them is normalized to obtain the attention weight. The output formula of the word-level attention layer is calculated as shown in formulas (3) - (5):

[0019] (3) (4) (5)

[0020] In the formula, W it is the weight parameter, b it is the bias parameter, h it is the output information of the hidden layer state of the BERT network, and Si is the output of the word attention layer.

[0021] Furthermore, the sentence attention layer is specifically:[[]]

[0022] Taking the word-level attention representation as the input, applying the sentence-level attention mechanism to calculate the importance weight of the sentence level, including a fully connected layer for mapping the hidden layer dimension hidden_size of BERT to a scalar attention score; the sentence-level attention mechanism calculates the text attention score, and the sentence vector U is obtained through the activation function softmax it and the context vector U w The relationship between them is normalized to obtain the attention weight. The output formula of the sentence-level attention layer is calculated as shown in formulas (6) - (8):

[0023] (6) (7) (8)

[0024] In the formula, W i is the weight parameter, b i is the bias parameter, h i is the output information of the hidden layer state of the BERT network, and O i is the output of the sentence attention layer.

[0025] Preferably, in step S4, the convolutional network layer is specifically:[[]]

[0026] In the malicious URL classification task of the BERT-CNN model based on the hierarchical attention mechanism, the convolutional network module is responsible for extracting the local features of the URL text; one-dimensional convolution (1D CNN) is used to process the context embedding representation output by BERT;

[0027] Define the number of convolutional kernels as 128, the size k of each convolutional kernel is 3, the height of the convolutional kernel, that is, the number of words covered, is 3, and the embedding dimension of BERT is 768; in sequence data, one-dimensional convolution Conv1D is more applicable than two-dimensional convolution, which can directly model the local dependencies of the sequence and the relationships between adjacent time points, avoiding the redundant calculations introduced by the false spatial dimensions of two-dimensional convolution;

[0028] Selecting a convolutional kernel with k = 3 can effectively capture short-range pattern trigrams, and at the same time, by stacking multiple small kernels, a receptive field similar to that of a large kernel can be achieved;

[0029] Average pooling is performed on the feature map output by convolution to reduce the feature dimension and retain important information. The formula is shown in formula (9):

[0030] (9)

[0031] The pooling outputs of all convolutional kernels are concatenated together to generate the final output vector.

[0033] Preferably, in step S5, the output network layer is specifically:

[0034] Detecting malicious URLs is a classification problem. The present invention uses a fully connected layer as the output network to detect and classify malicious URLs; by taking the output 0 of the sentence attention layer in the hierarchical attention mechanism as the input of the next fully connected layer, and then normalizing the vector output by the fully connected layer through the softmax function, finally, the output vector f of the model is obtained. The calculation formula is shown in formula (10):

[0035] (10)

[0036] In the formula, W o is the weight parameter, and b is the bias parameter;

[0037] The test result only outputs one predicted category, and the category with the largest probability value is selected as the classification result of the URL; when the model is trained on the training set, by comparing the predicted category output by the fully connected layer with the actual category, the training error of the model is obtained, and then the model parameters are continuously trained through forward propagation and backward propagation until a model with a relatively good classification effect is obtained;

[0038] Malicious URL detection is a classification problem. The training of the model of the present invention adopts a supervised learning framework, uses the backpropagation method to take the derivative of the loss function with respect to all model parameters, adopts the Adam optimization method to optimize and update all parameters, and uses the minimization of the cross-entropy loss function as the objective function to optimize the model. The formula is shown in Equation (11):

[0039] (11)

[0040] In the formula, D represents the size of the data volume of the training set, C represents the number of categories for URL classification, and the size of C in the task is 2, is the predicted category, y is the actual category of the data, is the cross-regularization term.

[0041] The present invention captures the global semantic information of the URL through the BERT module and extracts the local features of the URL through the convolutional neural network (CNN). A hierarchical attention mechanism is introduced between BERT and CNN, enabling the model to dynamically allocate attention weights at different levels, thereby better capturing the key information in the URL. In addition, by introducing a sparse attention mechanism, the computational complexity and memory overhead of the model are reduced, while retaining the global semantic understanding ability of BERT.

[0042] The present invention can solve the problem that traditional models may lose important local features when processing URL data. It captures the importance of key sub-words in the URL through word-level attention and integrates the semantic structure of the entire URL through sentence-level attention, and reduces the attention weight of the noise part through dynamic weight allocation. And the hierarchical attention layer can balance local and global features. The hierarchical fusion of the CNN module and the BERT module through semantic-local feature co-perception significantly improves the accuracy of the model for malicious URL detection.

[0043] The beneficial technical effects of the present invention are as follows:

[0044] (1) The present invention proposes a hybrid model combining BERT and CNN, making full use of the global semantic understanding ability of BERT and the local feature extraction ability of CNN; a hierarchical attention mechanism is introduced to capture the key information of malicious URL text through word-level and sentence-level attention.

[0045] (2) The present invention conducts experimental verification on a public dataset, and the results show that the model has high accuracy and robustness in the malicious URL detection task.

[0046] (3)The present invention provides an efficient and accurate solution for malicious URL detection, which has important theoretical significance and practical application value. Future research directions include further optimizing the attention mechanism, exploring multi-modal data fusion technology, and applying the model to other network security tasks. Description of the Drawings

[0047] Figure 1 It is the BERT-CNN network architecture introducing the hierarchical attention mechanism.

[0048] Figure 2 It is a schematic diagram of the input sequence representation.

[0049] Figure 3 It is a schematic diagram of generating the word vector representation.

[0050] Figure 4 It is the result graph of the precision curve and loss curve of the BERT baseline model and the model introducing the hierarchical attention mechanism module.

[0051] Figure 5 It is the ROC curve effect diagram and confusion matrix of the BERT-LSTM model.

[0052] Figure 6 It is the ROC curve effect diagram and confusion matrix of the ERT-GRU model.

[0053] Figure 7 It is the ROC curve effect diagram and confusion matrix of the BERT-CNN model.

[0054] Figure 8 It is the comparison diagram of the effects of the BERT-CNN, BERT-GRU, and BERT-LSTM malicious URL detection models after introducing the hierarchical attention mechanism. Detailed Embodiment

[0055] The technical solution of the present invention will be further described in detail below in conjunction with the drawings and specific embodiments.

[0056] Embodiment 1

[0057] This embodiment proposes a malicious URL detection method based on BERT-CNN and the hierarchical attention mechanism. The BERT model introducing the hierarchical attention mechanism layer is combined with three deep learning network architecture layers of CNN, LSTM, and GRU respectively for comparative experiments, so as to verify that the fusion of the CNN network and the BERT model introducing the attention mechanism can better improve the model performance and the accuracy of the experiment. Specifically:

[0058] Step 1: Obtain the URL dataset, divide the URL dataset into malicious URL and normal URL datasets, and sort out and analyze the data features;

[0059] The data of Alexa mainly comes from the toolbar installed by users to record their browsing behaviors, including the websites visited, the number of page views, etc. The existing Alexa historical dataset is obtained from the third-party platform Kaggle and used as training data to build a classification model. 1SCX-URL-2016 is an open web dataset mainly used for researching malicious URL detection. 30,000 URL datasets are collected for training the model. Phishing Tank is a platform specifically for collecting phishing website data, providing real-time updates and classification information of phishing URLs. 30,000 URL data are collected from Phishing Tank as training data. Kaggle, as a data science platform, provides various types of datasets, and a total of 40,000 URL data are collected from it. The dataset used to build the classification model contains a total of 100,000 URL data.

[0060] The dataset contains URLs and corresponding label files, for example: URL,Label feechka.ru / wdxwxoa,BAD google.com,GOOD example.com / login,BAD

[0061] The class labels of the URLs are BAD (malicious URL) and GOOD (normal URL). The annotation rules are defined to distinguish normal and abnormal URL data by whether they contain suspicious query parameters and by checking whether the path hierarchy exceeds 5 levels. Regular expressions are used to extract the path part and query parameters, and the defined rules are converted into code for automated annotation. Among them, the normal URL dataset contains 60,800 URL data, and the malicious URL dataset includes 39,200 URL data, and the ratio between the two is approximately 7:3. The data is divided into a training set and a test set, and the sample quantity ratio is adjusted to 8:2 to better evaluate the performance of the model. The accuracy and effect of the model are affected by the word vector expression method. For the detailed dataset distribution, please refer to Table 1.

[0062] Table 1 Details of Data Collection Training set (pieces) Test set (pieces) Total (pieces) Normal 52074 8726 60800 Malicious 35260 3940 39200 Total 87334 12666 100000

[0063] Step 2: By preprocessing the sample data, the dataset is refined into a training set, a test set, and a validation set, and the imbalanced data is processed;

[0064] Tokenize the URL character by character using the BERT tokenizer. Extract the URL from the text using regular expressions and encode the URL using the urllib.parse.quote() function in Python. Convert all URLs in the dataset to lowercase to unify the format, remove redundant parts such as "www" and " / ", standardize the URL to a fixed length of 120 by padding and truncation, use BERT's WordPiece tokenizer to split the URL into sub-units, and tokenize by delimiter, e.g., 'feechka.ruwdxwxoa' → '["fee", "##chka", ".", "ru", "wdxwxoa"]'.

[0065] Step 3: Perform feature extraction. Use the BERT model to convert the text into numerical features.

[0066] Through the BERT pre-trained model, input the combined vector e of the word vector Token Embeddings, the segment vector SegmentEmbeddings, and the position vector Position Embeddings. The representation of each embedding layer is as Figure 2 shown;

[0067] Because the word vectors generated by the BERT pre-trained model contain position encoding information, even if two links containing the same word are input into the malicious URL detection model designed in the present invention, as long as the words appear in different positions and different semantic environments, the output word vectors are also different; this encoding method enables the model to learn the ability of characters and the order of words before and after when processing text. Set the dimension of the BERT input word vector to demb, and set the dimensions of the position vector and the segment vector to the same size.

[0068] Encode the position of each character using sine and cosine functions with different frequencies to obtain the position vector formulas (1) - (2):

[0069] (1) (2)

[0070] Where pos represents the pos-th word, i represents the i-th dimension in the embedding, and dp represents the dimension of the attention head.

[0071] The word embedding layer converts the input words and characters into 768-dimensional word vector representations. The longest word-character sequence that the BERT model can handle is 512. One-hot encoding is introduced to generate aspect vectors and perform linear transformation. The transformed vectors are added to the word vectors e generated by BERT to reduce the classification result deviation caused by ignoring aspect encoding. The final vectors are input into the feature extraction network for learning to generate word vector representations as Figure 3 shown.

[0072] Step 4: Deploy the algorithms that fuse the attention mechanism and the convolutional neural network, the algorithms that fuse the attention mechanism and the gated recurrent unit, and the algorithms that fuse the attention mechanism and the long short-term memory network algorithm model respectively. Input the processed sample data into the models for training;

[0073] The fused attention mechanism is specifically as follows:

[0074] On the basis of adding the word attention mechanism, a sentence attention layer is added to allocate the text features and weights of the entire URL;

[0075] The word attention layer uses CNN to extract the context semantic information features of individual characters and trains them. The word attention mechanism calculates the weights of the extracted key features. The sentence attention layer uses CNN to extract the context semantic features of the entire URL text for model training. The sentence attention mechanism calculates the weights of the extracted key sentence features.

[0076] Furthermore, the word attention layer is specifically as follows:

[0077] The context embedding output by BERT is a tensor with the shape of (batch_size, seq_len, hidden_size), where the batch size batch_size = 32; the input sequence length seq_len = 50; the hidden layer dimension hidden_size = 768; the word-level attention mechanism calculates the attention scores of each word, and the word vectors U it and the context vectors U w The relationship between them is obtained through normalization to get the attention weights. The output formula of the word-level attention layer is calculated as shown in formulas (3) - (5).

[0078] (3) (4) (5)

[0079] In the formula, W it is the weight parameter, b it is the bias parameter, hit For the output information of the hidden layer state of the BERT network, and \(S_i\) is the output of the word attention layer.

[0080] Furthermore, the sentence attention layer is specifically as follows:

[0081] Taking the word-level attention representation as the input, applying the sentence-level attention mechanism to calculate the importance weight at the sentence level, including a fully connected layer for mapping the hidden layer dimension hidden_size of BERT to a scalar attention score; the sentence-level attention mechanism calculates the text attention score and obtains the sentence vector \(U\) through the activation function softmax it and the context vector \(U\) w The relationship between them is used to obtain the attention weight through normalization. The output formula of the sentence-level attention layer is calculated as shown in formulas (6) - (8):

[0082] (6) (7) (8)

[0083] In the formulas, \(W\) i is the weight parameter, \(b\) i is the bias parameter, \(h\) i is the output information of the hidden layer state of the BERT network, and \(O\) i is the output of the sentence attention layer.

[0084] The convolutional network layer is specifically as follows:

[0085] In the malicious URL classification task of the BERT-CNN model based on the hierarchical attention mechanism, the convolutional network module is responsible for extracting the local features of the URL text; using one-dimensional convolution (1D CNN) to process the context embedding representation output by BERT;

[0086] Define the number of convolutional kernels as 128, the size \(k\) of each convolutional kernel is 3, the height of the convolutional kernel, that is, the number of words covered, is 3, and the embedding dimension of BERT is 768; in sequence data, one-dimensional convolution Conv1D is more applicable than two-dimensional convolution, which can directly model the local dependencies of the sequence and the relationships between adjacent time points, avoiding the redundant calculations introduced by the false spatial dimension in two-dimensional convolution;

[0087] Selecting a convolutional kernel with \(k = 3\) can effectively capture short-range pattern trigrams, and at the same time, by stacking multiple small kernels, a receptive field similar to that of a large kernel can be achieved;

[0088] Performing average pooling on the feature map output by the convolution to reduce the feature dimension and retain important information, and the formula is as shown in formula (9):

[0089] (9)

[0090] Concatenate the pooling outputs of all convolutional kernels to generate the final output vector.

[0091] Step 5: Evaluate the model using evaluation metrics to determine whether the model can accurately identify malicious URLs and normal URLs.

[0092] The output network layer is specifically:

[0093] Malicious URL detection is a classification problem. In the present invention, a fully connected layer is used as the output network for malicious URL detection and classification; by taking the output 0 of the sentence attention layer in the hierarchical attention mechanism as the input of the next fully connected layer, and then normalizing the vector output by the fully connected layer through the softmax function, the output vector f of the model is finally obtained. The calculation formula is as shown in Equation (10):

[0094] (10)

[0095] In the formula, W o is the weight parameter, and b is the bias parameter;

[0096] The test result only outputs one predicted category, and the category with the largest probability value is selected as the classification result of the URL; when the model is trained on the training set, by comparing the predicted category output by the fully connected layer with the actual category, the training error of the model is obtained, and then the model parameters are continuously trained through forward propagation and backward propagation until a model with better classification effect is obtained;

[0097] Malicious URL detection is a classification problem. The training of the model in the present invention adopts a supervised learning framework to use the backpropagation method to take the derivative of the loss function with respect to all model parameters, adopts the Adam optimization method to optimize and update all parameters, and uses the minimization of the cross-entropy loss function as the objective function to optimize the model. The formula is as shown in Equation (11):

[0098] (11)

[0099] In the formula, D represents the size of the data volume of the training set, C represents the number of categories for URL classification, and the size of C in the task is 2, is the predicted category, y is the actual category of the data, is the cross-regularization term.

[0100] Model parameter configuration:

[0101] The deep learning framework PyTorch was adopted as the basic experimental framework, and Nvidia GPUs were used for the calculation work of this model, so as to achieve the purpose of improving the model training speed and reducing the time consumption required for model training. Through multiple rounds of iterative training, the parameters required by the model were optimized and adjusted, and finally the model parameter configuration shown in Table 2 was selected.

[0102] Table 2 Model Parameter Configuration

[0103] Evaluation Metrics:

[0104] The confusion matrix was adopted. Positive values indicate malicious URLs, and negative values indicate normal URLs. The structure of the CM is shown in Table 3.

[0105] Table 3 Confusion Matrix

[0106] The statistical method of the confusion matrix is an evaluation metric for the performance of the malicious URL classification model. We divide the confusion matrix into four types: true positive (TP), false negative (TN), true negative (TN), and false positive (FP). The experiment provides evaluation metrics by calculating Recall, Precision, F1 Score, Receiver Operating Characteristic (ROC), Area Under Curve (AUC), and confusion Matrix. Usually, the model has four types of values, as shown in the confusion matrix. Positive values indicate malicious URLs. The hybrid matrix constructed by the four metrics of accuracy, precision, recall, and F1 can effectively measure the effect of the malicious URL detection algorithm proposed in the research. Calculate each parameter of the confusion matrix accurately.

[0107] Example 2: Experimental Comparison of the BERT-CNN Model before and after Introducing the Hierarchical Attention Mechanism

[0108] The improved attention mechanism was introduced, and the BERT-CNN model was optimized through the hierarchical attention mechanism with the aim of improving the model performance and enhancing the long sequence modeling ability. In order to compare the BERT models before and after the introduction of the attention mechanism, the following experiment was designed: Through the classification experiment of the basic model, the original BERT-CNN model and the improved model with the hierarchical attention mechanism introduced in the BERT-CNN model on the URL dataset, the performance was evaluated with the accuracy as the evaluation metric, as shown in Table 4.

[0109] Table 4 Comparison of Training Results between the Basic BERT-CNN Model and the BERT-CNN Model with the Hierarchical Attention Mechanism Introduced BERT-CNN BERT-CNN (Hierarchical attention mechanism) Accuracy 94.3% 96.8% F1-score 93.2% 95.3% Training time (hours) 2.5 2.1 Memory occupancy (GB) 8 7

[0110] The experimental results show that the BERT-CNN model with a hierarchical attention mechanism is slightly superior to the standard BERT model in terms of accuracy and F1 score. The accuracy has increased from 94.3% to 96.8%, and the F1 score has increased from 93.2% to 95.3%. Although the performance improvement is small, the hierarchical attention mechanism enables the model to better capture key information in the URL text by dynamically allocating attention weights, thus improving the classification effect. In addition, the introduction of the hierarchical attention mechanism has significantly reduced the training time and memory usage. The training time has been shortened from 2.5 hours to 2.1 hours, and the memory usage has been reduced from 8GB to 7GB, indicating that this mechanism has obvious advantages in improving the model efficiency. In addition, the experiment visualized the accuracy curve and loss curve after training the BERT baseline model and the model with the hierarchical attention mechanism module introduced. The results are as Figure 4 shown.

[0111] By analyzing the accuracy curve and loss curve after training the BERT baseline model and the model with the hierarchical attention mechanism module introduced, the training loss of the model with the hierarchical attention mechanism module introduced is 0.08, and the test loss is 0.07; the training accuracy is 0.9682, and the test accuracy is 0.9658; the training loss of the BERT model without the hierarchical attention mechanism module introduced is 0.09, and the test loss is 0.10; the training accuracy is 0.9322, and the test accuracy is 0.9402.

[0112] Example 3: Comparison chart of the BERT model with a hierarchical attention mechanism and different deep learning networks in experiments

[0113] The BERT model with the hierarchical attention mechanism module introduced was combined with the three deep learning network architectures of CNN, LSTM, and GRU respectively for comparative experiments and trained on the same URL dataset. The experimental results were visualized, and the ROC curves and confusion matrix result charts of the BERT-LSTM, BERT-GRU, and BERT-CNN models are as Figures 5 - 8 shown;

[0114] By analyzing the experimental ROC curve chart and confusion matrix results, it is concluded that after introducing the BERT model with the combined attention mechanism, the AUC value of the BERT-LSTM model in the three research model network frameworks is 0.86, confirming that this model has good classification performance. The AUC value of the BERT-GRU model is 0.95, confirming that this model has excellent classification performance. The AUC value of the BERT-CNN model is 0.98, confirming that this model has excellent classification performance. After comparison, the BERT-CNN model has the highest AUC value, and the visualized ROC curve and confusion matrix have the best effect.

[0115] After experimental comparison and result analysis, it is found that the performance of three deep learning-based algorithms, namely BERT-CNN, BERT-GRU, and BERT-LSTM, is good. Among them, the BERT-GRU model with a hierarchical attention mechanism module can better capture the long-range associations in the dataset and utilize the advantages of the genetic algorithm to adjust the weights of the dataset, thus greatly improving the accuracy of the classification results. The BERT-CNN model algorithm with an attention mechanism has the best performance. The BERT-CNN model can process a large amount of data information more comprehensively and quickly identify problems. BERT-CNN can extract a large number of significant features from the URL dataset. After being processed by the BERT-CNN model with a hierarchical attention mechanism module, its performance is higher than that of the model without the hierarchical attention mechanism module, and the accuracy can reach 96%.

[0116] Example 4: Performance Evaluation of Three Model Algorithms Based on Deep Learning

[0117] The experiment was compared with the malicious URL detection experiment using the traditional random forest algorithm and evaluated by accuracy, precision, recall, and F1-score. The experimental results were organized into Table 5. From the results in Table 5, it can be seen that among the three deep learning-based model algorithms with a hierarchical attention mechanism module, the BERT-CNN model has the highest accuracy, reaching 96%, which is the highest among the four model algorithms, indicating that the proportion of correctly predicted samples is the highest. The accuracy of the BERT-GRU model is 93%, following closely. The accuracy of the BERT-LSTM model is 87%, which is relatively low. The accuracy of the random forest model is 81%, which is the lowest among the four models. The comparison with the random forest model highlights the advantages of deep learning models in complex feature extraction and non-linear relationship modeling: Random forests rely on hand-crafted features and the integration of decision trees, are suitable for structured data and have strong interpretability, but may be difficult to capture deep patterns in sequential or high-dimensional data; Deep learning models can automatically learn multi-level features, and the comparison results can verify that deep learning has a significant performance improvement in the malicious URL detection task.

[0118] Table 5 Experimental Evaluation Results of Different Classification Model Performances Model Accuracy Precision Recall F1-score BERT-CNN 96% 96% 96% 96% BERT-GRU 93% 93% 93% 93% BERT-LSTM 87% 88% 86% 88% Random forest 81% 81% 81% 81%

[0119] The present invention verifies the performance of the proposed model and conducts comparative experiments, ablation experiments, and visualization experiments on a publicly available malicious URL detection dataset. Through experimental verification, the BERT-CNN model based on the hierarchical attention mechanism achieves an accuracy of 96.8% in detecting new malicious URLs, which is 2.5% higher than the baseline model in terms of detection accuracy; the F1 score reaches 95.3%, which is 2.1% higher than the baseline model in terms of the F1 score. The experimental results show that the malicious URL detection model with the attention mechanism introduced has significant advantages in capturing global and local features of URLs.

[0120] The present invention implements a malicious URL detection algorithm BERT-CNN with an improved attention mechanism introduced. Through comparative experiments with the BERT model without the hierarchical attention mechanism on the URL dataset, it is verified that the accuracy of the deep neural network with the hierarchical attention mechanism introduced is the highest, reaching 96%. The BERT-CNN model successfully captures the features crucial for malicious URL detection in complex network traffic data by introducing the improved attention mechanism, significantly improving the detection accuracy and efficiency. The experimental results show that compared with the traditional abnormal URL detection model using the random forest algorithm, the BERT-CNN model with the hierarchical attention mechanism introduced shows higher classification accuracy on the sample dataset. This achievement not only proves the effectiveness of the attention mechanism in enhancing the model's feature extraction ability but also reflects the great potential of the BERT-CNN model with the hierarchical attention mechanism introduced in the field of network security, especially in the malicious URL detection task. However, although the BERT-CNN model with the hierarchical attention mechanism introduced has better performance than traditional methods, it still faces certain challenges when dealing with extremely complex or highly confused data. In addition, especially in resource-constrained environments, the computational complexity and training time of the model are also factors that need to be considered.

[0121] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A malicious URL detection method based on BERT-CNN and hierarchical attention mechanism, characterized in that It includes the following steps: S1. Obtain the URL dataset, divide the URL dataset into malicious URLs and normal URL datasets, and organize and analyze the data features; S2. Through preprocessing the sample data, refine the dataset into a training set, a test set, and a validation set, and process the imbalanced data; S3. Perform feature extraction, and use the BERT model to convert the text into numerical features; S4. Deploy algorithms that fuse the attention mechanism and the convolutional neural network, algorithms that fuse the attention mechanism and the gated recurrent unit, and algorithms that fuse the attention mechanism and the long short-term memory network algorithm models respectively, and input the processed sample data into the models for training; S5. Evaluate the models through evaluation metrics to determine whether the models can accurately identify malicious URLs and normal URLs.

2. The malicious URL detection method based on BERT-CNN and hierarchical attention mechanism according to claim 1, characterized in that In step S4, the fused attention mechanism is adopted, specifically: On the basis of adding the word attention mechanism, a sentence attention layer is added to allocate the text features and weights of the entire URL; The word attention layer uses CNN to extract the context semantic information features of individual characters and conducts training, and the word attention mechanism calculates the weights of the extracted key features; The sentence attention layer uses CNN to extract the context semantic features of the entire URL text for model training, and the sentence attention mechanism calculates the weights of the extracted key sentence features.

3. The malicious URL detection method based on BERT-CNN and hierarchical attention mechanism according to claim 2, characterized in that The word attention layer is specifically: The context embeddings output by BERT are tensors of shape (batch_size, seq_len, hidden_size), where the batch size batch_size = 32; the input sequence length seq_len = 50; the hidden layer dimension hidden_size = 768; the word-level attention mechanism calculates the attention scores for each word, and the word vector U is obtained through the activation function softmax it and the context vector U w The relationship between them is used to obtain the attention weights through normalization. The output formula of the word-level attention layer is calculated as shown in formulas (3) - (5): （3） （4） （5） where, W it is the weight parameter, b it is the bias parameter, h it is the output information of the hidden layer state of the BERT network, and Si is the output of the word attention layer.

4. The malicious URL detection method based on BERT-CNN and hierarchical attention mechanism according to claim 2, wherein The sentence attention layer is specifically: Taking the word-level attention representation as the input, applying the sentence-level attention mechanism, and calculating the importance weights at the sentence level, including a fully connected layer for mapping the hidden layer dimension hidden_size of BERT to a scalar attention score; The sentence-level attention mechanism calculates the attention scores of the text, and passes the sentence vector U through the activation function softmax it and the context vector U w The relationship between them is obtained through normalization to get the attention weights. The output formula of the sentence-level attention layer is calculated as shown in Formulas (6) to (8): （6） （7） （8） Where, W i is the weight parameter, b i is the bias parameter, h i is the output information of the hidden layer state of the BERT network, and O i is the output of the sentence attention layer.

5. The malicious URL detection method based on BERT-CNN and hierarchical attention mechanism according to claim 1, characterized in that, In step S4, the convolutional network layer is specifically: In the malicious URL classification task of the BERT-CNN model based on the hierarchical attention mechanism, the convolutional network module is responsible for extracting the local features of the URL text; use one-dimensional convolution (1D CNN) to process the context embedding representation output by BERT; Define the number of convolutional kernels as 128, the size k of each convolutional kernel as 3, the height of the convolutional kernel, that is, the number of covered words as 3, and the embedding dimension of BERT as 768; in sequential data, one-dimensional convolution Conv1D is more applicable than two-dimensional convolution, which can directly model the local dependencies of the sequence and the relationships between adjacent time points, and avoid the redundant calculations introduced by the false spatial dimension of two-dimensional convolution; Selecting a convolutional kernel with k = 3 can effectively capture short-range pattern trigrams, and at the same time achieve a receptive field similar to that of a large kernel by stacking multiple small kernels; Perform average pooling on the feature map output by the convolution to reduce the feature dimension and retain important information, as shown in formula (9): （9） Concatenate the pooling outputs of all convolutional kernels together to generate the final output vector.

6. The malicious URL detection method based on BERT-CNN and hierarchical attention mechanism according to claim 1, characterized in that In step S5, the output network layer is specifically: Malicious URL detection is a classification problem. The present invention uses a fully connected layer as the output network for malicious URL detection and classification; by taking the output 0 of the sentence attention layer in the hierarchical attention mechanism as the input of the next fully connected layer, and then normalizing the vector output by the fully connected layer through the softmax function, finally obtaining the output vector f of the model, and the calculation formula is shown in Equation (10): （10） where W o is the weight parameter and b is the bias parameter; The test result only outputs one predicted category, and the category with the largest probability value is selected as the classification result of the URL; When the model is trained on the training set, the training error of the model is obtained by comparing the predicted category output by the fully connected layer with the actual category, and then the model parameters are continuously trained through forward propagation and backward propagation until a model with better classification effect is obtained; Malicious URL detection is a classification problem. The training of the model of the present invention adopts a supervised learning framework to use the backpropagation method to take the derivative of the loss function with respect to all model parameters, adopts the Adam optimization method to optimize and update all parameters, and uses the minimization of the cross-entropy loss function as the objective function to optimize the model, and the formula is shown in Equation (11): （11） Where D represents the size of the data volume in the training set, C represents the number of categories for URL classification, and the size of C in the task is 2, is the predicted category, and y is the actual category of the data, is the cross-regularization term.

Images