An EMO-GAN-based malicious URL detection method

Abstract

The application belongs to the technical field of network security, and particularly relates to a malicious URL detection method based on EMO-GAN. The application realizes semi-supervised learning by introducing extremely randomized trees (ET) and combining with a margin generative adversarial network (MarginGAN), effectively overcoming the data acquisition difficulties caused by data imbalance and label scarcity. In addition, the application also deeply explores a new feature extraction method and an online learning technology, providing more accurate feature representation and stronger model expandability for malicious URL detection. The method aims to solve the data acquisition difficulties, insufficient feature representation and model concept drift challenges encountered in malicious URL detection. Through experimental verification, the application achieves good results on real-world data sets. Therefore, the application can be used for detecting malicious URLs and maintaining network space security.

Description

Technical Field

[0001] This invention belongs to the technical field of network security, specifically relating to a malicious URL detection method based on EMO-GAN. This malicious URL detection method aims to address challenges encountered in malicious URL detection, such as difficulties in data acquisition, insufficient feature representation, and model concept drift. Background Technology

[0002] With the rapid advancement of the digital age, the World Wide Web has become central to the daily lives and work of organizations and individuals. However, this has been accompanied by a dramatic increase in cyber threats. According to the information security vulnerability report released by the China National Vulnerability Database (CNNVD), cross-site scripting (XSS), SQL injection, and phishing attacks have consistently ranked high on the threat list. In these attacks, attackers carefully construct malicious URLs, disguising them as legitimate requests to lure users or bypass security mechanisms, thereby achieving illegal purposes. A malicious URL refers to a URL link used to launch a cyberattack. In the field of cybersecurity, malicious URLs have become a serious threat, with diverse attack types and far-reaching consequences. The frequent occurrence of common malicious URL attacks, including phishing, SQL injection, and cross-site scripting attacks, has already caused many serious consequences. Timely and effective detection of malicious URLs is crucial for addressing cybersecurity threats.

[0003] Currently, malicious URL detection mainly relies on two methods: blacklists and machine learning. While blacklisting techniques can effectively defend against known malicious URLs, they are limited by their update speed and attacker bypass techniques, resulting in lag and incomplete detection, often leading to unsatisfactory results. Machine learning-based methods, on the other hand, extract features and train models using algorithms. With training on large amounts of data, they can accurately identify complex and subtle anomaly patterns and exhibit higher generalization and accuracy. However, most of these methods rely on balanced datasets with sufficient labels. In practical applications, uneven data distribution and scarce labels become bottlenecks limiting their performance. Secondly, the feature extraction strategies employed in many methods often have limitations. Some methods, while performing well on specific models, have weak generalization capabilities and struggle to adapt to different scenarios; while others, due to relatively limited extracted feature information, leave room for improvement in the final model's detection performance. Furthermore, most methods are designed and applied offline. This offline mode means that the model is trained on a fixed dataset and is only applicable to static malicious URL detection scenarios. However, in real-world network environments, the distribution of malicious URLs changes with the emergence of new attack methods. Therefore, models trained solely on previously acquired datasets often experience a gradual decline in detection performance when faced with dynamic changes in the original sample distribution, making it difficult to maintain a high accuracy rate.

[0004] With the development of machine learning, semi-supervised learning and generative adversarial networks (GANs) have become effective means to solve the problems of imbalanced source data and scarce labels. The idea behind semi-supervised learning is to improve learning performance by utilizing unlabeled data and reduce the model's dependence on labeled data. Generative adversarial networks (GANs), due to their ability to generate highly realistic data, are often used to augment original datasets in malicious URL detection. Furthermore, the generative properties of GANs and the semi-supervised approach can work together to further improve model performance.

[0005] Currently, many detection models rely heavily on a single source for feature extraction, resulting in relatively limited information representation. While manually selected features demonstrate excellent accuracy and efficiency, inappropriate feature selection can negatively impact overall detection accuracy. In contrast, features automatically derived from the URL string itself by algorithms, without relying on third-party intervention or prior knowledge, can provide valuable supplements to the model. These automatically generated features exhibit positive effects on data representation, helping the model to more accurately identify data categories, thereby improving detection accuracy and reliability.

[0006] Most existing malicious URL detection models are based on offline methods, meaning they are trained on fixed datasets and applied to static malicious URL detection scenarios. However, in real-world applications, the distribution of malicious URLs constantly changes with the emergence of new attack methods. Therefore, models trained on previously acquired datasets gradually deteriorate in performance when faced with changes in the original sample distribution. To address this concept drift problem, scalable online learning has been introduced and extensively studied. Online learning emphasizes the real-time nature and accuracy of models, allowing them to receive new data and update themselves in real time, thus adapting to rapidly changing data environments. Furthermore, it solves the cost problem associated with retraining. Summary of the Invention

[0007] This invention addresses three core challenges in malicious URL detection: the difficulty of data acquisition, the challenge of feature representation, and the concept drift problem faced by the model. To effectively solve these challenges, this invention proposes a malicious URL detection method based on EMO-GAN. This framework combines semi-supervised learning with generative adversarial networks to effectively overcome the difficulty of data acquisition; by fusing features from different sources, it greatly enriches the data representation capabilities; and by utilizing an online learning strategy, the framework can adapt to and respond to the concept drift phenomenon of the model in real time, ensuring the accuracy and real-time performance of the detection.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: a malicious URL detection method based on EMO-GAN, which includes the following steps:

[0009] Step 1: Data preprocessing: Use web crawlers to obtain URL data from web pages, and then clean, decode, and standardize it to obtain sample data;

[0010] Step 2, Feature Engineering: Use Python tools to build a feature extraction module. The feature extraction module includes a statistical feature extraction submodule, a character feature extraction submodule, a vocabulary feature extraction submodule, and a fusion feature submodule. Input the sample data obtained in Step 1 into the statistical feature extraction submodule, the character feature extraction submodule, and the vocabulary feature extraction submodule respectively to obtain 128-dimensional statistical features, 64-dimensional character features, and 64-dimensional vocabulary features. Input the three types of features into the fusion feature submodule and merge to output 256-dimensional features, where the 256-dimensional features include labeled sample features and unlabeled sample features.

[0011] Step 3, Detection Model Construction and Training: Step 3.1, Model Construction: The detection model includes an extremely random tree classifier module, a generator module, a discriminator module, and a classifier module. The extremely random tree classifier module is built using the Scikit-learn library, and the generator, discriminator, and classifier modules are built using the PyTorch framework. Step 3.2, Model Training: The training rounds of the model are pre-set. The 256-dimensional features obtained in Step 2 are input into the detection model for training. The extremely random tree classifier module is trained using labeled features. The random distribution is input into the generator module to output 256-dimensional generated sample features. The generated sample features and unlabeled sample features are input into the trained extremely random tree classifier module to add labels. All labeled features are input into the discriminator module and the classifier module for training. The generator module is trained adversarially against the discriminator module and the classifier module.

[0012] Step 4: Model performance evaluation: The macro average of precision, recall, and F1 score is used as the evaluation metric for the model. If the precision is greater than 99% and the F1 score is greater than 80%, proceed to step 5; otherwise, proceed to step 3.2 to continue model training.

[0013] Step 5, Detection of Malicious URLs: Collect URL data from real web pages as test data, process the test data according to Step 1 and Step 2 to obtain 256-dimensional features, and then input them into the pre-trained classifier module for classification prediction to obtain the corresponding labels and complete the detection.

[0014] Step 6: Online update of the detection model: The predicted labels obtained in Step 5 are verified by human feedback to obtain the true labels of the test data. The 256-dimensional features of the test data obtained in Step 5 and the true labels of human feedback are input into the classifier module for online learning and updating.

[0015] As further illustrated in this embodiment, the specific steps of data preprocessing in step 1 are as follows:

[0016] Step 1.1, Data Acquisition: Use a web crawler to obtain URL data from the webpage, which will serve as the input for Step 1.2;

[0017] Step 1.2, Data Cleaning: Use the Python programming language to write a script to delete URL strings that belong to both malicious and benign categories. The resulting data will be used as input for Step 1.3.

[0018] Step 1.3: Data Decoding: Use the Python programming language to write a script to decode the encoded special characters and hexadecimal numbers in the URL data into their original form. The resulting data will be used as the input for Step 1.4.

[0019] Step 1.4: Data Standardization: Use the Python programming language to write a script to uniformly replace non-ASCII characters in the URL data with a fixed string str, so as to obtain clean and standard sample data.

[0020] As further illustrated in this embodiment, the specific steps of feature engineering in step 2 are as follows:

[0021] Step 2.1, Statistical Feature Extraction: In the statistical feature extraction submodule, the sample data obtained in Step 1.4 is used as input. A script is written using the Python programming language to calculate a total of 128 statistical indicators for the URL as statistical features. The statistical indicators include domain name length, number of decimal points in the domain name, whether the domain name contains -, top-level domain name length, path length, number of / , whether the protocol is HTTPS, number of &, total URL length, whether it contains redirects, whether it contains short links, character frequency, and keyword frequency.

[0022] Step 2.2, Character Feature Extraction: In the character feature extraction submodule, the sample data obtained in step 1.4 is used as input. A script is written using the Python programming language to convert each character of the URL into ASCII encoding and use it as a character feature. The character feature dimension of each URL is set to 64. For URLs with fewer than 64 characters, a zero-padding strategy is adopted; for URLs with more than 64 characters, truncation is performed.

[0023] Step 2.3, Lexical Feature Extraction: In the lexical feature extraction submodule, the sample data obtained in Step 1.4 is used as input. A script is written using the Python programming language. First, word segmentation is performed based on regular expressions. Second, the word frequency-inverse document frequency of each word is calculated, and the top 500 words are selected as the dictionary based on this as the filtering standard. Then, the Word2Vec model is trained using the dictionary to generate lexical features. Finally, the mean of all word vectors in the URL is calculated to obtain a 64-dimensional vector as the lexical feature representation of the URL.

[0024] Step 2.4, Feature Fusion: In the feature fusion submodule, the three character features obtained in steps 2.1, 2.2 and 2.3 are combined to obtain a total of 256-dimensional sample feature representation.

[0025] As a further illustration of this embodiment, the specific steps for constructing and training the detection model in step 3 are as follows:

[0026] Step 3.1, Building and training the extremely random tree classifier module: The extremely random tree classifier module is built using the Scikit-learn library. The labeled part of the sample features obtained in Step 2.4 is selected as the input of the extremely random tree classifier. The parameters of the extremely random tree are set, and then training is performed.

[0027] Step 3.2, Labeling: Use the trained Extremely Random Tree classifier obtained in Step 3.1 to predict the unlabeled sample features in the sample features obtained in Step 2.4, and use the prediction results as pseudo-labels;

[0028] Step 3.3, Building and training the generator module: Based on the PyTorch framework, several linear and convolutional layers are built as the generator. A 62-dimensional random distribution is selected as the input, and the output is a 256-dimensional feature. The output feature is used to obtain pseudo-labels through the trained extremely random tree classifier obtained in Step 3.1. Adversarial training with the classifier module is achieved by minimizing the cross-entropy loss between the pseudo-label and the predicted label of the classifier module. Adversarial training with the discriminator module is achieved by minimizing the cross-entropy loss between the predicted label and the real label of the discriminator module.

[0029] Step 3.4, construction and training of the discriminator module: Based on the PyTorch framework, several linear and convolutional layers are constructed as discriminators. The input includes the real samples obtained in step 2.4, the generated samples obtained in step 3.3, and the labels of the two types of data obtained in step 3.2. Adversarial training with the generator module is achieved by minimizing the cross-entropy loss between the real samples and the real labels and minimizing the cross-entropy loss between the generated samples and the generated labels.

[0030] Step 3.5, Classifier Module Construction and Training: Based on the PyTorch framework, several linear layers are constructed as classifiers. The inputs include the real samples obtained in Step 2.4, the generated samples obtained in Step 3.3, and the labels of the two types of data obtained in Step 3.2. For labeled samples, the classifier module is trained by minimizing the cross-entropy loss between the real label and the predicted label; for unlabeled samples, the classifier module is trained by minimizing the cross-entropy loss between the pseudo-label and the predicted label; for generated samples, the classifier module is trained by maximizing the cross-entropy between the pseudo-label and the predicted label.

[0031] As further illustrated in this embodiment, in step 3.5, the classifier is a multilayer perceptron structure containing five linear layers and one output layer. A separate linear layer is introduced after each linear layer as the output layer network, and its output is weighted to serve as the final prediction result. The classifier employs a hedging algorithm to learn the weighting parameters. In the initial iteration, weights are evenly distributed to each output layer. In subsequent iterations, the loss is calculated based on the classifier's predictions and the true labels, and the weights are updated accordingly. This allows the classifier module to automatically assign appropriate weights to each hidden layer, automatically adapting to different network depths and ensuring compatibility with online scenarios.

[0032] As a further explanation of this embodiment, the specific steps for malicious URL detection in step 5 are as follows:

[0033] Step 5.1, Data Preprocessing: The URL data to be tested is cleaned through Step 1.2, Decoding, and Standardization to obtain clean and standard sample data.

[0034] Step 5.2, Feature Engineering: The sample data obtained in Step 5.1 is processed through Steps 2.1 to 2.4 to obtain 256-dimensional features;

[0035] Step 5.3, Anomaly Detection: Input the 256-dimensional features obtained in Step 5.2 into the classifier module trained in Step 3.5 for prediction, and obtain label 1 indicating a prediction of malice and label 0 indicating a prediction of benignity.

[0036] As a further explanation of this embodiment, the specific steps for online updating of the detection model in step 6 are as follows:

[0037] Step 6.1, Manual Feedback: After obtaining the prediction results of the URL to be tested in Step 5.3, the prediction results are manually reviewed, and the true tags of the URL data to be tested are fed back based on the review results.

[0038] Step 6.2, Online Update: Input the 256-dimensional features obtained in Step 5.2 and the true labels obtained in Step 6.1 into the classifier module and perform training and update according to Step 3.5.

[0039] As a further explanation of this embodiment, in step 4, the performance evaluation index is calculated as follows:

[0040] Accuracy Macro Average ,

[0041] In the formula: ;

[0042] Macro average recall ,

[0043] In the formula: ;

[0044] F1 Score Macro Average ,

[0045] In the formula: ;

[0046] Accuracy: ;

[0047] Wherein, TP indicates that a malicious URL is correctly identified as malicious, TN indicates that a legitimate URL is correctly identified as legitimate, FP indicates that a legitimate URL is incorrectly identified as malicious, FN indicates that a malicious URL is incorrectly identified as legitimate, and n indicates the number of categories.

[0048] Compared with existing technologies, this invention proposes a malicious URL detection method based on EMO-GAN, the benefits of which are mainly reflected in the following four aspects:

[0049] 1. This invention combines Extremely Random Trees (ET) with MarginGAN (Marginal Generative Adversarial Network) to achieve efficient data processing and model training in situations of imbalanced data and scarce labels. This innovative solution effectively overcomes the difficulty of data acquisition and improves the model's generalization ability and accuracy.

[0050] 2. This invention proposes a feature extraction module that integrates statistical features, character features, lexical features, and fused features. This module can comprehensively capture key information in URLs and effectively improve feature information, providing a more accurate feature representation for malicious URL detection. This comprehensive feature extraction method enables the model to more accurately identify malicious URLs.

[0051] 3. This invention improves the MarginGAN model by introducing a semi-supervised learning mechanism and applies it to the malicious URL detection task. This mechanism can make full use of limited labeled and unlabeled data, achieving efficient learning in imbalanced and insufficiently labeled data scenarios. This not only improves the model's training efficiency but also enhances its robustness.

[0052] 4. This invention proposes a malicious URL classifier that supports online learning, possessing excellent scalability and adaptability. It can update the model in real time to adapt to new attack methods, effectively addressing the continuous changes and evolution of malicious URLs. This online learning capability makes the malicious URL detection system of this invention more flexible and reliable in practical applications. Attached Figure Description

[0053] Figure 1 This is an architecture diagram of the malicious URL detection method in this invention;

[0054] Figure 2 A detailed diagram of the statistical characteristics in this invention;

[0055] Figure 3 This refers to the character frequency component in the statistical features of this invention.

[0056] Figure 4 This refers to the keyword frequency component in the statistical features of this invention.

[0057] Figure 5 This is a structural diagram of the EMO-GAN in this invention;

[0058] Figure 6 This is a comparative experimental result diagram showing the evaluation of the feature extraction method under 100% labeling in this invention;

[0059] Figure 7 This is a comparative experimental result diagram showing the evaluation of the feature extraction method under 10% labeling in this invention;

[0060] Figure 8 This is a comparative experimental result diagram of the feature extraction method under 5% label in this invention;

[0061] Figure 9 The figure shows the experimental results comparing the performance of the model in this invention with other models;

[0062] Figure 10 The figure shows the experimental results for evaluating the online learning performance of the classifier in this invention.

[0063] Figure 11 The figure shows the experimental results for evaluating the performance of the model on different datasets in this invention;

[0064] Figure 12 The figure shows the experimental results comparing the model in this invention with other mainstream models. Detailed Implementation

[0065] To better understand this invention, the following description is in conjunction with the appendix. Figures 1 to 12 The present invention is further explained, but the content of the present invention is not limited to the following embodiments.

[0066] As attached Figure 1As shown, the malicious URL detection method used in this invention mainly consists of two modules: a feature engineering module, whose main function is to provide richer and more comprehensive data information representation by fusing features collected from three channels: statistical features, character features, and lexical features, facilitating subsequent model use; and a detection model module, whose main functions are: first, to train the classifier using the EMO-GAN model; and second, to use the classifier to detect malicious URLs. The overall process is as follows: First, the preprocessed samples enter the feature engineering module, where the three types of features are obtained and fused. Then, the features are input into the detection model module, where there are three types of data: labeled real data features, unlabeled real data features, and generated sample features generated by a generative adversarial network. The labels for the latter two are obtained through an extremely random tree trained on labeled data. Finally, the three types of data are trained adversarially on the EMO-GAN model to obtain an efficient classifier. After these operations, the resulting classifier can achieve high detection accuracy in imbalanced and scarce data environments. It can also learn online based on human feedback or newly arrived URL data, updating and improving the model to make it more adaptable and capable of detecting constantly changing malicious URLs.

[0067] A malicious URL detection method based on EMO-GAN, characterized by the following steps:

[0068] Step 1: Data preprocessing: Use web crawlers to obtain URL data from web pages, and then clean, decode, and standardize it to obtain sample data;

[0069] Step 2, Feature Engineering: Use Python tools to build a feature extraction module. The feature extraction module includes a statistical feature extraction submodule, a character feature extraction submodule, a vocabulary feature extraction submodule, and a fusion feature submodule. Input the sample data obtained in Step 1 into the statistical feature extraction submodule, the character feature extraction submodule, and the vocabulary feature extraction submodule respectively to obtain 128-dimensional statistical features, 64-dimensional character features, and 64-dimensional vocabulary features. Input the three types of features into the fusion feature submodule and merge to output 256-dimensional features, where the 256-dimensional features include labeled sample features and unlabeled sample features.

[0070] Step 3, Detection Model Construction and Training: Step 3.1, Model Construction: The detection model includes an extremely random tree classifier module, a generator module, a discriminator module, and a classifier module. The extremely random tree classifier module is built using the Scikit-learn library, and the generator, discriminator, and classifier modules are built using the PyTorch framework. Step 3.2, Model Training: The training rounds of the model are pre-set. The 256-dimensional features obtained in Step 2 are input into the detection model for training. The extremely random tree classifier module is trained using labeled features. The random distribution is input into the generator module to output 256-dimensional generated sample features. The generated sample features and unlabeled sample features are input into the trained extremely random tree classifier module to add labels. All labeled features are input into the discriminator module and the classifier module for training. The generator module is trained adversarially against the discriminator module and the classifier module.

[0071] Step 4: Model performance evaluation: The macro average of precision, recall, and F1 score is used as the evaluation metric for the model. If the precision is greater than 99% and the F1 score is greater than 80%, proceed to step 5; otherwise, proceed to step 3.2 to continue model training.

[0072] Step 5, Detection of Malicious URLs: Collect URL data from real web pages as test data, process the test data according to Step 1 and Step 2 to obtain 256-dimensional features, and then input them into the pre-trained classifier module for classification prediction to obtain the corresponding labels and complete the detection.

[0073] Step 6: Online update of the detection model: The predicted labels obtained in Step 5 are verified by human feedback to obtain the true labels of the test data. The 256-dimensional features of the test data obtained in Step 5 and the true labels of human feedback are input into the classifier module for online learning and updating.

[0074] In the above embodiment, the specific steps of data preprocessing in step 1 are as follows:

[0075] Step 1.1, Data Acquisition: Use a web crawler to obtain URL data from the webpage, which will serve as the input for Step 1.2;

[0076] Step 1.2, Data Cleaning: Use the Python programming language to write a script to delete URL strings that belong to both malicious and benign categories. The resulting data will be used as input for Step 1.3.

[0077] Step 1.3: Data Decoding: Use the Python programming language to write a script to decode the encoded special characters and hexadecimal numbers in the URL data into their original form. The resulting data will be used as the input for Step 1.4.

[0078] Step 1.4: Data Standardization: Use the Python programming language to write a script to uniformly replace non-ASCII characters in the URL data with a fixed string str, so as to obtain clean and standard sample data.

[0079] In the above embodiment, the specific steps of feature engineering in step 2 are as follows:

[0080] Step 2.1, Statistical Feature Extraction: In the statistical feature extraction submodule, the sample data obtained in Step 1.4 is used as input. A script is written using the Python programming language to calculate a total of 128 statistical indicators for the URL as statistical features. These indicators include domain name length, number of decimal points in the domain name, whether the domain name contains "-", top-level domain length, path length, number of " / ", whether the protocol is HTTPS, number of "&", total URL length, whether it contains redirects, whether it contains shortened links, character frequency, and keyword frequency; see attached. Figure 2 The feature items shown are attached. Figure 3 Indicates appendix Figure 2 Specific items regarding character frequency are attached. Figure 4 Indicates appendix Figure 2 The specific items of keyword frequency are used to extract a total of 128 features from the URL string as statistical features. The pseudocode is as follows: Algorithm 1:

[0081] Algorithm 1: Extracting Statistical Features from URLs

[0082]

[0083] ;

[0084] Step 2.2, Character Feature Extraction: In the character feature extraction submodule, the sample data obtained in Step 1.4 is used as input. A script is written using the Python programming language to convert each character of the URL into ASCII encoding and use it as character features. The character feature dimension of each URL is set to 64. For URLs with fewer than 64 characters, a zero-padding strategy is used; for URLs with more than 64 characters, truncation is performed. After encoding, the feature vector is normalized to obtain the final character feature representation. The pseudocode is as follows: Algorithm 2:

[0085] Algorithm 2: Extracting Character Features from URLs

[0086] ;

[0087] Step 2.3, Lexical Feature Extraction: Using the sample data obtained in Step 1.4 as input, a script is written in Python. First, the URL is segmented using regular expressions to extract the payload. Second, a dictionary of payloads is constructed using the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm. The TF-IDF value of each word is calculated using this algorithm, and the top 500 relatively important words are selected as the dictionary. Finally, the word embedding model is trained using the constructed dictionary to generate lexical features. This invention selects the Skip-gram architecture of Word2Vec for training to obtain the word vector of each word. To represent the lexical features of the entire URL, the mean of all word vectors in each URL is calculated, resulting in a 64-dimensional vector as the lexical feature representation of the URL. The pseudocode is as follows: Algorithm 3:

[0088] Algorithm 3 Extracts Lexical Features from URLs

[0089] ;

[0090] Step 2.4, Feature Fusion: In the feature fusion submodule, the three character features obtained in steps 2.1, 2.2 and 2.3 are combined to obtain a total of 256-dimensional sample feature representation.

[0091] In the above embodiments, the specific steps for constructing and training the detection model in step 3 are as follows:

[0092] Step 3.1, Building and training the extremely random tree classifier module: The extremely random tree classifier module is built using the Scikit-learn library. The labeled part of the sample features obtained in Step 2.4 is selected as the input of the extremely random tree classifier. The parameters of the extremely random tree are set, and then training is performed.

[0093] Step 3.2, Labeling: Use the trained Extremely Random Tree classifier obtained in Step 3.1 to predict the unlabeled sample features in the sample features obtained in Step 2.4, and use the prediction results as pseudo-labels;

[0094] Step 3.3, Generator Module Construction and Training: Based on the PyTorch framework, several linear and convolutional layers are constructed as the generator. A 62-dimensional random distribution is selected as the input, and the output is a 256-dimensional feature. The output features are used to obtain pseudo-labels through the trained extremely random tree classifier obtained in Step 3.1. One of the training objectives of the generator is to generate near-realistic samples to challenge the discriminator's discrimination ability. This is achieved by minimizing the cross-entropy loss between the discriminator's output label and label 1 (representing a real sample). Simultaneously, samples that help the classifier make accurate decisions are generated. The generated samples are optimized by minimizing the cross-entropy loss between the pseudo-label and the classifier's predicted label, while the classifier attempts to reduce the impact of these generated samples on its decision-making process. Through the adversarial interaction between the generator and the classifier, the robustness of the classifier to incorrect labels is enhanced.

[0095] Step 3.4, Discriminator Module Construction and Training: Based on the PyTorch framework, several linear and convolutional layers are constructed as discriminators. The input includes real data (with and without labels), generated data, and their labels, which come from steps 2.4, 3.2, and 3.3, respectively. The output is the data labels. The goal is to distinguish whether the input data is a real sample or a fake sample generated by the generator. The training task of the discriminator is to distinguish whether the input data is a real sample or a fake sample generated by the generator. This is achieved by minimizing the cross-entropy loss between real samples (including labeled and unlabeled) and label 1 (representing real samples) and minimizing the cross-entropy loss between generated samples and label 0 (representing generated samples). This is similar to a binary classification task. Through continuous adversarial interaction with the generator, the discriminator continuously improves its discrimination ability, thereby improving the overall model's decision performance and generalization ability.

[0096] Step 3.5, Classifier Module Construction and Training: Several linear layers are constructed based on the PyTorch framework, and a hedging algorithm is added to its backpropagation as a classifier. The input is the same as the discriminator module in Step 3.4, and the output is the data label, aiming to distinguish whether the input data is malicious or benign. For labeled samples, the classifier is trained by minimizing the cross-entropy loss between the true label and the predicted label; for unlabeled samples, the classifier is trained by minimizing the cross-entropy loss between the pseudo-label and the predicted label; for generated samples, the classifier is trained by maximizing the cross-entropy between the pseudo-label and the predicted label.

[0097] In this embodiment, the specific steps for malicious URL detection in step 5 are as follows:

[0098] Step 5.1, Data Preprocessing: For the URL to be tested, clean, decoded, and standardized samples are obtained through steps 1.2, 1.3, and 1.4.

[0099] Step 5.2, Feature Engineering: The samples obtained in Step 5.1 are processed through Steps 2.1 to 2.4 to obtain a 256-dimensional feature sequence;

[0100] Step 5.3, Anomaly Detection: Input the feature sequence obtained in Step 5.2 into the classifier trained in Step 3.5 for prediction, and obtain label 1 indicating a prediction of malice and label 0 indicating a prediction of benignity.

[0101] In this embodiment, the specific steps for online updating of the detection model in step 6 are as follows:

[0102] Step 6.1, Manual Feedback: After obtaining the prediction results of the URL to be tested in Step 5.3, the prediction results are manually reviewed, and the true tags of the URL to be tested are fed back based on the review results.

[0103] Step 6.2, Online Update: Input the features of the test sample obtained in Step 5.2 and the true label obtained in Step 6.1 into the classifier and perform training and update according to Step 3.5.

[0104] In step 4, the performance evaluation indicators are calculated as follows:

[0105] Accuracy Macro Average ,

[0106] In the formula: ;

[0107] Macro average recall ,

[0108] In the formula: ;

[0109] F1 Score Macro Average ,

[0110] In the formula: ;

[0111] Accuracy: ;

[0112] Wherein, TP indicates that a malicious URL is correctly identified as malicious, TN indicates that a legitimate URL is correctly identified as legitimate, FP indicates that a legitimate URL is incorrectly identified as malicious, FN indicates that a malicious URL is incorrectly identified as legitimate, and n indicates the number of categories.

[0113] As attached Figure 5As shown, the classifier is a multilayer perceptron (MLP) architecture consisting of five linear layers and one output layer. A separate linear layer is introduced after each linear layer as the output layer network, and its output is weighted to serve as the final prediction. This fully utilizes the information from each hidden layer. Furthermore, the classifier employs a hedging algorithm to learn the weighting parameters. In the initial iteration, weights are evenly distributed among each output layer. In subsequent iterations, the loss is calculated based on the classifier's predictions and the true labels, and the weights are updated accordingly. This allows the classifier module to automatically assign appropriate weights to each hidden layer, automatically adapting to different network depths and ensuring compatibility with online scenarios.

[0114] To verify the effectiveness of the detection method proposed in this invention, a large number of experiments were conducted for its evaluation. The datasets used are shown in Table 1 below, the dataset partitioning method is shown in Table 2 below, the experimental environment is shown in Table 3 below, and the EMO-GAN model parameters are shown in Table 4 below.

[0115] Table 1 Dataset

[0116] ;

[0117] Table 2 Dataset Splitting

[0118] ;

[0119] Table 3 Experimental Environment

[0120] ;

[0121] Table 4 EMO-GAN Model Parameters

[0122] .

[0123] The comparative experimental evaluation results of the feature engineering mentioned in this invention and other feature engineering methods in malicious URL detection tasks are attached. Figure 6 , 7 As shown in Tables 1 and 2, the experiments used the Malicious URL public dataset from Table 1, and nine sub-datasets obtained from Table 2. Various machine learning algorithms were selected, including basic machine learning algorithms (such as Support Vector Machine (SVM) and Logistic Regression (LR), ensemble learning algorithms (such as Extremely Random Trees (ET),) and deep learning algorithms (such as Convolutional Neural Networks (CNN) and Multilayer Perceptrons (MLP)). These were combined with four feature engineering techniques: statistical features, character features, lexical features, and hybrid features. Corresponding models were constructed and trained, and performance metrics were statistically analyzed.

[0124] The comparative experimental evaluation results of the EMO-GAN model mentioned in this invention with other detection models are attached. Figure 9As shown. The experiment aims to evaluate the model's performance in imbalanced, label-scarce data scenarios. To more closely resemble real-world scenarios, a URL dataset with a positive-to-negative sample ratio of 1:99 and a label percentage of only 5% was used, and hybrid features were selected as the feature extraction method. During the experiment, seven different models were tested, including two semi-supervised models (Co-training and Self-training), two generative adversarial network models (SGAN and MarginGAN), one ensemble learning model (RandomForest), one deep learning model (CNN), and the EMO-GAN model proposed in this invention.

[0125] The experimental results evaluating the online learning ability of the EMO-GAN model mentioned in this invention are attached. Figure 10 As shown in the figure. The experiment aims to evaluate the ability of the EMO-GAN model to better adapt to new data environments by fine-tuning its parameters after receiving new data. In the experiment, a pre-trained EMO-GAN classifier was used as the base model, and different numbers of new samples were introduced for fine-tuning. These new samples came from a dataset with a positive-to-negative sample ratio of 1:1. To comprehensively evaluate the online learning performance of EMO-GAN, accuracy, precision, recall, and F1 score were used as metrics. These metrics were generated cumulatively; that is, each time a new sample arrived, the classifier first made a prediction, then added the real label for training, and the predicted and real labels were accumulated to calculate the metrics.

[0126] The experimental results for evaluating the generalization ability of the EMO-GAN model mentioned in this invention are attached. Figure 11 As shown in Table 1, the experiments were comprehensively evaluated on four different public datasets. To ensure the fairness and validity of the evaluation, each dataset was randomly sampled according to a positive-to-negative sample ratio of 1:99 and a label percentage of 5%, with 100,000 samples drawn from each dataset. Six models were compared, covering supervised learning (ET), semi-supervised learning (self-trained ET, self-trained MLP), and deep learning (MLP, SGAN, MarginGAN).

[0127] The comparative experimental evaluation results of the EMO-GAN model mentioned in this invention with other mainstream detection methods are attached. Figure 12 As shown in the figure. The experiment aims to verify the advancement of the detection model of this invention, demonstrating that the method proposed in this invention can provide effective solutions to the problems of data acquisition, feature acquisition, and concept drift handling faced in malicious URL detection tasks. The experimental dataset is the Malicious URL public dataset in Table 1.

[0128] In summary, this invention addresses three major challenges in current malicious URL detection: difficulty in data acquisition, difficulty in feature representation, and concept drift in the model. It proposes an EMO-GAN-based malicious URL detection method. This framework integrates semi-supervised learning and Generative Adversarial Networks (GANs) to solve the data acquisition difficulties caused by imbalance and label scarcity. By introducing Extremely Random Trees (ET) to achieve semi-supervised learning, it effectively utilizes unlabeled data and reduces dependence on labeled data. Simultaneously, by leveraging the characteristics of MarginGAN, it generates samples and reduces the impact of erroneous pseudo-labels, further improving model performance. Furthermore, this invention proposes a feature representation scheme that integrates statistical features, character features, and lexical features, effectively improving feature information and providing a solution for efficient feature representation of URLs. Finally, this invention also introduces online learning technology to address the concept drift problem in malicious URL detection. Online learning allows the model to receive new data in real time and update itself, thereby adapting to a rapidly changing data environment.

[0129] Experimental results show that the EMO-GAN proposed in this invention achieves high results in terms of accuracy and F1 score, and has significant advantages over classical methods.

[0130] The foregoing has shown and described the main features and advantages of the present invention. It will be apparent to those skilled in the art that the specific embodiments of the present invention are not limited to the details of the exemplary embodiments described above. Furthermore, without departing from the spirit or essential characteristics of the present invention, the inventive concept and design ideas of the present invention can be implemented in other specific forms, and these should be equivalently included within the protection scope disclosed in the technical solutions of the present invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of the present invention is defined by the appended claims rather than the foregoing description, and thus all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included within the present invention.

[0131] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A malicious URL detection method based on EMO-GAN, characterized in that: It includes the following steps: Step 1: Data preprocessing, the specific steps are as follows: Step 1.1, Data Acquisition: Use a web crawler to obtain URL data from the webpage, which will serve as the input for Step 1.2; Step 1.2, Data Cleaning: Use the Python programming language to write a script to delete URL strings that belong to both malicious and benign categories. The resulting data will be used as input for Step 1.

3. Step 1.3: Data Decoding: Use the Python programming language to write a script to decode the encoded special characters and hexadecimal numbers in the URL data into their original form. The resulting data will be used as the input for Step 1.

4. Step 1.4: Data Standardization: Use the Python programming language to write a script to uniformly replace non-ASCII characters in the URL data with a fixed string str, so as to obtain clean and standard sample data; Step 2, Feature Engineering, the specific steps are as follows: Step 2.1, Statistical Feature Extraction: In the statistical feature extraction submodule, the sample data obtained in Step 1.4 is used as input. A script is written using the Python programming language to calculate a total of 128 statistical indicators for the URL as statistical features. The statistical indicators include domain name length, number of decimal points in the domain name, whether the domain name contains -, top-level domain name length, path length, number of / , whether the protocol is HTTPS, number of &, total URL length, whether it contains redirects, whether it contains short links, character frequency, and keyword frequency. Step 2.2, Character Feature Extraction: In the character feature extraction submodule, the sample data obtained in step 1.4 is used as input. A script is written using the Python programming language to convert each character of the URL into ASCII encoding and use it as a character feature. The character feature dimension of each URL is set to 64. For URLs with fewer than 64 characters, a zero-padding strategy is adopted; for URLs with more than 64 characters, truncation is performed. Step 2.3, Lexical Feature Extraction: In the lexical feature extraction submodule, the sample data obtained in step 1.4 is used as input. A script is written using the Python programming language. First, word segmentation is performed based on regular expressions. Secondly, the word frequency-inverse document frequency of each word is calculated, and the top 500 words are selected as the filtering criteria to form a dictionary. Then, the Word2Vec model is trained using the dictionary to generate lexical features. Finally, the mean of all word vectors in the URL is calculated to obtain a 64-dimensional vector as the lexical feature representation of the URL. Step 2.4, Feature Fusion: In the feature fusion submodule, the three character features obtained in Steps 2.1, 2.2 and 2.3 are combined to obtain a total of 256-dimensional sample feature representation; Step 3: Construction and training of the detection model, the specific steps are as follows: Step 3.1, Building and training the extremely random tree classifier module: The extremely random tree classifier module is built using the Scikit-learn library. The labeled part of the sample features obtained in Step 2.4 is selected as the input of the extremely random tree classifier. The parameters of the extremely random tree are set, and then training is performed. Step 3.2, Labeling: Use the trained Extremely Random Tree classifier obtained in Step 3.1 to predict the unlabeled sample features in the sample features obtained in Step 2.4, and use the prediction results as pseudo-labels; Step 3.3, Building and training the generator module: Based on the PyTorch framework, several linear and convolutional layers are built as the generator. A 62-dimensional random distribution is selected as the input, and the output is a 256-dimensional feature. The output feature is used to obtain pseudo-labels through the trained extremely random tree classifier obtained in Step 3.

1. Adversarial training with the classifier module is achieved by minimizing the cross-entropy loss between the pseudo-label and the predicted label of the classifier module. Adversarial training with the discriminator module is achieved by minimizing the cross-entropy loss between the predicted label and the real label of the discriminator module. Step 3.4, construction and training of the discriminator module: Based on the PyTorch framework, several linear and convolutional layers are constructed as discriminators. The input includes the real samples obtained in step 2.4, the generated samples obtained in step 3.3, and the labels of the two types of data obtained in step 3.

2. Adversarial training with the generator module is achieved by minimizing the cross-entropy loss between the real samples and the real labels and minimizing the cross-entropy loss between the generated samples and the generated labels. Step 3.5, Building and training the classifier module: Based on the PyTorch framework, several linear layers are built as classifiers. The inputs include the real samples obtained in step 2.4, the generated samples obtained in step 3.3, and the labels of the two types of data obtained in step 3.

2. For labeled samples, the classifier module is trained by minimizing the cross-entropy loss between the real labels and the predicted labels. For unlabeled samples, the classifier module is trained by minimizing the cross-entropy loss between the pseudo-label and the predicted label; For the generated samples, the classifier module is trained by maximizing the cross-entropy between the pseudo-label and the predicted label; Step 4: Model performance evaluation: The macro average of precision, recall, and F1 score is used as the evaluation metric for the model. If the precision is greater than 99% and the F1 score is greater than 80%, proceed to step 5; otherwise, proceed to step 3.2 to continue model training. Step 5, Detection of Malicious URLs: Collect URL data from real web pages as test data, process the test data according to Step 1 and Step 2 to obtain 256-dimensional features, and then input them into the pre-trained classifier module for classification prediction to obtain the corresponding labels and complete the detection. Step 6: Online update of the detection model: The predicted labels obtained in Step 5 are verified by human feedback to obtain the true labels of the test data. The 256-dimensional features of the test data obtained in Step 5 and the true labels of human feedback are input into the classifier module for online learning and updating.

2. The malicious URL detection method based on EMO-GAN according to claim 1, characterized in that: In step 3.5, the classifier is a multilayer perceptron structure containing five linear layers and one output layer. A separate linear layer is introduced after each linear layer as the output layer network, and the output results are weighted and used as the final prediction result. The classifier uses a hedging algorithm to learn the weighting parameters. In the initial iteration, the weights are evenly distributed to each output layer. In subsequent iterations, the loss is calculated based on the classifier's prediction and the true label, and the weights are updated accordingly.

3. A malicious URL detection method based on EMO-GAN according to any one of claims 1 to 2, characterized in that: In step 5, the specific steps for malicious URL detection are as follows: Step 5.1, Data Preprocessing: The URL data to be tested is cleaned through Step 1.2, Decoding, and Standardization to obtain clean and standard sample data. Step 5.2, Feature Engineering: The sample data obtained in Step 5.1 is processed through Steps 2.1 to 2.4 to obtain 256-dimensional features; Step 5.3, Anomaly Detection: Input the 256-dimensional features obtained in Step 5.2 into the classifier module trained in Step 3.5 for prediction, and obtain label 1 indicating a prediction of malice and label 0 indicating a prediction of benignity.

4. The malicious URL detection method based on EMO-GAN according to claim 3, characterized in that: In step 6, the specific steps for online updating of the detection model are as follows: Step 6.1, Manual Feedback: After obtaining the prediction results of the URL to be tested in Step 5.3, the prediction results are manually reviewed, and the true tags of the URL data to be tested are fed back based on the review results. Step 6.2, Online Update: Input the 256-dimensional features obtained in Step 5.2 and the true labels obtained in Step 6.1 into the classifier module and perform training and update according to Step 3.

5.

5. A malicious URL detection method based on EMO-GAN according to any one of claims 1 to 2 or 4, characterized in that: In step 4, the performance evaluation indicators are calculated as follows: Accuracy Macro Average , In the formula: ; Macro average recall , In the formula: ; F1 Score Macro Average , In the formula: ; Accuracy: ; Wherein, TP indicates that a malicious URL is correctly identified as malicious, TN indicates that a legitimate URL is correctly identified as legitimate, FP indicates that a legitimate URL is incorrectly identified as malicious, FN indicates that a malicious URL is incorrectly identified as legitimate, and n indicates the number of categories.

Images