A phishing website detection method and system based on multimodal fusion

Abstract

This invention belongs to the field of website security technology and discloses a phishing website detection method based on multimodal fusion. This invention proposes a risk identification method based on a multimodal deep learning model, which can better adapt to the diversity and imbalance of risk prevention data. This method utilizes the BERT model to extract text features and the ViT model to extract image features, concatenating the two and inputting them into a logistic classifier for risk identification. By fusing text and image information, this method improves the accuracy of identifying complex risk patterns. It alleviates the data imbalance problem: through multimodal feature fusion, this technology can extract risk features from data of different modalities, mitigating the model bias problem caused by the imbalance of positive and negative samples. The combination of multimodal features makes the model perform better in learning minority class samples, helping to improve the ability to identify scarce and abnormal samples.

Description

Technical Field

[0001] This invention belongs to the field of website security technology, and in particular relates to a phishing website detection method and system based on multimodal fusion. Background Technology

[0002] In today's society, intelligent technologies play a crucial role in various risk prevention and control efforts. By employing deep learning and multimodal data processing techniques, intelligent prevention and control methods can more comprehensively and accurately identify and predict potential risks, thereby providing effective support for decision-making and ensuring social safety and stability. The application of multimodal feature extraction and fusion technologies makes risk identification models more adaptable to complex and ever-changing risk environments, enabling them to quickly respond to the challenges of emerging risks.

[0003] The challenges in risk control primarily lie in the diversity and rapid changes of risk patterns, the massive volume of data, and the high demands for real-time response. Furthermore, risk data often suffers from severe imbalances, such as a significant difference in the proportion of normal behavior versus abnormal risky behavior. This imbalance can cause models to tend to classify data into common categories, thus affecting identification accuracy. This bias may lead models to overlook the existence of risky behaviors, reducing their generalization ability and impacting overall performance and stability.

[0004] Existing phishing website detection technologies are mainly divided into three categories: blacklist-based detection, heuristic-based detection, and machine learning-based detection. Blacklist-based detection methods are simple to implement and fast to query. For known phishing websites, they can achieve high accuracy and reduce the possibility of false positives and false negatives. However, over 93% of phishing web pages are not in mainstream blacklists, resulting in high maintenance costs and requiring significant human resources. Furthermore, this method is lagging, as newly emerging phishing websites cannot be detected immediately. Heuristic-based methods use predefined rules to analyze features and behavioral patterns extracted from web pages to identify potential phishing websites and determine their legitimacy. Compared to blacklists, which require frequent updates, heuristic detection only needs periodic adjustments and optimizations of features and rules, resulting in relatively low maintenance costs. However, heuristic rules are often not complex, and phishing attackers can circumvent them by forging relevant features. Machine learning-based methods use features extracted from websites to train classifiers using different algorithms based on samples. These classifiers are then used to predict and classify websites for phishing purposes. The process mainly consists of five parts: data collection, feature extraction, model training, model testing, and prediction.

[0005] With the rapid development of information technology and the diversification of data types, the effectiveness of single data sources in specific application scenarios has gradually been limited. Multimodal data fusion technology has emerged to address this, comprehensively utilizing different types of data (such as text, images, and audio) to capture key information in complex phenomena. In the field of risk prevention and control, the application of multimodal data fusion technology is particularly important. For example, by integrating textual and visual information, researchers can construct more accurate models. This technological background provides crucial support for this invention, making it possible to utilize multimodal features for risk detection.

[0006] The application of deep learning technology in risk prevention and control is receiving increasing attention, primarily due to its outstanding performance in large-scale data analysis and complex pattern recognition. With the diversification and increasing sophistication of cyberattacks, traditional rule-based methods are struggling to address new types of threats. Deep learning models can automatically extract meaningful features from massive amounts of data, identify potential risks, and perform particularly well when processing unstructured data (such as text and images). The introduction of this technology not only improves the accuracy and efficiency of risk detection but also enhances the model's adaptability to changing and complex environments, providing a solid foundation for more intelligent risk prevention and control solutions.

[0007] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0008] With the widespread use of the internet, phishing websites have become a major challenge in the field of cybersecurity. These websites typically disguise themselves as legitimate sites to trick users into revealing sensitive personal information, such as account passwords and credit card details. Traditional phishing website detection methods mainly rely on single feature extraction techniques, such as textual features of the URL or image features of the website. These methods have the following problems:

[0009] Limitations of Single Features: Traditional phishing website detection methods typically rely on textual or image analysis of URLs, but cannot fully uncover the correlation between the two. Relying on a single data source can lead to false negatives or missed detections, especially when facing hybrid phishing websites (i.e., websites that combine URL spoofing and image forgery techniques).

[0010] Lack of deep semantic understanding: Traditional keyword or rule-based phishing website detection methods cannot understand the potential semantic information in a page. Phishing websites often use clever designs to mislead users, so more advanced analysis techniques are needed to process website content.

[0011] Slow detection speed: Traditional methods often rely on manually designed rules or simple machine learning models, which cannot process large amounts of web page content quickly in real time, resulting in performance bottlenecks when processing real-time website traffic.

[0012] Low detection accuracy: Because phishing websites often evade detection through disguise and subtle adjustments, existing methods are less accurate when faced with sophisticated phishing tactics, especially in detecting dynamic changes on the page or deceptive elements in images. Summary of the Invention

[0013] To address the problems existing in the prior art, this invention provides a phishing website detection method based on multimodal fusion.

[0014] This invention is implemented as follows: A phishing website detection method based on multimodal fusion includes:

[0015] Step 1, Data Processing:

[0016] In the process of building a phishing website detection model, we first collect data containing historical URLs and their corresponding website screenshots; we then extract image information from text URLs and use web crawling technology to automatically extract website screenshots, ensuring that each URL has corresponding image data. At the same time, we assign a hash value to each URL as an index for website screenshots.

[0017] During the data processing phase, meticulous processing is carried out based on the data characteristics, including imputing missing values ​​and filtering outliers; at the same time, a label dimension will be added to the dataset, marking phishing websites as 1 and legitimate websites as 0.

[0018] Finally, the entire dataset is distributed in a 4:1 ratio to ensure that 80% of the samples are used for the training set and 20% for the test set, and to ensure that the proportion of phishing websites and legitimate websites in the training set and test set remains consistent.

[0019] Step 2, Extracting Multimodal Features: In this stage, multimodal features are extracted, and the extraction strategies for text and image features are explored in depth;

[0020] Step 3, Multimodal Fusion: After extracting multimodal features, the text feature vector and the image feature vector are concatenated together;

[0021] Step 4, Iterative optimization: In the final predicted classification, the loss function is log loss;

[0022] Step 5: Classify phishing websites.

[0023] Furthermore, the extraction of multimodal features:

[0024] 1) Image Features: In image feature extraction, the ViT model is used. ViT employs a self-attention mechanism and a multi-layer structure. Before inputting the image into the model, a resize operation is performed to unify the image size to 224×224. For the input image I, it is divided into several patches of size 16×16, forming a patch sequence P, i.e.:

[0025] P={p1,p2,…,pk},k=(224×224) / (16×16)=196;

[0026] The architecture of the image feature extraction model is as follows:

[0027] T image =F image (D image ),

[0028] T image For image features, T image ∈R n×m ;

[0029] 2) Text features.

[0030] Furthermore, regarding text features: In text feature extraction, the BERT model is used to process the URL.

[0031] The input text sequence is tokenized and converted into TokenID, with special markers added:

[0032] [CLS]: A vector added at the beginning of the text to represent the entire sentence;

[0033] [SEP]: A marker added to the end of text to separate different sentences or text.

[0034] After word segmentation, the input sequence “www.baidu.com” becomes “[CLS]wwwbaidu com[SEP]”; the BERT text feature extraction process can be represented by the formula:

[0035] T url =F text (D text ), T url For text features, T url ∈R n×d .

[0036] Furthermore, the multimodal fusion:

[0037] After extracting multimodal features, the text feature vector is concatenated with the image feature vector:

[0038] T f =cat(Turl T image ), T f For the final feature vector, T f ∈R n×(d+m) ;

[0039] The multimodal fusion network architecture will concatenate the resulting feature vector T. f As a common input to machine learning classifiers, the logistic regression classifier maps the input feature X to a real value through a linear function, and then transforms this real value into a probability value between 0 and 1 through a non-linear function. The formula for logistic regression is as follows:

[0040]

[0041] Where z = W T X+b represents the inner product of the feature vector X and the weight vector W plus the bias term b, which is expanded as follows:

[0042] z = b + β1X1 + β2X2 + ... + β d X d =X T β,

[0043] Where z is the result of a linear combination, β1,β2,...,β d β represents the weights of the features, where X is the feature vector and β is the parameter vector.

[0044] Weight vector W: A vector representing the importance of each feature in the decision; for a feature vector X (of size d), the dimension of the weight W is usually d×1;

[0045] Bias term b: A scalar that represents the threshold adjustment of the classifier and is typically used to control the position of the decision boundary.

[0046] Furthermore, the iterative optimization:

[0047] In the final classification prediction, the loss function is log loss (LogLoss), and its formula is:

[0048]

[0049] Among them, y i It's a real label. These are the labels predicted by the model;

[0050] Due to the large dataset and high feature dimensionality, the LBFGS optimizer was chosen because of its high efficiency, strong adaptability, and good convergence. LBFGS achieves efficient unconstrained optimization by approximating the inverse of the Hessian matrix, aiming to minimize a differentiable objective function f(x), where x is a parameter vector. In each iteration, LBFGS utilizes the current point x... k gradient To guide optimization; the parameter update formula for LBFGS is as follows:

[0051]

[0052] Among them, H k It is an approximation of the Hessian matrix, α k It's the step length.

[0053] Furthermore, the phishing websites are categorized as follows:

[0054] First, capture the corresponding URL text information and webpage screenshots, and then pass these data into the preprocessing module to ensure that the data format is consistent with that used during model training.

[0055] Secondly, after extracting text features and image features separately through the multimodal fusion model, the two are concatenated into a comprehensive feature vector;

[0056] Next, the fused feature vectors will be input into a trained logistic regression classifier for prediction; the classifier will determine whether the current website is a phishing website based on the learned patterns; the output will provide a probability value for each visited website, indicating the degree of risk of it being a phishing website.

[0057] Another objective of this invention is to provide a phishing website detection system based on multimodal fusion, comprising:

[0058] The data processing module is used to collect data containing historical URLs and their corresponding website screenshots during the construction of the phishing website detection model; obtain image information from text URLs; automatically extract website screenshots using web crawling technology to ensure that each URL has corresponding image data; and assign a hash value to each URL as an index for website screenshots.

[0059] The extraction module is used to extract multimodal features: In this stage, multimodal features are extracted, and the extraction strategies for text features and image features are explored in depth;

[0060] The multimodal fusion module is used to extract multimodal features and then concatenate the text feature vector with the image feature vector.

[0061] The iterative optimization module is used to apply a log loss function in the final classification prediction.

[0062] The categorization module is used to classify phishing websites.

[0063] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the phishing website detection method based on multimodal fusion.

[0064] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the phishing website detection method based on multimodal fusion.

[0065] Another objective of this invention is to provide an information data processing terminal for implementing the phishing website detection system based on multimodal fusion.

[0066] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0067] First, this invention proposes a risk identification method based on a multimodal deep learning model, which can better adapt to the diversity and imbalance of risk prevention and control data. This method utilizes the BERT model to extract text features and the ViT model to extract image features, concatenating the two and inputting the result into a logistic classifier for risk identification. By fusing text and image information, this method improves the accuracy of identifying complex risk patterns.

[0068] Mitigating data imbalance: By fusing multimodal features, this technique can extract risk features from data of different modalities, mitigating model bias caused by an imbalance in the ratio of positive to negative samples. The combination of multimodal features makes the model perform better in learning minority class samples, helping to improve the ability to identify scarce and outlier samples.

[0069] Improved recognition accuracy: This technology fully utilizes information from different modalities by fusing text features extracted by BERT and image features extracted by ViT, thereby enhancing the model's ability to recognize complex risk patterns. Compared to traditional methods with single feature input, this multimodal feature fusion exhibits higher accuracy when dealing with diverse and complex risks, thus significantly optimizing prediction performance.

[0070] Second, the expected benefits and commercial value of the technical solution of this invention after transformation are as follows:

[0071] Enhancing network security protection capabilities: The phishing website detection method of this invention can monitor and warn users in real time whether the URLs they visit are phishing websites, effectively reducing online fraud and information leakage incidents, and improving the overall level of network security protection.

[0072] Enhanced user experience: Through a plugin, users can obtain phishing website detection services without complicated operations, simplifying the process of identifying phishing websites.

[0073] Reduce business losses: By deploying the phishing website detection plugin of this invention, businesses can reduce economic losses and reputational damage caused by employees visiting phishing websites, thereby reducing operational risks.

[0074] Creating new revenue streams: The phishing website detection plugin of this invention can be provided to users as a value-added service, creating new revenue sources for businesses. Simultaneously, through advertising, data analysis, and other models, it can also diversify commercial value.

[0075] Third, the technical problems of the prior art solved and the significant technical progress achieved by the technical solution of the present invention in industrial applications:

[0076] The phishing website detection method proposed in this invention, based on multimodal fusion, innovatively combines image and text features. Through deep learning and multimodal data fusion, it effectively addresses the shortcomings of traditional methods and demonstrates significant technological advancements.

[0077] Multimodal Feature Fusion: This invention detects phishing websites by combining information from both image and text sources. The BERT model extracts textual features from the URL, while the ViT (Visual Transformer) model extracts image features from website screenshots. This multimodal feature extraction comprehensively captures the different forms of phishing websites. Furthermore, by concatenating the text and image feature vectors and inputting them into a machine learning classifier (such as logistic regression), the accuracy of detection is further improved.

[0078] Application of deep learning models: In the image feature extraction process, this invention uses the Visual Transformer (ViT) model, a deep learning method that processes images through self-attention mechanisms and multi-layer structures. This method can effectively extract deep features from images with fewer training samples, improving the ability to recognize complex image content (such as disguised logos, deceptive page layouts, etc.).

[0079] Applications of the BERT model in text feature extraction: Using the BERT model to process URL text enables a better understanding and extraction of semantic information from URLs, identifying potentially deceptive URLs. By considering contextual information, the BERT model can identify more complex and covert phishing techniques than traditional methods, such as attacks through URL spoofing.

[0080] Improved detection accuracy and robustness: The method of this invention can simultaneously identify deceptive behavior in URLs and images, with high detection accuracy. It can effectively identify hybrid phishing websites, especially those that use sophisticated camouflage techniques.

[0081] Real-time detection and high efficiency: This invention combines multimodal fusion and deep learning models to process and analyze website content in real time, achieving higher processing efficiency and a lower false positive rate compared to traditional methods. The multimodal fusion network can handle more complex network environments and provides rapid response while maintaining high detection accuracy.

[0082] Adaptable to complex network environments: The multimodal fusion detection method of this invention has strong adaptability and can cope with different types of phishing website attacks. It can effectively detect attacks whether they are spoofed by URLs, website layouts, or image content.

[0083] In summary, this invention, through its innovative multimodal feature fusion method, significantly improves the accuracy, efficiency, and robustness of phishing website detection, providing enhanced security, especially against complex and cleverly designed phishing attacks. These technological advancements have significant application value in the current cybersecurity field, particularly in protecting users from phishing attacks and enhancing network security, offering more reliable protection for various network applications. Attached Figure Description

[0084] Figure 1 This is a flowchart of a phishing website detection method based on multimodal fusion provided in an embodiment of the present invention.

[0085] Figure 2 This is a structural block diagram of a phishing website detection system based on multimodal fusion provided in an embodiment of the present invention.

[0086] Figure 3 This is a diagram of the multimodal fusion model architecture provided in an embodiment of the present invention.

[0087] Figure 4 This is a flowchart of the browser plugin workflow provided in an embodiment of the present invention.

[0088] Figure 5 This is a schematic diagram showing the comparison of experimental results provided in the embodiments of the present invention.

[0089] Figure 6 This is a comparative bar chart of experimental results provided in the embodiments of the present invention. Detailed Implementation

[0090] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0091] Example 1: Detection of phishing websites in online banking applications

[0092] In the financial industry, especially in online banking and payment platforms, phishing website attacks are one of the most common security threats. Hackers impersonate legitimate bank websites to trick users into entering sensitive information, such as usernames, passwords, and bank card details. To effectively combat this threat, online banking systems employ the phishing website detection method based on multimodal fusion, as described in this invention.

[0093] Application process:

[0094] 1. Data Acquisition: The system uses web crawling technology to collect bank-related URLs and webpage screenshots. Through automated programs, the system obtains the URL information of visited websites and corresponding webpage screenshot data in real time.

[0095] 2. Feature Extraction: The BERT model is used to extract features from the URL text to identify potential malicious and forgery behaviors. Simultaneously, the system uses the ViT model to perform deep analysis on the screenshot images, identifying forgery elements (such as fake bank logos, phishing buttons, etc.) within the images.

[0096] 3. Multimodal fusion: Text features and image features are fused to form a unified feature vector, which is then input into the classification model. The system will use a logistic regression classifier to determine whether the URL belongs to a phishing website.

[0097] 4. Real-time monitoring and early warning: The system continuously monitors accessed URLs and, through real-time detection, ensures that users are promptly identified and warned whether a website is a phishing site before they access it. If a phishing website is detected, the system will automatically block access to ensure that users' sensitive information is not stolen.

[0098] Technical advantages:

[0099] By utilizing multimodal data from text and images, the detection results are more accurate and can identify complex phishing attack methods.

[0100] Highly efficient real-time processing capabilities ensure that online banking platforms can respond instantly, reducing potential security threats to users;

[0101] This improved the user experience and platform security.

[0102] Example 2: Phishing Website Protection for E-commerce Platforms

[0103] E-commerce platforms face numerous phishing attacks from websites that impersonate legitimate e-commerce sites, enticing users to engage in fraudulent transactions or leaking their personal payment information. To protect user privacy and financial security, e-commerce platforms have deployed phishing website detection systems based on multimodal fusion.

[0104] Application process:

[0105] 1. Data Collection: The system uses web crawler technology to collect URL information and relevant webpage screenshots of e-commerce platforms, ensuring that relevant data from each e-commerce website is collected and analyzed.

[0106] 2. Feature Extraction: The system analyzes the text content of the URLs, extracts text features, and uses the BERT model to identify potential risks associated with malicious URLs. Simultaneously, it extracts image features from webpage screenshots using the ViT model to identify forged page elements and layouts (such as disguised payment buttons and fake product descriptions).

[0107] 3. Feature Fusion and Classification: The extracted text and image features are concatenated and input into a logistic regression classifier to classify phishing websites. The system improves detection accuracy by training and optimizing the classification model.

[0108] 4. Real-time Feedback and User Protection: The system monitors the e-commerce website's access traffic in real time, quickly identifying and categorizing the websites visited by users. During user visits, the system automatically detects phishing websites; if a phishing website is detected, the system will immediately issue a warning to prevent users from entering sensitive information or engaging in fraudulent transactions.

[0109] Technical advantages:

[0110] The fusion of multimodal features improves the accuracy of phishing website identification and can effectively prevent new and covert phishing attacks.

[0111] The system responds in real time, providing timely protection and reducing risks when users access e-commerce platforms;

[0112] By combining BERT and ViT models, the system is able to cope with various types of phishing attacks, including text spoofing and image spoofing.

[0113] These two embodiments demonstrate the powerful capabilities of the method of the present invention in practical applications. It not only improves the detection accuracy of phishing websites, but also greatly enhances the platform's security protection capabilities, and safeguards users' privacy and financial security.

[0114] like Figure 1 As shown, the phishing website detection method based on multimodal fusion provided by this embodiment of the invention includes the following steps:

[0115] S101, Data Processing:

[0116] In the process of building a phishing website detection model, we first collect data containing historical URLs and their corresponding website screenshots; we then extract image information from text URLs and use web crawling technology to automatically extract website screenshots, ensuring that each URL has corresponding image data. At the same time, we assign a hash value to each URL as an index for website screenshots.

[0117] During the data processing phase, meticulous processing is carried out based on the data characteristics, including imputing missing values ​​and filtering outliers; at the same time, a label dimension will be added to the dataset, marking phishing websites as 1 and legitimate websites as 0.

[0118] Finally, the entire dataset is distributed in a 4:1 ratio to ensure that 80% of the samples are used for the training set and 20% for the test set, and to ensure that the proportion of phishing websites and legitimate websites in the training set and test set remains consistent.

[0119] S102, Extracting Multimodal Features: In this stage, multimodal features are extracted, and the extraction strategies for text features and image features are explored in depth;

[0120] S103, Multimodal Fusion: After extracting multimodal features, the text feature vector and the image feature vector are concatenated together;

[0121] S104, Iterative optimization: In the final predicted classification, the loss function is log loss;

[0122] S105, Phishing Website Category.

[0123] The multimodal feature extraction provided by the embodiments of the present invention:

[0124] 1) Image Features: In image feature extraction, the ViT model is used. ViT employs a self-attention mechanism and a multi-layer structure. Before inputting the image into the model, a resize operation is performed to unify the image size to 224×224. For the input image I, it is divided into several patches of size 16×16, forming a patch sequence P, i.e.:

[0125] P={p1,p2,…,pk},k=(224×224) / (16×16)=196;

[0126] The architecture of the image feature extraction model is as follows:

[0127] Timage =F image (D image ),

[0128] T image For image features, T image ∈R n×m ;

[0129] 2) Text features.

[0130] The text features provided in this embodiment of the invention: In text feature extraction, the BERT model is used to process the URL:

[0131] The input text sequence is tokenized and converted into TokenID, with special markers added:

[0132] [CLS]: A vector added at the beginning of the text to represent the entire sentence;

[0133] [SEP]: A marker added to the end of text to separate different sentences or text.

[0134] After word segmentation, the input sequence “www.baidu.com” becomes “[CLS]wwwbaidu com[SEP]”; the BERT text feature extraction process can be represented by the formula:

[0135] T url =F text (D text ), T url For text features, T url ∈R n×d .

[0136] Multimodal fusion provided by embodiments of the present invention:

[0137] After extracting multimodal features, the text feature vector is concatenated with the image feature vector:

[0138] T f =cat(T url T image ), T f For the final feature vector, T f ∈R n×(d+m) ;

[0139] The multimodal fusion network architecture will concatenate the resulting feature vector T. f As a common input to machine learning classifiers, the logistic regression classifier maps the input feature X to a real value through a linear function, and then transforms this real value into a probability value between 0 and 1 through a non-linear function. The formula for logistic regression is as follows:

[0140]

[0141] Where z = W T X+b represents the inner product of the feature vector X and the weight vector W plus the bias term b, which is expanded as follows:

[0142] z = b + β1X1 + β2X2 + ... + β d X d =X T β,

[0143] Where z is the result of a linear combination, β1,β2,...,β d β represents the weights of the features, where X is the feature vector and β is the parameter vector.

[0144] Weight vector W: A vector representing the importance of each feature in the decision; for a feature vector X (of size d), the dimension of the weight W is usually d×1;

[0145] Bias term b: A scalar that represents the threshold adjustment of the classifier and is typically used to control the position of the decision boundary.

[0146] Iterative optimization provided by embodiments of the present invention:

[0147] In the final classification prediction, the loss function is log loss (LogLoss), and its formula is:

[0148]

[0149] Among them, y i It's a real label. These are the labels predicted by the model;

[0150] Due to the large dataset and high feature dimensionality, the LBFGS optimizer was chosen because of its high efficiency, strong adaptability, and good convergence. LBFGS achieves efficient unconstrained optimization by approximating the inverse of the Hessian matrix, aiming to minimize a differentiable objective function f(x), where x is a parameter vector. In each iteration, LBFGS utilizes the gradient of the current point xk. To guide optimization; the parameter update formula for LBFGS is as follows:

[0151]

[0152] Among them, H k It is an approximation of the Hessian matrix, α k It's the step length.

[0153] The phishing website classification provided in this embodiment of the invention:

[0154] First, capture the corresponding URL text information and webpage screenshots, and then pass these data into the preprocessing module to ensure that the data format is consistent with that used during model training.

[0155] Secondly, after extracting text features and image features separately through the multimodal fusion model, the two are concatenated into a comprehensive feature vector;

[0156] Next, the fused feature vectors will be input into a trained logistic regression classifier for prediction; the classifier will determine whether the current website is a phishing website based on the learned patterns; the output will provide a probability value for each visited website, indicating the degree of risk of it being a phishing website.

[0157] like Figure 2 As shown, an embodiment of the present invention provides a phishing website detection system based on multimodal fusion, comprising:

[0158] The data processing module is used to collect data containing historical URLs and their corresponding website screenshots during the construction of the phishing website detection model; obtain image information from text URLs; automatically extract website screenshots using web crawling technology to ensure that each URL has corresponding image data; and assign a hash value to each URL as an index for website screenshots.

[0159] The extraction module is used to extract multimodal features: In this stage, multimodal features are extracted, and the extraction strategies for text features and image features are explored in depth;

[0160] The multimodal fusion module is used to extract multimodal features and then concatenate the text feature vector with the image feature vector.

[0161] The iterative optimization module is used to apply a log loss function in the final classification prediction.

[0162] The categorization module is used to classify phishing websites.

[0163] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the phishing website detection method based on multimodal fusion.

[0164] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the phishing website detection method based on multimodal fusion.

[0165] Another objective of this invention is to provide an information data processing terminal for implementing the phishing website detection system based on multimodal fusion.

[0166] Specific implementation of the present invention:

[0167] This technology focuses on the field of social risk prevention and control, particularly providing intelligent solutions for phishing website detection. Compared to traditional phishing website detection technologies, which often rely more on tedious manual identification and analysis, traditional methods require manual data sampling, cleaning, and labeling during the data preparation phase. In the feature extraction phase, simple statistical methods and extensive manual extraction are necessary. These methods are not only costly and inefficient but may also miss potential risk signals. Website homepage information, as a visual signal directly encountered by users, often contains more phishing elements; therefore, fully utilizing website screenshot information is crucial for phishing website detection. This technology utilizes multimodal feature information, which not only improves detection accuracy but also significantly reduces the need for manual intervention.

[0168] In phishing website detection tasks, the goal is to effectively identify malicious websites using the feature information of images and URLs. The following notation is defined: a text dataset Dtext = {u1, u2, ..., un} containing multiple URLs, and an image dataset D containing screenshots of the corresponding URLs. image ={I1,I2,…In}. Use a text feature extraction model to extract text features F from the URL. text :D text →R d Using image feature extraction model F image :D image →R m Extract image features, where d is the dimension of the text features and m is the dimension of the image features. Fusion model F r This is used to concatenate text features and image features for phishing website prediction. The joint optimization objective is shown in the following formula:

[0169] θ(F image ,F text ,F r ) = argminL(F r (F image (D image ),F text ,(D text )),y)

[0170] Where L represents the loss function, which reflects the difference between the model's predicted output and the true label.

[0171] First, necessary data preprocessing is performed, including data cleaning, noise removal, and handling of incomplete samples to ensure data quality and accuracy. Next, the BERT model is used to extract text features from URLs. Then, the ViT model is used to extract image features from website screenshots. Subsequently, the extracted text and image features are concatenated to form a comprehensive feature vector, which is then fed into the logistic classifier for phishing website detection. Finally, a comprehensive evaluation system is established to analyze and improve the overall performance of the model, aiming to address the issue of improving prediction accuracy. The specific steps are as follows:

[0172] Step 1: Data Processing: In building the phishing website detection model, the first step is to collect data containing historical URLs and their corresponding website screenshots. To extract image information from text URLs, web crawling is used to automatically extract website screenshots, ensuring that each URL has corresponding image data. A hash value is assigned to each URL as an index for the website screenshot. This data may come from various sources, such as network monitoring, user feedback, and security reports, and therefore may differ in storage format and attribute categories. During the data processing phase, meticulous processing is performed based on data characteristics, including imputation of missing values ​​and screening of outliers to improve data reliability. Simultaneously, a label dimension is added to the dataset, marking phishing websites as 1 and legitimate websites as 0. Finally, the entire dataset is allocated in a 4:1 ratio, ensuring that 80% of the samples are used for the training set and 20% for the test set, maintaining a consistent ratio of phishing websites to legitimate websites in both sets. This allocation strategy helps the model fully reflect real-world conditions during training and testing, thereby improving detection performance.

[0173] Step 2: Extracting Multimodal Features: In this stage, we focus on extracting multimodal features and explore in depth the extraction strategies for text features and image features.

[0174] 1. Image Features: The Visual Transformer (ViT) model is used for image feature extraction. ViT, through its self-attention mechanism and multi-layer structure, can efficiently identify key visual elements in images, such as color, layout, and pattern features. Before inputting the image into the model, a resize operation is performed to unify the image size to 224×224. For the input image I, it is divided into several 16×16 patches, forming a patch sequence P, i.e.:

[0175] P = {p1, p2, ..., pk}, k = (224 × 224) / (16 × 16) = 196. The architecture of the image feature extraction model is shown in the table below:

[0176] Parameter name parameter Patch size 16 number of transformer layers 12 Number of attention heads 12 Output feature dimension 768

[0177] T image =F image (D image ), T image For image features, T image ∈R n×m .

[0178] 2. Text Features: In text feature extraction, the BERT model is used to process the URL. BERT possesses powerful language processing capabilities, enabling it to deeply understand the complex semantics and contextual relationships within the text, thereby generating high-dimensional feature vectors. This process can be described in the following form:

[0179] Parameter name parameter Maximum sequence length 512 number of transformer layers 12 Number of attention heads 12 Output feature dimension 768 Total number of parameters 110M

[0180] The input text sequence is tokenized and converted into TokenID, with special markers added:

[0181] [CLS]: Added at the beginning of the text to represent the vector of the entire sentence.

[0182] [SEP]: A marker added to the end of text to separate different sentences or text.

[0183] After word segmentation, the input sequence “www.baidu.com” becomes “[CLS]wwwbaidu com[SEP]”. The BERT text feature extraction process can be represented by the formula:

[0184] T url =F text (D text ), T url For text features, T url ∈R n×d .

[0185] Step 3: Multimodal Fusion: After extracting multimodal features, the text feature vector and the image feature vector are concatenated together.

[0186] T f =cat(T url T image ), T f For the final feature vector, T f ∈R n×(d+m) Multimodal fusion network architecture, such as Figure 3 As shown. The concatenated feature vector T fThe logistic regression classifier uses this input as input in common machine learning classifiers, such as Logistic Regression. It maps the input feature X (including a bias term) to a real value using a linear function, and then transforms this real value into a probability value between 0 and 1 using a non-linear function (usually the sigmoid function). The formula for logistic regression is as follows:

[0187]

[0188] Where z = W T X+b represents the inner product of the feature vector X and the weight vector W plus the bias term b, which is expanded as follows:

[0189] z = b + β1X1 + β2X2 + ... + β d X d =X T β,

[0190] Where z is the result of linear combination, β1, β2, ..., βd are the feature weights, X is the feature vector, and β is the parameter vector.

[0191] Weight vector W: A vector representing the importance of each feature in the decision. For a feature vector X (of size d), the dimension of the weight W is typically d×1.

[0192] Bias term b: A scalar that represents the threshold adjustment of the classifier and is typically used to control the position of the decision boundary.

[0193] Step 4: Iterative Optimization: In the final classification prediction, the loss function is log loss (LogLoss), and its formula is:

[0194]

[0195] Among them, y i It's a real label. These are the labels predicted by the model.

[0196] Due to the large dataset and high feature dimensionality, the LBFGS optimizer was chosen because of its high efficiency, strong adaptability, and good convergence. LBFGS achieves efficient unconstrained optimization by approximating the inverse of the Hessian matrix, aiming to minimize a differentiable objective function f(x), where x is a parameter vector. In each iteration, LBFGS utilizes the current point x... k gradient This guides optimization. The parameter update formula for LBFGS is as follows:

[0197]

[0198] Among them, H k It is an approximation of the Hessian matrix, α k It's the step length.

[0199] Step 5: Phishing Website Classification: In this stage, a trained multimodal fusion network model is obtained. When performing the phishing website classification task, the corresponding URL text information and webpage screenshots are first captured and fed into the preprocessing module to ensure the data format is consistent with that used during model training. Next, the multimodal fusion model extracts text and image features separately, then concatenates them into a comprehensive feature vector. Finally, the fused feature vector is input into a trained logistic regression classifier for prediction. The classifier determines whether the current website is a phishing website based on the learned patterns. The output provides a probability value for each visited website, indicating the level of risk associated with it being a phishing website.

[0200] 4.1 Data Preparation

[0201] The Phishingwebsite phishing website detection dataset was collected from PhishTank, a well-known community-driven platform dedicated to collecting and sharing information about phishing websites. This dataset contains a large number of website URLs labeled as either phishing or legitimate, including URLs and website screenshots. This data provides rich training samples for the model, helping to identify and classify potential phishing websites. By utilizing this dataset, it is possible to deeply analyze the characteristics of phishing websites to improve network security. 0 represents a legitimate website, and 1 represents a phishing website; the original dataset was split into 80% training set and 20% test set.

[0202] 4.2 Indicator Design

[0203] In the field of phishing website detection research, fraud detection has become a significant research hotspot, with researchers proposing various models to identify fraudulent activities in transaction data. However, most studies often rely on only a single metric, such as accuracy or precision, when evaluating model performance. Given that phishing websites constitute a very small proportion of all websites, and datasets suffer from severe imbalance, simply using conventional evaluation metrics such as accuracy is insufficient to comprehensively assess model effectiveness. Therefore, choosing a more suitable evaluation metric is particularly important.

[0204] Especially in phishing website detection scenarios, it is crucial to minimize false negatives of phishing websites while ensuring high accuracy and further improving the accuracy of identifying legitimate websites. To address this issue of selecting evaluation metrics, this study proposes a strategy: while controlling the Type I error rate (FNR), minimize false negatives of phishing websites, and further reduce the Type II error rate (FPR) based on this. This strategy aims to ensure that the model effectively reduces false negatives while also minimizing false positives of legitimate websites.

[0205] By controlling the FNR (Fulfilled Noise Ratio), the model can more accurately identify phishing behavior, reducing the risk of false negatives. Simultaneously, reducing the FPR (Fulfilled Percentage Ratio) while controlling the FNR means the model can maintain high accuracy when identifying legitimate websites, reducing unnecessary false positives. Furthermore, simultaneously focusing on both FNR and FPR allows for a better balance between the model's performance in identifying phishing and legitimate websites, making the model more robust and providing a clear basis for subsequent tuning and improvement.

[0206] Table 4.1 Confusion Matrix

[0207]

[0208] The table shows the four types of prediction results of the confusion matrix. A correct prediction where the actual situation is risky and the predicted situation is also risky is denoted as TP; a Type I error where the actual situation is risky and the predicted situation is risk-free is denoted as FN; a Type II error where the actual situation is risk-free but the model misclassifies it as risky is denoted as FP; and a correct prediction where the actual situation is risk-free and the predicted situation is also risk-free is denoted as TN. The model is evaluated using five metrics: False Positive Rate (FPR) and False Negative Rate (FNR) as expressed in formulas (1) and (2), Precision as expressed in formula (3), Recall as expressed in formula (4), and F1 Score as expressed in formula (5). Lower FNR and FPR indicate a better model.

[0209] FNR=FN / (TP+FN)#(1)

[0210] FPR=FP / (FP+TN)#(2)

[0211]

[0212] 4.3 Results of Phishing Website Detection Experiment

[0213] Five classifiers—LogisticRegression, LinearSVC, RandomForestClassifier, DecisionTreeClassifier, and SGDClassifier—were used to classify and predict multimodal features. The experimental results are as follows:

[0214]

[0215] On the Phishingwebsite phishing website detection dataset, predictions were successfully made using various classifiers, achieving good results and demonstrating the effectiveness of the multimodal feature fusion method used in this paper.

[0216] 4.4 Phishing Website Detection Plugin

[0217] Based on this technology, a browser plugin can be created to detect phishing websites. The plugin usage process is as follows: Figure 4 As shown,

[0218] First, clicking the plugin button causes the front-end program to obtain the website's URL and a screenshot of the current page through the browser port, and then send them to the back-end program. The back-end program decodes the data and then loads the pre-trained BERT and ViT models to extract multimodal features. After obtaining the extracted features, it loads the logistic regression classifier that has been trained on the phishingwebsite dataset to predict phishing websites. If the prediction result is a phishing website, a warning is issued to the user on the front end; if it is a legitimate website, a prompt is given.

[0219] Evidence related to the technical effects obtained by the embodiments of the present invention.

[0220] Comparison of experimental results (Table)

[0221]

[0222] LogisticRegression and SGDClassifier performed best across all metrics, demonstrating high accuracy and a good balance. LinearSVC followed closely behind, although its performance was slightly inferior, it still maintained high accuracy and recall. Figure 6As shown, RandomForestClassifier and DecisionTreeClassifier performed worse, especially in Recall and F-Score, where their scores were significantly lower than the others, indicating their weaker ability to capture positive samples. Overall, LogisticRegression and SGDClassifier performed excellently on all three metrics, fully demonstrating the superiority of this method in phishing website identification tasks.

[0223] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0224] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A phishing website detection method based on multi-modal fusion, characterized in that, Comprising the following steps: Step 1, data processing: In the process of constructing the phishing website detection model, first collect data containing historical URLs and their corresponding website screenshots; obtain image information from the text URL, automatically extract the screenshots of the website using the crawler technology, ensure that each URL has corresponding image data, and assign a hash value to each URL as an index for the website screenshot; In the data processing stage, detailed processing is carried out according to the data characteristics, including filling of missing values and screening of abnormal values; at the same time, a label dimension is added in the data set, and phishing websites are marked as 1 and normal websites are marked as 0; Finally, the overall data set is allocated according to a ratio of 4:1, ensuring that 80% of the samples are used for the training set and 20% of the samples are used for the test set, and ensuring that the proportion of phishing websites and normal websites in the training set and test set remains consistent; Step 2, extract multi-modal features; Step 3, multi-modal fusion: after extracting multi-modal features, the text feature vector and the image feature vector are spliced together; Step 4, iterative optimization: in the final prediction classification, the loss function is the log loss; Step 5, phishing website classification; The multi-modal feature extraction: 1) Image features: in image feature extraction, the visual ViT model is used; ViT uses self-attention mechanism and multi-layer structure; before inputting the model, the image size is unified to 224x224 through resize operation; for input image I, it is divided into several patches of size 16x16 to form a patch sequence P, that is: P={p1,p2,…,pk},k=(224×224) / (16×16)=196; The architecture of the image feature extraction model is shown in the following formula: T image = F image (D image ), T image For image features, F denotes a feature extraction model, D denotes raw data, T image ∈R n×m ; 2) Text features; The text features: in text feature extraction, the BERT model is used to process the URL: The input text sequence is tokenized and converted to Token ID, with special markers added: [CLS]: added at the beginning of the text, used to represent the vector of the entire sentence; [SEP]: added at the end of the text, used to separate different sentences or text markers; The BERT text feature extraction process can be represented by the following formula: T url = F text (D text ), T url is a text feature, T url ∈ R n×d .

2. The phishing website detection method based on multi-modal fusion according to claim 1, wherein, The multi-modal fusion: After extracting multi-modal features, the text feature vector and the image feature vector are spliced together: T f = cat(T url , T image ), T f is the final feature vector, T f ∈R n×(d+m) ; A multi-modal fusion network architecture, which concatenates the obtained feature vectors T f into a machine learning classifier; a logistic regression classifier maps the input features X through a linear function to a real number, and then through a non-linear function to a probability value between 0 and 1; the logistic formula is as follows: where z = W T X + b, which represents the inner product of the feature vector X with the weight vector W plus the bias term b, which expands as follows: z = b + β1X1+ β2X2+... + β d X d = X T β, Where z is the result of linear combination, β1, β2, …, βd are the weights of the features, X is the feature vector, and β is the parameter vector; Weight vector W: a vector representing the importance of each feature in decision-making; for a feature vector X of size d, the weight W has a dimension of d x 1; Bias term b: a scalar representing the threshold adjustment of the classifier, used to control the position of the decision boundary. 3.The phishing website detection method based on multi-modal fusion of claim 1, wherein, The iterative optimization: In the final prediction classification, the loss function is the log loss (Log Loss), whose formula is: where y i is the true label, is the model predicted label; Due to the large dataset and high dimensionality of features, the optimizer selects the LBFGS optimizer; LBFGS realizes efficient unconstrained optimization by approximating the inverse of the Hessian matrix, and the goal is to minimize a differentiable objective function f(x), where x is the parameter vector; in each iteration, LBFGS uses the gradient k of the current point x to guide the optimization; the parameter update formula of LBFGS is as follows: where H k is an approximation of the Hessian matrix, a k is the step size. 4.The phishing website detection method based on multi-modal fusion of claim 1, wherein, The phishing website classification: First, capture the corresponding URL text information and webpage screenshots, and pass these data into the preprocessing module to ensure that the data format is consistent with the model training; Secondly, after the multi-modal fusion model extracts the text features and image features respectively, the two are spliced into a comprehensive feature vector; Thirdly, the fused feature vector is input into the trained logistic regression classifier for prediction; the classifier will determine whether the current website is a phishing website based on the learned patterns; the output result will provide a probability value for each visited website, indicating the risk level of being a phishing website.

5. A phishing website detection system based on multi-modal fusion, which implements the phishing website detection method based on multi-modal fusion according to any one of claims 1-4. The phishing website detection system based on multi-modal fusion comprises: A data processing module is used to collect data containing historical URLs and their corresponding website screenshots in the process of building a phishing website detection model; image information is obtained from the text URL, and the screenshot of the website is automatically extracted using the crawler technology to ensure that each URL has corresponding image data, and a hash value is assigned to each URL as an index for the website screenshot; An extraction module is used to extract multi-modal features: in this stage, multi-modal features are extracted, and the extraction strategy of text features and image features is further explored; A multi-modal fusion module is used to splice the text feature vector and the image feature vector after extracting the multi-modal features; An iterative optimization module is used in the final prediction classification, and the loss function is the logarithmic loss; A classification module is used for phishing website classification.

6. A computer device, comprising: The computer device comprises a memory and a processor, and the memory stores a computer program, which, when executed by the processor, causes the processor to execute the steps of the phishing website detection method based on multi-modal fusion according to any one of claims 1-4. 7.A computer readable storage medium storing a computer program, which, when executed by a processor, causes the processor to execute the steps of the phishing website detection method based on multi-modal fusion according to any one of claims 1-4.

8. An information data processing terminal, characterized by The information data processing terminal is used to implement the phishing website detection system based on multi-modal fusion according to claim 5.

Images