Phishing website detection method and system based on multi-modal fusion

Abstract

The invention belongs to the technical field of website security, discloses a phishing website detection method based on multi-modal fusion, and provides a risk identification method based on a multi-modal deep learning model, which can better adapt to the diversity and imbalance of risk prevention and control data. According to the method, a BERT model is used for extracting text features, a ViT model is used for extracting picture features, and the text features and the picture features are spliced and then input into a logistic classifier for risk identification. Through fusion of text and image information, the method improves the recognition accuracy of the complex risk mode. Through multi-modal feature fusion, risk features can be extracted from data of different modals, and the problem of model deviation caused by proportional imbalance of positive and negative samples is relieved. The combination of the multi-modal features enables the model to be more excellent in learning of minority class samples, and is helpful for improving the recognition capability of scarce and abnormal samples.

Description

Technical Field

[0001] The present invention belongs to the technical field of website security, and in particular relates to a phishing website detection method and system based on multimodal fusion. Background Art

[0002] In today's society, intelligent technology plays a vital role in all kinds of risk prevention and control. By using deep learning and multimodal data processing technology, intelligent prevention and control measures can more comprehensively and accurately identify and predict potential risks, thereby providing effective support for decision-making and ensuring social security and stability. The application of multimodal feature extraction and fusion technology makes the risk identification model more adaptable in dealing with complex and changing risk environments, and can quickly respond to the challenges of new risks.

[0003] The challenges faced in the field of risk prevention and control are mainly reflected in the diversity and rapid changes of risk patterns, the huge amount of data, and the high requirements for real-time response. In addition, risk data often has serious imbalance problems, such as the significant difference in the data ratio between normal behavior and abnormal risk behavior. This imbalance will cause the model to tend to classify data into common categories, thereby affecting recognition accuracy. This bias may cause the model to ignore the existence of risky behavior, reduce the generalization ability of the model, and affect the overall performance and stability.

[0004] Existing phishing website detection technologies are mainly divided into three categories: blacklist-based detection, heuristic-based detection, and machine learning-based detection. The blacklist-based detection method is simple to implement and has a fast query speed. For known phishing websites, it can achieve high accuracy and reduce the possibility of misjudgment and missed judgment. However, more than 93% of phishing web pages are not in the mainstream blacklist records, and the maintenance cost is high, requiring a lot of human resources. At the same time, this method has a lag, and newly emerging phishing websites cannot be detected immediately. The heuristic-based method uses predefined rules to analyze the features and behavior patterns extracted from web pages to identify potential phishing websites to determine whether the web pages are legitimate. Compared with the blacklist, which requires frequent updates, heuristic detection only needs to regularly adjust and optimize features and rules, and the maintenance cost is relatively low, but the heuristic rules are often not complex, and phishers can circumvent heuristic rules by forging relevant features. The machine learning-based method uses different algorithms to train a classifier based on the features extracted from the website according to the samples, and then uses the classifier to classify the website for phishing prediction. Its process mainly consists of five parts: data collection, feature extraction, model training, model testing, and prediction.

[0005] With the rapid development of information technology, the diversification of data types has gradually limited the effectiveness of a single data source in specific application scenarios. Multimodal data fusion technology has emerged, which makes comprehensive use of different types of data (such as text, images, and audio) to more comprehensively capture key information in complex phenomena. In the field of risk prevention and control, the application of multimodal data fusion technology is particularly important. For example, by integrating text information with visual information, researchers can build more accurate models. This technical background provides important support for the present invention, making it possible to use multimodal features for risk detection.

[0006] The application of deep learning technology in the field of risk prevention and control has also received increasing attention, mainly due to its outstanding performance in large-scale data analysis and complex pattern recognition. As the means of cyber attacks become more diverse and concealed, traditional rule-based methods have become difficult to deal with new risk threats. Deep learning models can automatically extract meaningful features from large amounts of data and identify potential risks, and are particularly outstanding when processing unstructured data (such as text and images). The introduction of this technology not only improves the accuracy and efficiency of risk detection, but also enhances the model's adaptability in the face of changes and complex environments, providing a solid foundation for the realization of more intelligent risk prevention and control solutions.

[0007] Through the above analysis, the problems and defects of the prior art are as follows:

[0008] With the popularization of the Internet, phishing websites have become one of the major challenges in the field of network security. These websites usually disguise themselves as legitimate websites to deceive users into obtaining personal sensitive information, such as account passwords, credit card information, etc. Traditional phishing website detection methods mainly rely on single feature extraction technology, such as URL text features or website image features. These methods have the following problems:

[0009] Limitations of single features: Traditional phishing website detection methods usually rely on text analysis or image analysis of URLs, but cannot fully tap the correlation between the two. Relying on a single data source can lead to missed or false positives, especially when facing hybrid phishing websites (i.e. websites that combine URL disguise and image forgery technology).

[0010] Lack of deep semantic understanding: Traditional keyword or rule-based phishing website detection methods cannot understand the potential semantic information in the page. Phishing websites usually use clever designs to mislead users, so more advanced analysis techniques are needed to process website content.

[0011] Slow detection speed: Traditional methods often rely on manually designed rules or simple machine learning models, which cannot quickly process large amounts of web content in real time, resulting in performance bottlenecks when processing real-time website traffic.

[0012] Low detection accuracy: As phishing websites often evade detection through disguise and detail adjustments, existing methods have low accuracy when faced with complex phishing methods, especially the inability to detect dynamic changes in pages or deceptive elements in images. Summary of the invention

[0013] In view of the problems existing in the prior art, the present invention provides a phishing website detection method based on multimodal fusion.

[0014] The present invention is implemented in this way: a phishing website detection method based on multimodal fusion includes:

[0015] Step 1, data processing:

[0016] In the process of building a phishing website detection model, we first collect data containing historical URLs and their corresponding website screenshots; obtain image information from text URLs, use crawler technology to automatically extract website screenshots, ensure that each URL has corresponding image data, and assign a hash value to each URL to be used as an index for website screenshots;

[0017] In the data processing stage, the data is processed in detail according to its characteristics, including filling in missing values ​​and filtering outliers. At the same time, a label dimension is added to the data set, marking phishing websites as 1 and normal websites as 0.

[0018] Finally, the overall data set is allocated in a 4:1 ratio, ensuring that 80% of the samples are used for the training set and 20% of the samples are used for the test set, and that the ratio of phishing websites and normal websites in the training set and test set is kept consistent;

[0019] Step 2, extracting multimodal features: In this stage, we extract multimodal features and explore in depth the extraction strategies of text features and image features;

[0020] Step 3, multimodal fusion: after extracting multimodal features, concatenate the text feature vector and the image feature vector together;

[0021] Step 4, iterative optimization: In the final prediction classification, the loss function is logarithmic loss;

[0022] Step 5: Classify phishing websites.

[0023] Further, the multimodal features are extracted:

[0024] 1) Image features: In image feature extraction, the visual ViT model is used; ViT uses a self-attention mechanism and a multi-layer structure; before entering the model, the image size is unified to 224×224 through a resize operation. For the input image I, it is divided into several patches of size 16×16 to form a patch sequence P, that is:

[0025] P={p1,p2,…,pk},k=(224×224) / (16×16)=196;

[0026] The architecture of the image feature extraction model is:

[0027] T image =F image (D image ),

[0028] T image is the image feature, T image ∈R n×m ;

[0029] 2) Text features.

[0030] Furthermore, the text features: In the text feature extraction, the URL is processed using the BERT model:

[0031] The input text sequence is tokenized and converted to TokenIDs, with special tags added:

[0032] [CLS]: Added at the beginning of the text to represent the vector of the entire sentence;

[0033] [SEP]: A marker added at the end of the text to separate different sentences or texts;

[0034] After the word segmentation stage, the input sequence of "www.baidu.com" becomes "[CLS]wwwbaidu com[SEP]"; the process of BERT extracting text features can be expressed by the formula:

[0035] T url =F text (D text ), T url is the text feature, T url ∈R n×d .

[0036] Furthermore, the multimodal fusion:

[0037] After extracting multimodal features, concatenate the text feature vector with the image feature vector:

[0038] T f =cat(Turl , T image ), T f is the final eigenvector, T f ∈R n×(d+m) ;

[0039] Multimodal fusion network architecture, the concatenated feature vector T f As input to a common machine learning classifier; the logistic regression classifier maps the input feature X to a real value through a linear function, and then converts this real value into a probability value between 0 and 1 through a nonlinear function; the logistic formula is as follows:

[0040]

[0041] where z = W T X+b represents the inner product of the feature vector X and the weight vector W plus the bias term b, which is expanded as follows:

[0042] z=b+β1X1+β2X2+…+β d X d =X T β,

[0043] Among them, z is the result of linear combination, β1,β2,...,β d is the weight of the feature, X is the feature vector, and β is the parameter vector;

[0044] Weight vector W: A vector that represents the importance of each feature in the decision; for a feature vector X (size d), the dimension of weight W is usually d×1;

[0045] Bias term b: A scalar that represents the threshold adjustment of the classifier, usually used to control the location of the decision boundary.

[0046] Furthermore, the iterative optimization:

[0047] In the final prediction classification, the loss function is logarithmic loss (LogLoss), and its formula is:

[0048]

[0049] Among them, y i is the true label, is the model prediction label;

[0050] Due to the large data set and high feature dimension, the optimizer selects the LBFGS optimizer, which is efficient, adaptable and has good convergence. LBFGS achieves efficient unconstrained optimization by approximating the inverse of the Hessian matrix. The goal is to minimize a differentiable objective function f(x), where x is a parameter vector. In each iteration, LBFGS uses the current point x k Gradient To guide the optimization; the parameter update formula of LBFGS is as follows:

[0051]

[0052] Among them, H k is an approximation of the Hessian matrix, α k is the step length.

[0053] Furthermore, the phishing websites are classified as follows:

[0054] First, capture the corresponding URL text information and webpage screenshots, and pass these data into the preprocessing module to ensure that the data format is consistent with that during model training;

[0055] Secondly, after the multimodal fusion model extracts text features and image features respectively, the two are spliced ​​into a comprehensive feature vector;

[0056] Again, the fused feature vector will be input into the trained logistic regression classifier for prediction; the classifier will determine whether the current website is a phishing website based on the learned pattern; the output result will provide a probability value for each visited website, indicating the risk level of it being a phishing website.

[0057] Another object of the present invention is to provide a phishing website detection system based on multimodal fusion, comprising:

[0058] The data processing module is used to first collect data including historical URLs and their corresponding website screenshots in the process of building a phishing website detection model; obtain image information from the text URL, automatically extract website screenshots using crawler technology, ensure that each URL has corresponding image data, and assign a hash value to each URL as an index for the website screenshots;

[0059] Extraction module, used to extract multimodal features: In this stage, multimodal features are extracted and the extraction strategies of text features and image features are explored in depth;

[0060] The multimodal fusion module is used to extract multimodal features and then concatenate the text feature vector with the image feature vector;

[0061] Iterative optimization module, used in the final prediction classification, the loss function is logarithmic loss;

[0062] Classification module, used for phishing website classification.

[0063] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the phishing website detection method based on multimodal fusion.

[0064] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the phishing website detection method based on multimodal fusion.

[0065] Another object of the present invention is to provide an information data processing terminal, which is used to implement the phishing website detection system based on multimodal fusion.

[0066] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0067] First, the present invention proposes a risk identification method based on a multimodal deep learning model, which can better adapt to the diversity and imbalance of risk prevention and control data. This method uses the BERT model to extract text features and the ViT model to extract image features, and then splices the two and inputs them into the logistic classifier for risk identification. Through the fusion of text and image information, this method improves the recognition accuracy of complex risk patterns.

[0068] Alleviate the problem of data imbalance: Through multimodal feature fusion, this technology can extract risk features from data of different modalities, alleviating the model bias problem caused by the imbalance of positive and negative sample ratios. The combination of multimodal features makes the model perform better in learning minority samples, which helps improve the ability to identify scarce and abnormal samples.

[0069] Improved recognition accuracy: This technology fully utilizes the information of different modal features by fusing the text features extracted by BERT and the image features extracted by ViT, and improves the model's ability to recognize complex risk patterns. Compared with the traditional method of single feature input, this multimodal feature fusion shows higher accuracy when dealing with diverse and complex risks, thereby significantly optimizing the prediction performance.

[0070] Second, the expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0071] Improve network security protection capabilities: The phishing website detection method of the present invention can monitor in real time and warn whether the website visited by the user is a phishing website, effectively reducing network fraud and information leakage incidents, and improving the overall network security protection level.

[0072] Enhanced user experience: Through the plug-in form, users can obtain phishing website detection services without complicated operations, simplifying the process of users identifying phishing websites.

[0073] Reduce enterprise losses: By deploying the phishing website detection plug-in of the present invention, enterprises can reduce economic losses and reputation damage caused by employees visiting phishing websites, thereby reducing enterprise operation risks.

[0074] Create new revenue channels: The phishing website detection plug-in of the present invention can be provided to users as a value-added service, creating a new source of revenue for enterprises. At the same time, through advertising, data analysis and other modes, it can also achieve diversified commercial value.

[0075] Third, the technical problems of the prior art solved by the technical solution of the present invention in industrial applications and the significant technical progress achieved:

[0076] The phishing website detection method based on multimodal fusion proposed in this invention innovatively combines image features and text features, effectively solves the shortcomings of traditional methods through deep learning and multimodal data fusion, and has significant technical progress:

[0077] Multimodal feature fusion: The present invention detects phishing websites by combining two information sources, image and text. The BERT model is used to extract the text features of the URL, and the ViT (Visual Transformer) model extracts the image features of the website screenshot. This multimodal feature extraction can more comprehensively capture the different manifestations of phishing websites. By concatenating the text and image feature vectors and inputting them into a machine learning classifier (such as logistic regression) for processing, the accuracy of detection is further improved.

[0078] Application of deep learning model: In the process of image feature extraction, the present invention uses the Visual Transformer (ViT) model, which is a deep learning method for processing images through a self-attention mechanism and a multi-layer structure. This method can effectively extract deep features in images with fewer training samples, and improves the recognition ability of complex image content (such as disguised logos, deceptive page layouts, etc.).

[0079] Application of BERT model in text feature extraction: Using BERT model to process URL text can better understand and extract semantic information in URL and identify potential deceptive URLs. By considering contextual information, BERT model can identify phishing techniques that are more complex and covert than traditional methods, such as attacks through URL disguise.

[0080] Improve detection accuracy and robustness: The method of the present invention can simultaneously identify deceptive behaviors in URLs and images, has high detection accuracy, and can effectively identify hybrid phishing websites, especially those that use complex camouflage techniques.

[0081] Real-time detection and high efficiency: The present invention can process and analyze website content in real time through the combination of multimodal fusion and deep learning models, and has higher processing efficiency and lower false alarm rate than traditional methods. The multimodal fusion network can handle more complex network environments and provide fast response while ensuring high detection accuracy.

[0082] Adaptability to complex network environments: The multimodal fusion detection method of the present invention has strong adaptability and can cope with different types of phishing website attacks, whether it is through URL camouflage, website layout camouflage, or deception through image content, it can effectively detect.

[0083] In summary, the present invention greatly improves the accuracy, efficiency and robustness of phishing website detection through the innovative method of multimodal feature fusion, and can provide higher security protection, especially when facing complex and cleverly designed phishing attacks. These technological advances have important application value in the current field of network security, especially in protecting users from phishing website attacks and improving network security, and can provide more reliable protection for various network applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 It is a flow chart of a phishing website detection method based on multimodal fusion provided in an embodiment of the present invention.

[0085] Figure 2 It is a structural block diagram of a phishing website detection system based on multimodal fusion provided in an embodiment of the present invention.

[0086] Figure 3 It is a diagram of the multimodal fusion model architecture provided by an embodiment of the present invention.

[0087] Figure 4 This is a workflow diagram of a browser plug-in provided by an embodiment of the present invention.

[0088] Figure 5 It is a schematic diagram for comparing experimental results provided by an embodiment of the present invention.

[0089] Figure 6 It is a comparative bar chart of experimental results provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0091] Example 1: Phishing website detection in online banking applications

[0092] In the financial industry, especially in online banking and payment platforms, phishing website attacks are one of the most common security threats. Hackers disguise themselves as legitimate banking websites to trick users into entering sensitive information, such as usernames, passwords, and bank card information. In order to effectively deal with this threat, the online banking system adopts the phishing website detection method based on multimodal fusion of the present invention.

[0093] Application Process:

[0094] 1. Data collection: The system collects bank-related URLs and web page screenshots through crawler technology. Through automated procedures, the system obtains URL information of visited websites and corresponding web page screenshot data in real time.

[0095] 2. Feature extraction: The BERT model is used to extract features from the URL text to identify potential malicious and forged behaviors. At the same time, the system uses the ViT model to perform in-depth analysis of the images in the screenshots to identify forged elements in the images (such as disguised bank logos, phishing buttons, etc.).

[0096] 3. Multimodal fusion: Fusion of text features and image features to form a unified feature vector that is input into the classification model. The system will use a logistic regression classifier to determine whether the URL is a phishing website.

[0097] 4. Real-time monitoring and early warning: The system continuously monitors the URLs visited, and through real-time detection, ensures that before the user visits, it promptly identifies and warns the user whether the website is a phishing website. If a phishing website is detected, the system will automatically block access to ensure that the user's sensitive information is not stolen.

[0098] Technical advantages:

[0099] Utilizes multimodal data of text and images to make detection results more accurate and able to identify complex phishing attacks;

[0100] Efficient real-time processing capabilities ensure that the online banking platform can respond immediately and reduce potential security threats to users;

[0101] Improved user experience and platform security.

[0102] Example 2: Phishing protection for e-commerce platforms

[0103] E-commerce platforms are facing a large number of phishing websites, which disguise themselves as legitimate e-commerce websites and trick users into making fake transactions or leaking personal payment information. In order to protect users' privacy and financial security, e-commerce platforms have deployed phishing website detection systems based on multimodal fusion.

[0104] Application Process:

[0105] 1. Data collection: The system collects URL information and related web page screenshots of e-commerce platforms through web crawler technology to ensure that relevant data of each e-commerce website is collected and analyzed.

[0106] 2. Feature extraction: The system analyzes the text content of the URL, extracts text features, and uses the BERT model to identify potential risks of malicious URLs. At the same time, the ViT model extracts image features from web page screenshots to identify forged page elements and layouts (such as disguised payment buttons, false product descriptions, etc.).

[0107] 3. Feature fusion and classification: The extracted text features and image features are combined and input into the logistic regression classifier to classify phishing websites. The system improves detection accuracy by training and optimizing the classification model.

[0108] 4. Real-time feedback and user protection: The system monitors the access traffic of e-commerce websites in real time and can quickly identify and classify the websites visited by users. When users visit, the system automatically detects phishing websites. If a phishing website is found, the system will immediately issue a warning to prevent users from entering sensitive information or conducting false transactions.

[0109] Technical advantages:

[0110] The fusion of multimodal features improves the recognition accuracy of phishing websites and can effectively prevent new and hidden phishing attacks;

[0111] The system responds in real time and can provide timely protection and reduce risks when users access e-commerce platforms;

[0112] By combining the BERT and ViT models, the system can cope with various types of phishing attacks, including text disguise and image disguise.

[0113] These two embodiments demonstrate the powerful capabilities of the method of the present invention in practical applications. It not only improves the detection accuracy of phishing websites, but also greatly enhances the security protection capabilities of the platform and protects the privacy and financial security of users.

[0114] like Figure 1 As shown, a phishing website detection method based on multimodal fusion provided by an embodiment of the present invention includes the following steps:

[0115] S101, data processing:

[0116] In the process of building a phishing website detection model, we first collect data containing historical URLs and their corresponding website screenshots; obtain image information from text URLs, use crawler technology to automatically extract website screenshots, ensure that each URL has corresponding image data, and assign a hash value to each URL to be used as an index for website screenshots;

[0117] In the data processing stage, the data is processed in detail according to its characteristics, including filling in missing values ​​and filtering outliers. At the same time, a label dimension is added to the data set, marking phishing websites as 1 and normal websites as 0.

[0118] Finally, the overall data set is allocated in a 4:1 ratio, ensuring that 80% of the samples are used for the training set and 20% of the samples are used for the test set, and that the ratio of phishing websites and normal websites in the training set and test set is kept consistent;

[0119] S102, extract multimodal features: In this stage, extract multimodal features and explore in depth the extraction strategies of text features and image features;

[0120] S103, multimodal fusion: after extracting multimodal features, concatenate the text feature vector and the image feature vector together;

[0121] S104, iterative optimization: in the final prediction classification, the loss function is logarithmic loss;

[0122] S105, phishing website classification.

[0123] The multimodal feature extraction provided by the embodiment of the present invention:

[0124] 1) Image features: In image feature extraction, the visual ViT model is used; ViT uses a self-attention mechanism and a multi-layer structure; before entering the model, the image size is unified to 224×224 through a resize operation. For the input image I, it is divided into several patches of size 16×16 to form a patch sequence P, that is:

[0125] P={p1,p2,…,pk},k=(224×224) / (16×16)=196;

[0126] The architecture of the image feature extraction model is:

[0127] Timage =F image (D image ),

[0128] T image is the image feature, T image ∈R n×m ;

[0129] 2) Text features.

[0130] Text features provided by an embodiment of the present invention: In text feature extraction, the BERT model is used to process the URL:

[0131] The input text sequence is tokenized and converted to TokenIDs, with special tags added:

[0132] [CLS]: Added at the beginning of the text to represent the vector of the entire sentence;

[0133] [SEP]: A marker added at the end of the text to separate different sentences or texts;

[0134] After the word segmentation stage, the input sequence of "www.baidu.com" becomes "[CLS]wwwbaidu com[SEP]"; the process of BERT extracting text features can be expressed by the formula:

[0135] T url =F text (D text ), T url is the text feature, T url ∈R n×d .

[0136] The multimodal fusion provided by the embodiment of the present invention:

[0137] After extracting multimodal features, concatenate the text feature vector with the image feature vector:

[0138] T f =cat(T url , T image ), T f is the final eigenvector, T f ∈R n×(d+m) ;

[0139] Multimodal fusion network architecture, the concatenated feature vector T f As input to a common machine learning classifier; the logistic regression classifier maps the input feature X to a real value through a linear function, and then converts this real value into a probability value between 0 and 1 through a nonlinear function; the logistic formula is as follows:

[0140]

[0141] where z = W T X+b represents the inner product of the feature vector X and the weight vector W plus the bias term b, which is expanded as follows:

[0142] z=b+β1X1+β2X2+…+β d X d =X T β,

[0143] Among them, z is the result of linear combination, β1,β2,...,β d is the weight of the feature, X is the feature vector, and β is the parameter vector;

[0144] Weight vector W: A vector that represents the importance of each feature in the decision; for a feature vector X (size d), the dimension of weight W is usually d×1;

[0145] Bias term b: A scalar that represents the threshold adjustment of the classifier, usually used to control the location of the decision boundary.

[0146] Iterative optimization provided by the embodiment of the present invention:

[0147] In the final prediction classification, the loss function is logarithmic loss (LogLoss), and its formula is:

[0148]

[0149] Among them, y i is the true label, is the model prediction label;

[0150] Due to the large data set and high feature dimension, the optimizer selects the LBFGS optimizer, which is efficient, adaptable and has good convergence. LBFGS achieves efficient unconstrained optimization by approximating the inverse of the Hessian matrix. The goal is to minimize a differentiable objective function f(x), where x is a parameter vector. In each iteration, LBFGS uses the gradient of the current point xk. To guide the optimization; the parameter update formula of LBFGS is as follows:

[0151]

[0152] Among them, H k is an approximation of the Hessian matrix, α k is the step length.

[0153] Phishing website classification provided by the embodiment of the present invention:

[0154] First, capture the corresponding URL text information and webpage screenshots, and pass these data into the preprocessing module to ensure that the data format is consistent with that during model training;

[0155] Secondly, after the multimodal fusion model extracts text features and image features respectively, the two are spliced ​​into a comprehensive feature vector;

[0156] Again, the fused feature vector will be input into the trained logistic regression classifier for prediction; the classifier will determine whether the current website is a phishing website based on the learned pattern; the output result will provide a probability value for each visited website, indicating the risk level of it being a phishing website.

[0157] like Figure 2 As shown, a phishing website detection system based on multimodal fusion provided by an embodiment of the present invention includes:

[0158] The data processing module is used to first collect data including historical URLs and their corresponding website screenshots in the process of building a phishing website detection model; obtain image information from the text URL, automatically extract website screenshots using crawler technology, ensure that each URL has corresponding image data, and assign a hash value to each URL as an index for the website screenshots;

[0159] Extraction module, used to extract multimodal features: In this stage, multimodal features are extracted and the extraction strategies of text features and image features are explored in depth;

[0160] The multimodal fusion module is used to extract multimodal features and then concatenate the text feature vector with the image feature vector;

[0161] Iterative optimization module, used in the final prediction classification, the loss function is logarithmic loss;

[0162] Classification module, used for phishing website classification.

[0163] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the phishing website detection method based on multimodal fusion.

[0164] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the phishing website detection method based on multimodal fusion.

[0165] Another object of the present invention is to provide an information data processing terminal, which is used to implement the phishing website detection system based on multimodal fusion.

[0166] The present invention is specifically implemented:

[0167] This technology focuses on the field of social risk prevention and control, especially intelligent solutions for phishing website detection. Compared with traditional phishing website detection technology, the latter often relies more on tedious manual identification and analysis. In the data preparation stage, traditional methods require manual data sampling, cleaning and labeling; in the feature extraction stage, simple statistical methods and a large amount of manual extraction are required. These methods are not only labor-intensive and inefficient, but may also miss potential risk signals. The homepage information of the web page, as a visual signal that users are directly exposed to, often contains more phishing elements. Therefore, making full use of web page screenshot information is crucial for phishing website detection. This technology utilizes multimodal feature information, which not only improves the accuracy of detection, but also significantly reduces the need for manual intervention.

[0168] In the phishing website detection task, the goal is to use the feature information of images and URLs to effectively identify malicious websites. Define the following symbols: a text dataset Dtext = {u1,u2,…,un} containing multiple URLs, and an image dataset D containing screenshots of the corresponding URLs image ={I1,I2,…In}. Use the text feature extraction model to extract text features F from the URL text :D text →R d , using the image feature extraction model F image :D image →R m Extract image features, where d is the dimension of text features and m is the dimension of image features. Fusion model F r , which is used to combine text features and image features for phishing website prediction. The joint optimization objective is as follows:

[0169] θ(F image ,F text ,F r )=argminL(F r (F image (D image ),F text ,(D text )),y)

[0170] Where L represents the loss function, which reflects the difference between the model prediction output and the true label.

[0171] First, the necessary data preprocessing is performed, which includes data cleaning, noise removal, and incomplete sample processing to ensure data quality and accuracy. Next, the BERT model is used to extract text features from the URL. Secondly, the ViT model is used to extract image features from website screenshots. Subsequently, the extracted text features and image features are concatenated to form a comprehensive feature vector, which is fed into the logistic classifier as input for phishing website detection. Finally, a comprehensive evaluation system is established to analyze and improve the overall performance of the model, aiming to solve the problem of improving prediction accuracy. The specific steps are as follows:

[0172] Step 1: Data processing: In the process of building a phishing website detection model, we first collect data containing historical URLs and their corresponding website screenshots. In order to obtain image information from text URLs, we use crawler technology to automatically extract screenshots of websites, ensure that each URL has corresponding image data, and assign a hash value to each URL as an index for website screenshots. These data may come from a variety of channels, such as network monitoring, user feedback, and security reports, so there may be differences in storage format and attribute categories. In the data processing stage, we perform detailed processing based on data characteristics, including filling in missing values ​​and screening outliers to improve data reliability. At the same time, we will add a label dimension to the data set, marking phishing websites as 1 and normal websites as 0. Finally, the overall data set is allocated in a ratio of 4:1, ensuring that 80% of the samples are used for the training set and 20% of the samples are used for the test set, and the ratio of phishing websites and normal websites in the training set and test set is kept consistent. This allocation strategy helps the model fully reflect the real situation in the training and testing stages, thereby improving the detection effect.

[0173] Step 2: Extract multimodal features: In this stage, we focus on extracting multimodal features and explore in depth the extraction strategies of text features and image features.

[0174] 1. Image features: In image feature extraction, the Visual Transformer (ViT) model is used. ViT can efficiently identify key visual elements in images, such as color, layout, and pattern features, through the self-attention mechanism and multi-layer structure. Before entering the model, the image size is unified to 224×224 through the resize operation. For the input image I, it is divided into several patches of size 16×16 to form a patch sequence P, that is:

[0175] P = {p1, p2, ..., pk}, k = (224 × 224) / (16 × 16) = 196. The architecture of the image feature extraction model is shown in the table:

[0176] Parameter name parameter Patch size 16 transformer layers 12 Number of attention heads 12 Output feature dimension 768

[0177] T image =F image (D image ), T image is the image feature, T image ∈R n×m .

[0178] 2. Text features: In text feature extraction, the BERT model is used to process the URL. BERT has powerful language processing capabilities and can deeply understand the complex semantics and contextual relationships in the text to generate high-dimensional feature vectors. If the structure of this process can be described in the following form:

[0179] Parameter name parameter Maximum sequence length 512 transformer layers 12 Number of attention heads 12 Output feature dimension 768 Total parameters 110M

[0180] The input text sequence is tokenized and converted to TokenIDs, with special tags added:

[0181] [CLS]: Added at the beginning of the text to represent the vector of the entire sentence.

[0182] [SEP]: A tag added at the end of a text to separate different sentences or texts.

[0183] After the word segmentation stage, the input sequence of "www.baidu.com" becomes "[CLS]wwwbaidu com[SEP]". The process of BERT extracting text features can be expressed by the formula:

[0184] T url =F text (D text ), T url is the text feature, T url ∈R n×d .

[0185] Step 3: Multimodal fusion: After extracting multimodal features, concatenate the text feature vector and the image feature vector together.

[0186] T f =cat(T url , T image ), T f is the final eigenvector, T f ∈R n×(d+m) The multimodal fusion network architecture is as follows: Figure 3 As shown. The concatenated feature vector T fAs input to common machine learning classifiers, such as LogisticRegression. The logistic regression classifier maps the input feature X (including a bias term) to a real value through a linear function, and then converts this real value into a probability value between 0 and 1 through a nonlinear function (usually the Sigmoid function). The logistic formula is as follows:

[0187]

[0188] where z = W T X+b represents the inner product of the feature vector X and the weight vector W plus the bias term b, which is expanded as follows:

[0189] z=b+β1X1+β2X2+…+β d X d =X T β,

[0190] Among them, z is the result of linear combination, β1,β2,...,βd are the weights of features, X is the feature vector, and β is the parameter vector.

[0191] Weight vector W: A vector that indicates the importance of each feature in the decision. For a feature vector X (of size d), the dimension of weight W is usually d×1.

[0192] Bias term b: A scalar that represents the threshold adjustment of the classifier, usually used to control the location of the decision boundary.

[0193] Step 4: Iterative optimization: In the final prediction classification, the loss function is logarithmic loss (LogLoss), and its formula is:

[0194]

[0195] Among them, y i is the true label, is the model prediction label.

[0196] Due to the large data set and high feature dimension, the optimizer selects the LBFGS optimizer, which is efficient, adaptable and has good convergence. LBFGS achieves efficient unconstrained optimization by approximating the inverse of the Hessian matrix. The goal is to minimize a differentiable objective function f(x), where x is a parameter vector. In each iteration, LBFGS uses the current point x k Gradient To guide the optimization. The parameter update formula of LBFGS is as follows:

[0197]

[0198] Among them, H k is an approximation of the Hessian matrix, α k is the step length.

[0199] Step 5: Phishing website classification: In this stage, a trained multimodal fusion network model is obtained. When performing the phishing website classification task, the corresponding URL text information and webpage screenshots are first captured, and these data are passed into the preprocessing module to ensure that the data format is consistent with the model training. Secondly, after the multimodal fusion model extracts the text features and image features respectively, the two are spliced ​​into a comprehensive feature vector. Thirdly, the fused feature vector will be input into the trained logistic regression classifier for prediction. The classifier will determine whether the current website is a phishing website based on the learned pattern. The output result will provide a probability value for each visited website, indicating the risk level of it being a phishing website.

[0200] 4.1 Data Preparation

[0201] The Phishingwebsite phishing website detection dataset is collected from PhishTank. PhishTank is a well-known community-driven platform dedicated to collecting and sharing information about phishing websites. In this dataset, a large number of website URLs marked as phishing or normal are obtained, including URLs and website screenshots. These data provide rich training samples for the model, which helps to identify and classify potential phishing websites. By utilizing these datasets, the characteristics of phishing websites can be deeply analyzed to improve network security. 0 represents a normal website, 1 represents a phishing website, and the original dataset is divided into 80% training set and 20% test set.

[0202] 4.2 Indicator Design

[0203] In the field of phishing website detection, fraud detection has become an important research hotspot, and researchers have proposed a variety of models to identify fraudulent behavior in transaction data. However, most studies often rely on a single indicator, such as accuracy and precision, when evaluating model performance. Given that phishing websites account for a very small proportion of all websites, the data set faces a serious imbalance problem, and simply using conventional evaluation indicators such as accuracy is not enough to fully evaluate the effectiveness of the model. Therefore, it is particularly important to choose a more suitable evaluation indicator.

[0204] Especially in the scenario of phishing website detection, it is necessary to minimize missed detection of phishing websites, ensure high accuracy and further improve the accuracy of normal website recognition. In order to solve the problem of selecting evaluation indicators, this study proposes a strategy: while controlling the first type error rate (FNR), minimize missed detection of phishing websites, and on this basis reduce the second type error rate (FPR). This strategy aims to ensure that the model can effectively reduce the misjudgment of normal websites while reducing the risk of missed detection.

[0205] By controlling FNR, the model can more accurately identify phishing behaviors and reduce the risk of missed reports. At the same time, reducing FPR while controlling FNR means that the model can also maintain a high accuracy when identifying normal websites and reduce unnecessary false positives. In addition, paying attention to both FNR and FPR can better balance the model's performance in identifying phishing websites and normal websites, making the model more robust and providing an intuitive basis for subsequent tuning and improvement.

[0206] Table 4.1 Confusion Matrix

[0207]

[0208] The table shows the four types of prediction results of the confusion matrix. The correct prediction is TP, which means the actual situation is risky and the predicted situation is risky; the first type of error is FN, which means the actual situation is risky and the predicted situation is risk-free; the second type of error is FP, which means the actual situation is risk-free but the model misjudges it as risky; the correct prediction is TN, which means the actual situation is risk-free and the predicted situation is risk-free. The model is evaluated using five indicators: the false positive rate (FPR) and false negative rate (FNR) expressed by formula (1) and formula (2), the accuracy expressed by formula (3), the recall rate expressed by formula (4), and the F1 score expressed by formula (5). The lower the FNR and FPR, the better the model.

[0209] FNR=FN / (TP+FN)#(1)

[0210] FPR=FP / (FP+TN)#(2)

[0211]

[0212] 4.3 Phishing website detection experimental results

[0213] Five classifiers, LogisticRegression, LinearSVC, RandomForestClassifier, DecisionTreeClassifier, and SGDClassifier, were used to classify and predict multimodal features. The experimental results are as follows:

[0214]

[0215] On the Phishingwebsite phishing website detection dataset, multiple classifiers were able to complete the prediction and achieved good results, which proved the effectiveness of the multimodal feature fusion used in this method.

[0216] 4.4 Phishing URL Detection Plugin

[0217] Based on this technical method, a browser plug-in can be made to detect phishing websites. The plug-in usage process is as follows: Figure 4 As shown,

[0218] First, click the plug-in button, the front-end program obtains the website URL and the current page screenshot through the browser port, and then sends it to the back-end program. The back-end program obtains the data through decoding. Secondly, load the pre-trained BERT model and ViT model for multimodal feature extraction. After obtaining the extracted features, load the logistic regression classifier that has been trained on the phishingwebsite dataset to predict phishing websites. If the predicted result is a phishing website, a warning will be issued to the user on the front end. If it is a legitimate website, a prompt will be given.

[0219] Relevant evidence of the technical effects achieved by the embodiments of the present invention.

[0220] Table 1 Comparison of experimental results

[0221]

[0222] LogisticRegression and SGDClassifier perform best in all indicators, showing their high accuracy and good balance. LinearSVC follows closely, although its performance is slightly worse, but still maintains high precision and recall. Figure 6As shown in the figure, in comparison, RandomForestClassifier and DecisionTreeClassifier perform poorly, especially in Recall and F-Score, where the scores are significantly lower than the former ones, indicating that they are weak in capturing positive samples. Overall, LogisticRegression and SGDClassifier perform well in all three indicators, fully demonstrating the superiority of this method in the task of phishing website identification.

[0223] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0224] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A phishing website detection method based on multimodal fusion, characterized in that: The following steps are involved: Step 1, data processing: In the process of building a phishing website detection model, we first collect data containing historical URLs and their corresponding website screenshots; obtain image information from text URLs, use crawler technology to automatically extract website screenshots, ensure that each URL has corresponding image data, and assign a hash value to each URL to be used as an index for website screenshots; In the data processing stage, the data is processed in detail according to its characteristics, including filling in missing values ​​and filtering outliers. At the same time, a label dimension is added to the data set, marking phishing websites as 1 and normal websites as 0. Finally, the overall data set is allocated in a 4:1 ratio, ensuring that 80% of the samples are used for the training set and 20% of the samples are used for the test set, and that the ratio of phishing websites and normal websites in the training set and test set is kept consistent; Step 2, extract multimodal features: In this stage, extract multimodal features and explore in depth the extraction strategies of text features and image features; Step 3, multimodal fusion: after extracting multimodal features, concatenate the text feature vector and the image feature vector together; Step 4, iterative optimization: In the final prediction classification, the loss function is logarithmic loss; Step 5: Classify phishing websites.

2. The phishing website detection method based on multimodal fusion as claimed in claim 1, characterized in that: The multimodal feature extraction: 1) Image features: In image feature extraction, the visual ViT model is used; ViT uses a self-attention mechanism and a multi-layer structure; before entering the model, the image size is unified to 224×224 through a resize operation. For the input image I, it is divided into several patches of size 16×16 to form a patch sequence P, that is: P={p1,p2,…,pk},k=(224×224) / (16×16)=196; The architecture of the image feature extraction model is: T image =F image (D image ), T image is the image feature, T image ∈R n×m ; 2) Text features.

3. The phishing website detection method based on multimodal fusion as claimed in claim 1, characterized in that: The text features: In text feature extraction, the BERT model is used to process the URL: The input text sequence is tokenized and converted into token IDs, and special tags are added: [CLS]: Added at the beginning of the text to represent the vector of the entire sentence; [SEP]: A marker added at the end of the text to separate different sentences or texts; After the word segmentation stage, the input sequence of "www.baidu.com" becomes "[CLS]wwwbaidu com[SEP]"; the process of BERT extracting text features can be expressed by the formula: T url =F text (D text ), T url is the text feature, T url ∈R n×d .

4. The phishing website detection method based on multimodal fusion as claimed in claim 1, characterized in that: The multimodal fusion: After extracting multimodal features, concatenate the text feature vector with the image feature vector: T f =cat(T url , T image ), T f is the final eigenvector, T f ∈R n×(d+m) ; Multimodal fusion network architecture, the concatenated feature vector T f As input to a common machine learning classifier; the logistic regression classifier maps the input feature X to a real value through a linear function, and then converts this real value into a probability value between 0 and 1 through a nonlinear function; the logistic formula is as follows: where z = W T X+b represents the inner product of the feature vector X and the weight vector W plus the bias term b, which is expanded as follows: z=b+β1X1+β2X2+…+β d X d =X T b, Among them, z is the result of linear combination, β1,β2,...,βd are the weights of features, X is the feature vector, and β is the parameter vector; Weight vector W: A vector that represents the importance of each feature in the decision; for a feature vector X (size d), the dimension of weight W is usually d×1; Bias term b: A scalar that represents the threshold adjustment of the classifier, usually used to control the location of the decision boundary.

5. The phishing website detection method based on multimodal fusion as claimed in claim 1, characterized in that: The iterative optimization: In the final prediction classification, the loss function is log loss (Log Loss), and its formula is: Among them, y i is the true label, is the model prediction label; Due to the large data set and high feature dimension, the optimizer selects the LBFGS optimizer, which is efficient, adaptable and has good convergence. LBFGS achieves efficient unconstrained optimization by approximating the inverse of the Hessian matrix. The goal is to minimize a differentiable objective function f(x), where x is a parameter vector. In each iteration, LBFGS uses the current point x k Gradient To guide the optimization; the parameter update formula of LBFGS is as follows: Among them, Hk is the approximation of the Hessian matrix and αk is the step size.

6. The phishing website detection method based on multimodal fusion as claimed in claim 1, characterized in that: The phishing website classification: First, capture the corresponding URL text information and webpage screenshots, and pass these data into the preprocessing module to ensure that the data format is consistent with that during model training; Secondly, after the multimodal fusion model extracts text features and image features respectively, the two are concatenated into a comprehensive feature vector; Again, the fused feature vector will be input into the trained logistic regression classifier for prediction; the classifier will determine whether the current website is a phishing website based on the learned pattern; the output result will provide a probability value for each visited website, indicating the risk level of it being a phishing website.

7. A phishing website detection system based on multimodal fusion implementing the phishing website detection method based on multimodal fusion as claimed in any one of claims 1 to 6, characterized in that: The phishing website detection system based on multimodal fusion includes: The data processing module is used to first collect data including historical URLs and their corresponding website screenshots in the process of building a phishing website detection model; obtain image information from the text URL, automatically extract website screenshots using crawler technology, ensure that each URL has corresponding image data, and assign a hash value to each URL as an index for the website screenshots; Extraction module, used to extract multimodal features: In this stage, multimodal features are extracted and the extraction strategies of text features and image features are explored in depth; The multimodal fusion module is used to extract multimodal features and then concatenate the text feature vector with the image feature vector; Iterative optimization module, used in the final prediction classification, the loss function is logarithmic loss; Classification module, used for phishing website classification.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the phishing website detection method based on multimodal fusion as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the phishing website detection method based on multimodal fusion as described in any one of claims 1 to 6.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the phishing website detection system based on multimodal fusion as described in claim 7.

Images