A fraud website detection method and device based on multimodal large-scale language model

Abstract

The present invention provides a fraudulent website detection method and device based on a large multimodal language model, relating to the technical field of network fraud detection. The method comprises: performing image processing on a webpage screenshot to obtain an enhanced screenshot and a first judgment basis; constructing a graph model based on HTML source code to obtain an HTML graph model; generating a second judgment basis based on the enhanced screenshot, a preset true label, and the first judgment basis based on a GPT-4o model; performing fraudulent website detection inference using the large multimodal model based on an adjustable temperature coefficient to obtain a fine-tuned multimodal model and a fifth judgment basis; constructing samples using a scoring model to obtain selected samples and rejected samples; using the selected samples and rejected samples to perform reinforcement learning training on the fine-tuned multimodal model, and using the optimized multimodal model to detect fraudulent websites. The present invention provides an efficient and accurate fraudulent website detection method based on a large multimodal language model.

Description

Technical Field

[0001] The present invention relates to the technical field of network fraud detection, and in particular to a fraud website detection method and device based on a multimodal large-scale language model. Background Art

[0002] In the field of online fraud detection, some research argues that character-level domain name features are key to distinguishing fraudulent websites from legitimate ones. Based on this insight, a large encoder-language model (LLLM) has been used to classify fraudulent websites. However, this approach is susceptible to evasive tactics used by fraudsters, highlighting the importance of analyzing actual website content. Artificial features based on website screenshots and Hypertext Markup Language (HTML) content have been designed to detect fraudulent websites. Integrating as many multimodal cues as possible is essential for accurately identifying fraudulent websites. With the advent of multimodal large language models (MLLMs), detection methods based on multimodal information have made significant progress.

[0003] However, the performance of current models for classifying different types of fraudulent websites has not been fully validated. Existing research primarily focuses on distinguishing specific categories, such as phishing, fraudulent e-commerce websites, and lottery scams. While there have been attempts to use ensemble machine learning to distinguish between various types of fraudulent websites, the lack of unified standards and publicly available datasets complicates performance evaluation.

[0004] The generalization ability of visual features in fraudulent website detection remains limited. Phishing websites are detected by extracting brand-related visual features. However, in practice, web crawlers often affect the quality of website screenshots, resulting in distortion, occlusion, or incomplete capture of web pages. Ensuring the robustness and generalization ability of models when dealing with such imperfect data is a key challenge.

[0005] The complex visual features of fraudulent websites pose significant challenges to model reasoning. While stacked machine learning techniques have been employed to address this, these approaches are prone to noise propagation. While the emergence of machine learning models like GPT-4o has explored leveraging the inherent reasoning capabilities of models to detect phishing websites, limitations of commercial application programming interfaces (APIs) have made large-scale validation difficult. Furthermore, while the anti-fraud network ScamNet improves interpretability by fine-tuning a large language model (LLM), this reasoning process increases the cost of expert annotation.

[0006] In the existing technology, there is a lack of an efficient and accurate fraud website detection method based on a multimodal large language model. Summary of the Invention

[0007] In order to solve the technical problem of the existing technology that fails to fully utilize website content, resulting in inaccurate fraud website detection results, the embodiment of the present invention provides a fraud website detection method and device based on a multimodal large-scale language model. The technical solution is as follows:

[0008] In one aspect, a fraudulent website detection method based on a multimodal large-scale language model is provided. The method is implemented by a fraudulent website detection device and includes:

[0009] Based on crawler technology, crawl data from fraudulent websites to obtain web page screenshots and HTML source code;

[0010] Performing image processing on the webpage screenshot to obtain an enhanced screenshot and a first judgment basis; constructing a graph model according to the HTML source code to obtain an HTML graph model;

[0011] Based on the GPT-4o model, the second judgment basis is obtained by using the Few-shot Image Prompting technology to generate the judgment basis according to the enhanced screenshot, the preset real label and the first judgment basis;

[0012] Based on the adjustable temperature coefficient, the multimodal large model is used to perform fraudulent website detection reasoning according to the enhanced screenshots, the preset true labels, and the second judgment basis, obtaining a fine-tuned multimodal large model and the fifth judgment basis;

[0013] Using the second judgment basis and the HTML graph model, the scoring model to be trained is optimized to obtain a scoring model;

[0014] Based on the enhanced screenshots and the preset true labels, the scoring model is used to construct samples according to the fifth judgment criterion and the HTML graph model to obtain selected samples and rejected samples;

[0015] Using selected samples and rejected samples, the fine-tuned multimodal large model is trained through reinforcement learning to obtain an optimized multimodal large model.

[0016] Obtain a screenshot and HTML source code of the web page to be detected; based on the screenshot and HTML source code of the web page to be detected, use the optimized multimodal large model to detect fraudulent websites.

[0017] On the other hand, a fraudulent website detection device based on a multimodal large-scale language model is provided. The device is applied to a fraudulent website detection method based on a multimodal large-scale language model. The device includes:

[0018] The data acquisition module is used to crawl data from fraudulent websites based on crawler technology, and obtain web page screenshots and HTML source code;

[0019] The data processing module is used to perform image processing on the webpage screenshot to obtain an enhanced screenshot and a first judgment basis; construct a graph model according to the HTML source code to obtain an HTML graph model;

[0020] The basic judgment basis generation module is used to generate the judgment basis based on the GPT-4o model, the enhanced screenshot, the preset real label and the first judgment basis, and the Few-shot Image Prompting technology to obtain the second judgment basis;

[0021] An enhanced judgment basis generation module is used to use the multimodal large model to perform fraudulent website detection reasoning based on the enhanced screenshot, the preset true label, and the second judgment basis based on the adjustable temperature coefficient, to obtain a fine-tuned multimodal large model and the fifth judgment basis;

[0022] The scoring model optimization module is used to optimize the scoring model to be trained using the second judgment basis and the HTML graph model to obtain a scoring model;

[0023] A sample construction module is used to construct samples based on the enhanced screenshots and preset true labels, according to the fifth judgment basis and the HTML graph model, through the scoring model to obtain selected samples and rejected samples;

[0024] The large model reinforcement learning module is used to use selected samples and rejected samples to perform reinforcement learning training on the fine-tuned multimodal large model to obtain an optimized multimodal large model;

[0025] The fraudulent website detection module is used to obtain screenshots and HTML source code of the web page to be detected; based on the screenshots and HTML source code of the web page to be detected, the optimized multimodal large model is used to detect fraudulent websites.

[0026] On the other hand, a fraudulent website detection device is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned fraudulent website detection methods based on a multimodal large language model is implemented.

[0027] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned fraudulent website detection methods based on a multimodal large language model.

[0028] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0029] This paper proposes a fraudulent website detection method based on a large, multimodal language model. This method enhances the MLLM's detection capabilities through long-chain reasoning and self-improvement. The model is guided to output the correct reasoning process using a small number of examples and expert reasoning logic. A scoring model is trained by combining graph information extracted from HTML with the reasoning process. Based on the diverse judgment criteria generated by the MLLM, the most accurate judgment criteria are selected to optimize the model's reasoning process and improve classification accuracy. This presents an efficient and accurate fraudulent website detection method based on a large, multimodal language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 This is a flow chart of a fraudulent website detection method based on a multimodal large-scale language model provided by an embodiment of the present invention;

[0032] Figure 2 This is a block diagram of a fraudulent website detection device based on a multimodal large-scale language model provided by an embodiment of the present invention;

[0033] Figure 3 The figure is a structural diagram of a fraudulent website detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0035] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0036] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0037] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0038] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0039] The embodiment of the present invention provides a fraudulent website detection method based on a multimodal large language model. The method can be implemented by a fraudulent website detection device, which can be a terminal or a server. Figure 1 The flowchart of the fraud website detection method based on a multimodal large language model is shown. The processing flow of the method may include the following steps:

[0040] S1. Based on crawler technology, crawl data from fraudulent websites to obtain web page screenshots and HTML source code.

[0041] In one feasible implementation, the present invention obtains training information from the official website of the Global Anti-Scam Organization. Web crawling technology is used to obtain webpage screenshots and HTML source code corresponding to these domain names. All captured webpage screenshots are preprocessed to remove invalid, blank, and duplicate images, ensuring that each image corresponds to its corresponding HTML source code.

[0042] S2. Perform image processing on the webpage screenshot to obtain an enhanced screenshot and a first judgment basis; construct a graph model according to the HTML source code to obtain an HTML graph model.

[0043] Optionally, performing image processing on the webpage screenshot to obtain an enhanced screenshot and a first judgment basis includes:

[0044] Perform image preprocessing on the webpage screenshot to obtain the processed screenshot;

[0045] Performing image transformation on the processed screenshot to obtain an enhanced screenshot;

[0046] Mark the enhanced screenshots to obtain the first basis for judgment.

[0047] In one feasible implementation, all webpage screenshots are cropped, masked, and stretched for data augmentation, generating more samples. These are annotated by experts and categorized into eight categories: gambling, pornography, fake order fraud, phishing, malicious distribution, fake financial services, fake shopping mall services, and legitimate domain names.

[0048] Among them, the nodes and edges of the HTML graph model; the nodes include domain name nodes, URL nodes and IP address nodes; the edges include: inclusion relationship edges, resolution relationship edges and hosting relationship edges.

[0049] In one feasible implementation, the corresponding information is extracted from the website HTML to construct a knowledge graph, wherein the URL is extracted from the HTML to construct the Domain-Contain-URL relationship; the domain name / IP in the URL is extracted to construct the URL-Host-Domain / IP relationship; and the Domain-Resolve-IP relationship is constructed through DNS resolution. Finally, a graph model is formed. , node type , edge type , are the characteristics of different types of nodes.

[0050] S3. Based on the GPT-4o model, the Few-shot Image Prompting technology is used to generate the judgment basis according to the enhanced screenshot, the preset real label and the first judgment basis to obtain the second judgment basis.

[0051] In one feasible implementation, a small number of expert examples are used to guide the MLLM to generate the correct judgment basis. Given a training set containing N samples ,in, is a screenshot of the i-th domain name, is a fraud label. Select k cross-category samples as demonstration samples ,in . Through natural language templates Take a screenshot ,Label ,hint Judgment basis with expert annotation Convert to a format acceptable to MLLM. Final splicing formation prompt , so that the MLLM output is reasonable second judgment basis .

[0052] S4. Based on the adjustable temperature coefficient, according to the enhanced screenshots, the preset true labels and the second judgment basis, the multimodal large model is used to perform fraudulent website detection reasoning to obtain the fine-tuned multimodal large model and the fifth judgment basis.

[0053] Optionally, based on the adjustable temperature coefficient, according to the enhanced screenshot, the preset true label, and the second judgment basis, a multimodal large model is used to perform fraudulent website detection reasoning to obtain a fine-tuned multimodal large model and a fifth judgment basis, including:

[0054] Fine-tune the multimodal large model based on the enhanced screenshots, the preset true labels, and the second judgment basis to obtain a fine-tuned multimodal large model;

[0055] Based on the adjustable temperature coefficient, the enhanced screenshots and the preset true labels are used to perform fraudulent website detection reasoning using a parameter-frozen multimodal large model to obtain a third-party judgment basis.

[0056] Based on the adjustable temperature coefficient and the enhanced screenshots, a fine-tuned multimodal large model is used to perform fraudulent website detection reasoning, obtaining the fourth judgment basis;

[0057] The fifth judgment basis is obtained based on the third judgment basis and the fourth judgment basis.

[0058] In a feasible implementation, the present invention divides the fraud domain name classification task into two sub-problems: 1 (sq1) requires the model to directly determine whether the website is fraudulent, and its text prompt Contains question text (Q), visual information (V) and binary options (B), output Sub-question 2 (sq2) requires distinguishing specific fraud types based on website content. Contains multiple options (M), and the output must include the selection and judgment basis. Output of sq2 By original label and reasonable judgment basis for MLLM output Composition, using question-answer pairs ( , )and( , ) Fine-tune the multimodal large model (Qwen2-vl-2b) with a small number of parameters to obtain the multimodal large model after fine-tuning the model and The corresponding loss functions are as follows (1) and (2):

[0059] (1);

[0060] (2);

[0061] in, , the loss of Sq2 is to calculate the probability of generating the target text R (length L) .

[0062] S5. Use the second judgment basis and the HTML graph model to optimize the scoring model to be trained to obtain a scoring model.

[0063] Among them, the scoring model includes input layer, fusion layer and output layer;

[0064] The input layer includes a judgment basis encoder and a graph model encoder; the judgment basis encoder is built based on the pre-trained BERT model; the graph model encoder is built based on the graph neural network model;

[0065] The output layer is a fully connected layer based on the neural network structure.

[0066] In one feasible implementation, the scoring model It includes input layer, fusion layer and output layer; in the input layer, the judgment basis encoder uses the DomURL_Bert model to encode the string; and uses the graph neural network to encode the graph model; in the fusion layer, feature fusion is performed based on the cross-attention mechanism; the output layer uses a fully connected layer with Logit-Norm technology to prevent overfitting of the final generated result.

[0067] S6. Based on the enhanced screenshot and the preset true label, according to the fifth judgment basis and the HTML graph model, the sample is constructed through the scoring model to obtain the selected sample and the rejected sample.

[0068] Optionally, based on the enhanced screenshot and the preset true label, according to the fifth judgment criterion and the HTML graph model, a scoring model is used to construct samples to obtain selected samples and rejected samples, including:

[0069] Based on the fifth judgment basis and the HTML graph model, the scoring model is used to perform fraudulent website detection reasoning to obtain a predicted probability score for the fraudulent website;

[0070] Based on the enhanced screenshot and the preset true label, according to the fifth judgment criterion, the judgment criterion corresponding to the maximum value of the predicted probability score is selected to construct the selection sample;

[0071] Based on the enhanced screenshot and the preset true label, according to the fifth judgment criterion, the judgment criterion corresponding to the minimum value of the predicted probability score is selected to construct a rejection sample.

[0072] In one feasible implementation, the fine-tuned multimodal model is used to re-infer the training set S, and the temperature parameters are adjusted to generate diversified inferences. To prevent error accumulation, the multimodal model with frozen parameters ( ) to generate correct inferences. By scoring the model Calculate the correct category of each inference The probability under which the sample with the highest probability is selected as the adopted sample ( ), the lowest is the rejection sample ( ) and conduct reinforcement learning.

[0073] The goal is to use the reward model to calculate the probability that the inference text is correctly classified by selecting the correct response y w and reject the error response y l , to improve the fine-tuned multimodal large model π θ The performance of the algorithm is to generate a more reasonable reasoning process. The corresponding loss function is as follows (3):

[0074] (3);

[0075] in, Represents the expected value of all samples in the training dataset D. It is used to summarize the loss over the entire dataset in order to optimize the strategy as a whole during training. Represents an activation function, usually a sigmoid function, which is used to map the input to the (0, 1) interval and regard it as a probability value. Is a temperature parameter that controls how conservative the policy update is. Larger values of β make the policy update more cautious, while smaller values allow for more aggressive updates.

[0076] Optionally, based on the fifth judgment criterion and the HTML graph model, a scoring model is used to perform fraudulent website detection reasoning to obtain a predicted probability score for the fraudulent website, including:

[0077] Based on the fifth judgment basis, use the pre-trained BERT model to extract features and obtain judgment basis features;

[0078] According to the HTML graph model, use the graph neural network to extract features and obtain graph model features;

[0079] Based on the cross-attention mechanism, feature fusion is performed according to the judgment basis features and the graph model features to obtain the fused features;

[0080] The fused features are input into the fully connected layer to predict the probability of fraudulent websites and obtain a preliminary prediction probability score;

[0081] Based on the preliminary predicted probability scores, the Logit-Norm technique is used for smoothing to obtain the predicted probability scores.

[0082] In a feasible implementation, the present invention combines the information of the graph model G and the MLLM output of the fifth judgment basis The scoring model was trained.

[0083] For domain name and URL type nodes, the DomURL_Bert model is used to encode the strings. The DomURL_Bert model is used because it has been pre-trained on a large number of domain names and URLs and can better capture similar patterns between different domain names or URL strings. For IP type nodes, the corresponding Autonomous System Number (ASN), operator, location, latitude and longitude, and other information are queried through IP info. The corresponding IP description is generated through a natural language template and then encoded using the Bidirectional Encoder Representations from Transformers (BERT) model.

[0084] A specially designed graph neural network is used to encode the graph model G. In this process, the node influence is calculated through a hierarchical attention mechanism. The calculation process is as follows (4) and (5):

[0085] (4);

[0086] (5);

[0087] in, is the characteristic of neighbor node j, is the characteristic of node i itself, is the set of neighbor nodes of node i, the edge type is r, r∈R, R is the set of all edge types; is the linear transformation matrix, ||| represents the splicing operation, is the dimension adjustment matrix. The final node-based attention score It is calculated by performing a Softmax operation on the information after using the LeakyReLu activation function in the fully connected layer.

[0088] Graph information is obtained by integrating information from different types of neighbors. The process is as follows:

[0089] (6);

[0090] (7);

[0091] (8);

[0092] in is the node indegree, is the linear transformation matrix corresponding to the relationship r, is a numerical stability term; ReLu is an activation function; for the fifth judgment basis of MLLM output , encoded using BERT to obtain , and adopt gated fusion mechanism to and h i Fusion is performed to calculate the contribution of text and image to the final reward score. The fused feature vector Obtained by the following formulas (9), (10) and (11):

[0093] (9);

[0094] (10)

[0095] (11);

[0096] in, and is a learnable parameter; For the splicing operation, sigmoid is the activation function to ensure The value is between 0-1, and the fusion feature is finally obtained .

[0097] Will The linear layer reduces the dimension to 7 dimensions corresponding to the 7 fraud types, and the Softmax function is used to obtain the classification score logits. However, using cross-entropy loss can easily lead to the problem of overconfidence in the model, making the score lack of discrimination, and may give a particularly large score to a certain type, while other types are too small. Therefore, a regularized loss function is used to constrain the model and require the model to output a smooth score. The loss function is as follows (12):

[0098] (12);

[0099] in, is the model parameter of the output layer, f represents logits, f y represents the yth element of logits corresponding to the true label y, k represents the dimension of logits, is the L2 norm of the logits.

[0100] S7. Use selected samples and rejected samples to perform reinforcement learning training on the fine-tuned multimodal large model to obtain an optimized multimodal large model.

[0101] In one feasible implementation, selected samples are high-quality data that can significantly improve model performance. They provide the model with rich learning signals and help it better understand the requirements of complex tasks. Rejected samples are data that may introduce noise or mislead the model. By excluding these samples, the model can effectively prevent learning incorrect patterns during training.

[0102] This combined selection and rejection strategy allows the model to focus more on valuable information during training, gradually optimizing its parameters and structure. After multiple rounds of reinforcement learning training, the resulting optimized multimodal large model demonstrates greater accuracy and robustness when handling a variety of complex tasks.

[0103] S8. Obtain a screenshot and HTML source code of the web page to be detected; and use the optimized multimodal large model to detect fraudulent websites based on the screenshot and HTML source code of the web page to be detected.

[0104] In a feasible implementation, the present invention organically combines expert knowledge, graph structure information and self-supervised learning through a phased optimization strategy, effectively improving the reasoning ability and classification accuracy of MLLM in the fraudulent website detection task.

[0105] This paper proposes a fraudulent website detection method based on a large, multimodal language model. This method enhances the MLLM's detection capabilities through long-chain reasoning and self-improvement. The model is guided to output the correct reasoning process using a small number of examples and expert reasoning logic. A scoring model is trained by combining graph information extracted from HTML with the reasoning process. Based on the diverse judgment criteria generated by the MLLM, the most accurate judgment criteria are selected to optimize the model's reasoning process and improve classification accuracy. This presents an efficient and accurate fraudulent website detection method based on a large, multimodal language model.

[0106] Figure 2 This is a block diagram of a fraud website detection device based on a multimodal large language model according to an exemplary embodiment. The device is used in a fraud website detection method based on a multimodal large language model. Figure 2 The device includes a data acquisition module 210, a data processing module 220, a basic judgment basis generation module 230, an enhanced judgment basis generation module 240, a scoring model optimization module 250, a sample construction module 260, a large model reinforcement learning module 270, and a fraudulent website detection module 280. Among them:

[0107] The data acquisition module 210 is used to crawl the fraudulent website based on crawler technology to obtain web page screenshots and HTML source code;

[0108] The data processing module 220 is used to perform image processing on the webpage screenshot to obtain an enhanced screenshot and a first judgment basis; construct a graph model according to the HTML source code to obtain an HTML graph model;

[0109] A basic judgment basis generation module 230 is configured to generate a judgment basis based on the GPT-4o model, the enhanced screenshot, the preset true label, and the first judgment basis using the Few-shot Image Prompting technology to obtain a second judgment basis;

[0110] Enhanced judgment basis generation module 240 is configured to use the multimodal large model to perform fraudulent website detection reasoning based on the adjustable temperature coefficient, the enhanced screenshot, the preset true label, and the second judgment basis, to obtain a fine-tuned multimodal large model and a fifth judgment basis;

[0111] The scoring model optimization module 250 is configured to optimize the scoring model to be trained using the second judgment basis and the HTML graph model to obtain a scoring model;

[0112] A sample construction module 260 is configured to construct samples based on the enhanced screenshots and preset true labels, according to the fifth judgment criterion and the HTML graph model, and through a scoring model to obtain selected samples and rejected samples;

[0113] A large model reinforcement learning module 270 is used to perform reinforcement learning training on the fine-tuned multimodal large model using selected samples and rejected samples to obtain an optimized multimodal large model;

[0114] The fraudulent website detection module 280 is used to obtain a screenshot and HTML source code of the web page to be detected; based on the screenshot and HTML source code of the web page to be detected, the optimized multimodal large model is used to detect fraudulent websites.

[0115] Optionally, the data processing module 220 is further configured to:

[0116] Perform image preprocessing on the webpage screenshot to obtain the processed screenshot;

[0117] Performing image transformation on the processed screenshot to obtain an enhanced screenshot;

[0118] Mark the enhanced screenshots to obtain the first basis for judgment.

[0119] Among them, the nodes and edges of the html graph model; the nodes include domain name nodes, URL nodes and IP address nodes; the edges include: inclusion relationship edges, resolution relationship edges and hosting relationship edges.

[0120] Optionally, the enhanced judgment basis generating module 240 is further configured to:

[0121] Fine-tune the multimodal large model based on the enhanced screenshots, the preset true labels, and the second judgment basis to obtain a fine-tuned multimodal large model;

[0122] Based on the adjustable temperature coefficient, the enhanced screenshots and the preset true labels are used to perform fraudulent website detection reasoning using a parameter-frozen multimodal large model to obtain a third-party judgment basis.

[0123] Based on the adjustable temperature coefficient and the enhanced screenshots, a fine-tuned multimodal large model is used to perform fraudulent website detection reasoning, obtaining the fourth judgment basis;

[0124] The fifth judgment basis is obtained based on the third judgment basis and the fourth judgment basis.

[0125] Among them, the scoring model includes input layer, fusion layer and output layer;

[0126] The input layer includes a judgment basis encoder and a graph model encoder; the judgment basis encoder is built based on the pre-trained BERT model; the graph model encoder is built based on the graph neural network model;

[0127] The output layer is a fully connected layer based on the neural network structure.

[0128] Optionally, the sample construction module 260 is further configured to:

[0129] Based on the fifth judgment basis and the HTML graph model, the scoring model is used to perform fraudulent website detection reasoning to obtain a predicted probability score for the fraudulent website;

[0130] Based on the enhanced screenshot and the preset true label, according to the fifth judgment criterion, the judgment criterion corresponding to the maximum value of the predicted probability score is selected to construct the selection sample;

[0131] Based on the enhanced screenshot and the preset true label, according to the fifth judgment criterion, the judgment criterion corresponding to the minimum value of the predicted probability score is selected to construct a rejection sample.

[0132] Optionally, the sample construction module 260 is further configured to:

[0133] Based on the fifth judgment basis, use the pre-trained BERT model to extract features and obtain judgment basis features;

[0134] According to the HTML graph model, use the graph neural network to extract features and obtain graph model features;

[0135] Based on the cross-attention mechanism, feature fusion is performed according to the judgment basis features and the graph model features to obtain the fused features;

[0136] The fused features are input into the fully connected layer to predict the probability of fraudulent websites and obtain a preliminary prediction probability score;

[0137] Based on the preliminary predicted probability scores, the Logit-Norm technique is used for smoothing to obtain the predicted probability scores.

[0138] This paper proposes a fraudulent website detection method based on a large, multimodal language model. This method enhances the MLLM's detection capabilities through long-chain reasoning and self-improvement. The model is guided to output the correct reasoning process using a small number of examples and expert reasoning logic. A scoring model is trained by combining graph information extracted from HTML with the reasoning process. Based on the diverse judgment criteria generated by the MLLM, the most accurate judgment criteria are selected to optimize the model's reasoning process and improve classification accuracy. This presents an efficient and accurate fraudulent website detection method based on a large, multimodal language model.

[0139] Figure 3 This is a schematic diagram of the structure of a fraudulent website detection device provided by an embodiment of the present invention. Figure 3 As shown, the fraud website detection device may include the above Figure 2 The fraudulent website detection device based on a multimodal large language model is shown. Optionally, the fraudulent website detection device 310 may include a first processor 2001.

[0140] Optionally, the fraudulent website detection device 310 may further include a memory 2002 and a transceiver 2003 .

[0141] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0142] The following combination Figure 3 The following describes the various components of the fraudulent website detection device 310:

[0143] The first processor 2001 is the control center of the fraudulent website detection device 310 and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0144] Optionally, the first processor 2001 can perform various functions of the fraudulent website detection device 310 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0145] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in FIG.

[0146] In a specific implementation, as an embodiment, the fraudulent website detection device 310 may also include multiple processors, such as Figure 3 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0147] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0148] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently and accessed through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0149] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0150] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 3 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0151] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and be connected to the fraudulent website detection device 310 through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0152] It should be noted that Figure 3 The structure of the fraudulent website detection device 310 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0153] In addition, the technical effects of the fraudulent website detection device 310 can refer to the technical effects of the fraudulent website detection method based on a multimodal large language model described in the above method embodiment, and will not be repeated here.

[0154] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0155] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0156] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0157] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0158] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0159] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0160] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0161] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0162] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0163] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0164] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0165] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical disks.

[0166] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A fraudulent website detection method based on a multimodal large-scale language model, characterized in that: The method comprises: Based on crawler technology, crawl data from fraudulent websites to obtain web page screenshots and HTML source code; Performing image processing on the webpage screenshot to obtain an enhanced screenshot and a first judgment basis; constructing a graph model according to the HTML source code to obtain an HTML graph model; Based on the GPT-4o model, the Few-shot Image Prompting technology is used to generate the judgment basis according to the enhanced screenshot, the preset real label and the first judgment basis to obtain the second judgment basis; Based on the adjustable temperature coefficient, the multimodal large model is used to perform fraudulent website detection reasoning according to the enhanced screenshots, the preset true labels, and the second judgment basis, obtaining a fine-tuned multimodal large model and the fifth judgment basis; Using the second judgment basis and the HTML graph model, the scoring model to be trained is optimized to obtain a scoring model; Based on the enhanced screenshots and the preset true labels, the scoring model is used to construct samples according to the fifth judgment criterion and the HTML graph model to obtain selected samples and rejected samples; Using selected samples and rejected samples, the fine-tuned multimodal large model is trained through reinforcement learning to obtain an optimized multimodal large model. Obtain a screenshot and HTML source code of the web page to be detected; based on the screenshot and HTML source code of the web page to be detected, use the optimized multimodal large model to detect fraudulent websites.

2. The fraudulent website detection method based on a multimodal large language model according to claim 1 is characterized in that: The performing image processing on the webpage screenshot to obtain the enhanced screenshot and the first judgment basis includes: Perform image preprocessing on the webpage screenshot to obtain the processed screenshot; Performing image transformation on the processed screenshot to obtain an enhanced screenshot; Mark the enhanced screenshots to obtain the first basis for judgment.

3. The fraudulent website detection method based on a multimodal large language model according to claim 1 is characterized in that: The nodes and edges of the HTML graph model; the nodes include domain name nodes, URL nodes and IP address nodes; the edges include: inclusion relationship edges, resolution relationship edges and hosting relationship edges.

4. The fraudulent website detection method based on a multimodal large language model according to claim 1 is characterized in that: Based on the adjustable temperature coefficient, the multimodal large model is used to perform fraudulent website detection reasoning according to the enhanced screenshot, the preset real label, and the second judgment basis, to obtain a fine-tuned multimodal large model and the fifth judgment basis, including: Fine-tune the multimodal large model based on the enhanced screenshots, the preset true labels, and the second judgment basis to obtain a fine-tuned multimodal large model; Based on the adjustable temperature coefficient, the enhanced screenshots and the preset true labels are used to perform fraudulent website detection reasoning using a parameter-frozen multimodal large model to obtain a third-party judgment basis. Based on the adjustable temperature coefficient and the enhanced screenshots, a fine-tuned multimodal large model is used to perform fraudulent website detection reasoning, obtaining the fourth judgment basis; The fifth judgment basis is obtained based on the third judgment basis and the fourth judgment basis.

5. The fraudulent website detection method based on a multimodal large language model according to claim 1 is characterized in that: The scoring model includes an input layer, a fusion layer and an output layer; The input layer includes a judgment basis encoder and a graph model encoder; the judgment basis encoder is constructed based on the pre-trained BERT model; the graph model encoder is constructed based on the graph neural network model; The output layer is a fully connected layer based on a neural network structure.

6. The fraudulent website detection method based on a multimodal large language model according to claim 1, characterized in that: The method of constructing samples based on the enhanced screenshots and the preset true labels, according to the fifth judgment criterion and the HTML graph model, and obtaining selected samples and rejected samples through the scoring model includes: Based on the fifth judgment basis and the HTML graph model, the scoring model is used to perform fraudulent website detection reasoning to obtain a predicted probability score for the fraudulent website; Based on the enhanced screenshot and the preset true label, according to the fifth judgment criterion, the judgment criterion corresponding to the maximum value of the predicted probability score is selected to construct the selection sample; Based on the enhanced screenshot and the preset true label, according to the fifth judgment criterion, the judgment criterion corresponding to the minimum value of the predicted probability score is selected to construct a rejection sample.

7. The fraudulent website detection method based on a multimodal large language model according to claim 6, characterized in that: The method of using the scoring model to perform fraudulent website detection reasoning based on the fifth judgment criterion and the HTML graph model to obtain a predicted probability score of the fraudulent website includes: Based on the fifth judgment basis, use the pre-trained BERT model to extract features and obtain judgment basis features; According to the HTML graph model, use the graph neural network to extract features and obtain graph model features; Based on the cross-attention mechanism, feature fusion is performed according to the judgment basis features and the graph model features to obtain the fused features; The fused features are input into the fully connected layer to predict the probability of fraudulent websites and obtain a preliminary prediction probability score; Based on the preliminary predicted probability scores, the Logit-Norm technique is used for smoothing to obtain the predicted probability scores.

8. A fraudulent website detection device based on a multimodal large-scale language model, wherein the fraudulent website detection device based on a multimodal large-scale language model is used to implement the fraudulent website detection method based on a multimodal large-scale language model as described in any one of claims 1 to 7, characterized in that: The device comprises: The data acquisition module is used to crawl data from fraudulent websites based on crawler technology, and obtain web page screenshots and HTML source code; The data processing module is used to perform image processing on the webpage screenshot to obtain an enhanced screenshot and a first judgment basis; construct a graph model according to the HTML source code to obtain an HTML graph model; The basic judgment basis generation module is used to generate the judgment basis based on the GPT-4o model, the enhanced screenshot, the preset real label and the first judgment basis, and the Few-shot Image Prompting technology to obtain the second judgment basis; An enhanced judgment basis generation module is used to use the multimodal large model to perform fraudulent website detection reasoning based on the enhanced screenshot, the preset true label, and the second judgment basis based on the adjustable temperature coefficient, to obtain a fine-tuned multimodal large model and the fifth judgment basis; The scoring model optimization module is used to optimize the scoring model to be trained using the second judgment basis and the HTML graph model to obtain a scoring model; A sample construction module is used to construct samples based on the enhanced screenshots and preset true labels, according to the fifth judgment basis and the HTML graph model, through the scoring model to obtain selected samples and rejected samples; The large model reinforcement learning module is used to use selected samples and rejected samples to perform reinforcement learning training on the fine-tuned multimodal large model to obtain an optimized multimodal large model; The fraudulent website detection module is used to obtain screenshots and HTML source code of the web page to be detected; based on the screenshots and HTML source code of the web page to be detected, the optimized multimodal large model is used to detect fraudulent websites.

9. A fraud website detection device, characterized in that: The fraud website detection device includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images