Method and apparatus for identifying a risky website

Abstract

The application provides a method and device for identifying a risk website. The method comprises: acquiring a multi-modal feature of a website to be identified, the multi-modal feature comprising an image feature and a corresponding text feature; encoding the image feature to obtain an image encoding vector; inputting the image encoding vector into a connector to obtain an optimized image encoding vector; encoding the text feature to obtain a text encoding vector; fusing the optimized image encoding vector and the text encoding vector to obtain a fused vector; and inputting the fused vector into a multi-modal large model for identifying a risk website to determine whether the website to be identified is a risk website.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and in particular to a method and device for identifying risky websites. Background Art

[0002] With the rapid development of the Internet, the number of risky websites in areas such as e-commerce has increased dramatically. These risky websites can cause huge economic losses, so risk identification of website threats is crucial.

[0003] Currently, website identification is typically performed manually, requiring personnel to identify information such as website URLs and webpage content. However, due to the large number of websites, this method requires significant labor costs. Furthermore, due to the varying levels of human identification skills, identification efficiency is low and the error rate is high.

[0004] When using traditional machine learning methods to identify risky websites, due to the constant changes and high complexity of the network environment, the machine learning model cannot be trained and updated in a timely manner, so it often has defects such as low accuracy and poor generalization.

[0005] Therefore, there is a need for improved methods and apparatus for identifying risky websites. Summary of the Invention

[0006] This application aims to provide an efficient and reliable solution for identifying risky websites.

[0007] According to a first aspect, the present application provides a method for identifying risky websites, comprising: obtaining multimodal features of a website to be identified, the multimodal features including image features and corresponding text features; encoding the image features to obtain an image encoding vector; inputting the image encoding vector into a connector to obtain an optimized image encoding vector; encoding the text features to obtain a text encoding vector; fusing the optimized image encoding vector and the text encoding vector to obtain a fused vector; and inputting the fused vector into a multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website.

[0008] According to a second aspect, the present application provides an apparatus for identifying risky websites, comprising: a processor configured to execute the method according to the first aspect.

[0009] According to a third aspect, the present application provides an apparatus for identifying risky websites, comprising: a component for executing the method according to the first aspect.

[0010] According to a fourth aspect, the present application provides a computer-readable storage medium storing instructions, which, when executed by a processor, causes the processor to perform the method according to the first aspect.

[0011] According to a fifth aspect, the present application provides a computer program product storing instructions which, when executed by a processor, cause the processor to perform the method according to the first aspect.

[0012] According to the sixth aspect, the present application provides a device for identifying risky websites, including: an acquisition module for acquiring multimodal features of a website to be identified, the multimodal features including image features and corresponding text features; an image encoding module for encoding image features to obtain an image encoding vector; an optimized image encoding module for inputting the image encoding vector into a connector to obtain an optimized image encoding vector; a text encoding module for encoding text features to obtain a text encoding vector; a fusion module for fusing the optimized image encoding vector and the text encoding vector to obtain a fused vector; and a risky website identification module for inputting the fused vector into a multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website.

[0013] This application utilizes a large multimodal model for identifying risky websites, combined with multiple data inputs such as text and images, to accurately classify and identify websites, thereby improving the accuracy and efficiency of identification and reducing labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] A more thorough understanding of the present application may be obtained by referring to the following detailed description of specific embodiments in conjunction with the following drawings.

[0015] Figure 1 is a flowchart of a method for identifying risky websites according to some embodiments of the present application.

[0016] Figure 2 is a block diagram of a connector according to some embodiments of the present application.

[0017] Figure 3 is a block diagram of an apparatus for identifying risky websites according to some embodiments of the present application.

[0018] Figure 4 is a block diagram of an apparatus for identifying risky websites according to some embodiments of the present application. DETAILED DESCRIPTION

[0019] The concepts of the present application will be described below with reference to the accompanying drawings, in which some embodiments of the concepts of the present application are shown. However, the concepts of the present application can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

[0020] In general, all terms used in this application are to be interpreted according to their ordinary meaning in the art, unless otherwise expressly defined herein. All references to "a / an / the element, device, component, part, step, etc." are to be interpreted openly as referring to at least one instance of the element, device, component, part, step, etc., unless expressly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless expressly stated otherwise.

[0021] This application provides an efficient and reliable solution for identifying risky websites. This solution utilizes a large multimodal model for identifying risky websites, combining multiple data inputs such as text and images to accurately classify and identify websites, thereby improving identification accuracy and efficiency and reducing labor costs.

[0022] Figure 1 is a flowchart of a method for identifying risky websites according to some embodiments of the present application.

[0023] Step S110: Acquire multimodal features of the website to be identified, where the multimodal features include image features and corresponding text features.

[0024] In one embodiment of the present application, a website information feature extraction method is used to obtain image features and corresponding text features of the website to be identified. This website information feature extraction method automatically extracts information features from websites with a specified domain name. For example, information feature extraction can be performed using internet data crawling technology. The extracted information features primarily include two types of features: the overall page image of the website, which serves as image features; and the website's main core keywords, which serve as text features.

[0025] This website information feature extraction method includes the following steps: analyzing the website URL information input by the user, automatically intercepting the website's information code through a browser-simulated access method to obtain the website's source code, and automatically saving screenshots to save the entire website page; then, using a keyword extraction algorithm to extract high-frequency keywords from the website's core code, using them as text features to represent the website's text information, and using the automatically saved screenshot images as image features to represent the website's image information. The image features and corresponding text features constitute the image-text pairs of the website's information features.

[0026] Step S120: Encode the image features to obtain an image encoding vector. The image encoding vector is a vector that can be operated on and can spatially represent the distribution of image features in a multi-dimensional space.

[0027] In some embodiments of the present application, an image encoder is used to encode image features. An example of an image encoder is a pre-trained image encoding model. This image encoding model can use various neural network architectures, such as CLIP (Contrastive Language-Image Pre-Training). The input information of this image encoding model is image features, and the output information is a multi-dimensional vector that can be operated on.

[0028] Step S130: Input the image coding vector into the connector to obtain an optimized image coding vector. The optimized image coding vector can better represent the image information of the website.

[0029] In some embodiments of the present application, the connector is trained based on training data, which includes multimodal general domain training data and multimodal risky website training data.

[0030] Step S140: Encode the text features to obtain a text encoding vector. The text encoding vector is a vector that can be operated on and can spatially represent the distribution of text features in a multi-dimensional space.

[0031] In some embodiments of the present application, a text encoder is used to encode text features. An example of a text encoder is a pre-trained vocabulary of multidimensional vectors. This vocabulary can be used to convert tokens in the text into multidimensional vectors. The vector length of each token is a fixed number of dimensions. For example, it can be assumed that the vector length of each token is 768 dimensions. It will be understood that the vector length of each token is not limited to 768 dimensions, but can be other values.

[0032] Step S150: Fusing the optimized image coding vector and the text coding vector to obtain a fused vector.

[0033] In some embodiments of the present application, a fused vector is obtained by concatenating an optimized image encoding vector and a text encoding vector.

[0034] Step S160: Input the fused vector into a multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website.

[0035] In some embodiments of the present application, a multimodal large model for identifying risky websites is trained based on training data, and the training data includes multimodal general domain training data and multimodal risky website training data.

[0036] According to some embodiments of the present application, the multimodal large model for identifying risky websites is capable of two types of identification: website type identification and website content identification. Website type identification is used to identify whether a website belongs to a high-risk website type or a general website type. If the business scenario of the website involves monetary transactions, such as e-commerce, such websites can be marked as belonging to a high-risk website type. The remaining websites belong to the general website type, that is, non-high-risk website types, for example, they can be marked as "others". Website content identification is performed through the image and language understanding capabilities of the multimodal large model for identifying risky websites, for example, identifying misleading sentences, typos, common sense errors, and the design style of the website.

[0037] Based on the identified website type and content, as well as pre-set judgment rules, a comprehensive assessment of the website's risk type and risk level can be conducted. For example, a weighted sum of the identified website type and content can be used to calculate a comprehensive score, and the risk level can be determined based on the comprehensive score. Subsequent monitoring and early warning work can be carried out based on the determined risk level.

[0038] From the perspective of precise classification, the multimodal large-scale model used in this application to identify risky websites can accurately classify websites by comprehensively analyzing keywords and semantic information contained in text content, as well as visual elements and layout features in images. Whether a website contains textual hints of malicious code or a website that spreads harmful information through images, it can be accurately classified into the corresponding category, greatly improving classification accuracy.

[0039] In terms of identification efficiency, the multimodal large-scale model for identifying risky websites in this application achieves rapid data processing and analysis thanks to its advanced algorithms and architecture. Compared to traditional single-modal identification methods, it eliminates the need to process different types of data separately, reducing the complexity and time consumption of the processing flow. This means that even with large amounts of website data, risky websites can be quickly and accurately identified, effectively saving time and costs.

[0040] Furthermore, in terms of cost control, the multimodal large-scale model used in this application for identifying risky websites significantly reduces labor costs. Previously, identifying risky websites often required extensive manual effort, with professionals spending considerable time reviewing text and analyzing images, a cumbersome and error-prone process. However, the multimodal large-scale model used in this application for identifying risky websites replaces much of this manual work, requiring only minimal manual supervision and maintenance, significantly reducing labor costs and improving overall economic benefits.

[0041] As described above, according to some embodiments of the present application, both the connector and the multimodal large model for identifying risky websites are trained based on training data, including multimodal general domain training data and multimodal risky website training data. The following describes the construction of training data, the training of the connector, and the training of the multimodal large model for identifying risky websites.

[0042] Construction of training data

[0043] According to some embodiments of the present application, the training data is used to train the connector and the multimodal large model for identifying risky websites, and may include multimodal general domain training data and multimodal risky website training data.

[0044] The main purpose of multimodal general domain training data is to inject extensive world knowledge into the multimodal large model used to identify risky websites, and to improve the generalization ability, zero-sample or few-sample ability of the multimodal large model used to identify risky websites. It can be divided into two categories: coarse-grained multimodal general domain training data and fine-grained multimodal general domain training data.

[0045] Coarse-grained multimodal general-domain training data can be based on internet data. This internet-based data is large in volume and can include image data and corresponding text data from various scenarios. This image data and text data can typically be directly obtained through internet data crawling techniques. For example, video images and their corresponding subtitles, pictures and their corresponding news descriptions, websites and their corresponding website descriptions, etc. Coarse-grained multimodal general-domain training data can be obtained by preprocessing this image data and corresponding text data.

[0046] According to some embodiments of the present application, preprocessing may include image preprocessing, text preprocessing, and / or duplicate filtering. Image preprocessing may include: image size filtering and / or image horizontal and vertical filtering; text preprocessing may include: text length filtering and / or text normalization; and duplicate filtering may include: identifying and removing duplicate images.

[0047] In order to make the multimodal big model used to identify risky websites have a deeper understanding of images, fine-grained multimodal general domain training data can be obtained by processing the coarse-grained multimodal general domain training data using another multimodal big model. The other multimodal big model is, for example, a commercial multimodal big model. The processing may include: text expansion and / or accuracy optimization. Text expansion can enrich and expand the original text without changing the main idea of ​​the original text. Accuracy optimization can sort out the content of the text to make the expression more accurate and clear, and avoid ambiguity and misunderstanding.

[0048] According to some embodiments of the present application, fine-grained multimodal general domain training data can be obtained by inputting coarse-grained multimodal general domain training data, "prompt words" and "requirements" into another multimodal large model.

[0049] The main purpose of the multimodal risk website training data is to provide professional knowledge about identifying risky websites to the multimodal big model used to identify risky websites. It mainly contains data on risky websites and low-risk websites, which is used to improve the ability of the multimodal big model used to identify risky websites to identify risky websites.

[0050] Multimodal risky website training data can be obtained by: collecting information about risky and low-risk websites and performing feature extraction on the collected information to obtain image features and corresponding text features; and processing the obtained image features and corresponding text features using another multimodal large model to obtain the multimodal risky website training data. The other multimodal large model can be, for example, a commercially available multimodal large model.

[0051] According to some embodiments of the present application, multimodal risk website training data can be obtained by inputting the obtained image features and corresponding text features, "prompt words" and "requirements" into another multimodal large model.

[0052] The data sources for information on risky and low-risk websites include: official anti-fraud website information channels, including but not limited to internal corporate information; and public data on the Internet, including but not limited to websites of public welfare organizations, anti-fraud platforms, self-media, etc.

[0053] An example of a "prompt word" for entering another multimodal model is as follows: Please refine the text content based on the image content. An example of a "requirement" for entering another multimodal model is as follows: Regenerate the text content, and do not exceed 400 words.

[0054] Another example of a "prompt word" for inputting into another multimodal large model is as follows: Based on the given image content, please refine and summarize the text information in the image. Another example of a "requirement" for inputting into another multimodal large model is as follows: The generated text content must be highly consistent with the information in the image and cannot be self-edited or include content unrelated to the image content; the text is limited to no more than 400 words, and the sentences must be concise and the expression must be clear; the output content should focus on summaries and key points to ensure that it accurately conveys the key information contained in the image and is suitable for direct use in briefly describing the image content.

[0055] According to some embodiments of the present application, coarse-grained multimodal general domain training data and / or fine-grained multimodal general domain training data can constitute a multimodal general domain training dataset, and multimodal risk website training data can constitute a multimodal risk website training dataset. The multimodal general domain training dataset and the multimodal risk website training dataset can constitute a training dataset. By including the multimodal general domain training dataset and the multimodal risk website training dataset in the training dataset, the multimodal large model for identifying risky websites can learn more representation information of various images, thereby improving the generalization ability of the multimodal large model for identifying risky websites.

[0056] Furthermore, to ensure balanced data knowledge during training, a model data sampling and matching module can be constructed. This module uses a matching sampling algorithm to control the appropriate ratio of general domain knowledge to risky website domain knowledge, ultimately constructing a training dataset. This training dataset combines rich external world knowledge with risky website domain knowledge and serves as a training tool for connectors and the training of large multimodal models for identifying risky websites, such as instruction fine-tuning and alignment fine-tuning.

[0057] According to some embodiments of the present application, the ratio sampling algorithm may adopt a random lottery method. For example, sampling may be performed in a 1:1 ratio of general domain training data to risk website training data.

[0058] Connector training

[0059] According to some embodiments of the present application, a connector is used to connect an image encoder and a multimodal large model for identifying risky websites. As described above, after the image encoder encodes the image features of the website to be identified to obtain an image encoding vector, the image encoding vector is input into the connector to obtain an optimized image encoding vector; the text encoder encodes the corresponding text features of the website to be identified to obtain a text encoding vector; the optimized image encoding vector and the corresponding text encoding vector are fused and input into the multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website.

[0060] According to some embodiments of the present application, the connector is trained based on training data, the training data including multimodal general domain training data and multimodal risk website training data. The multimodal general domain training data includes coarse-grained multimodal general domain training data and / or fine-grained multimodal general domain training data.

[0061] According to some embodiments of the present application, a training connector includes: encoding image features of training data to obtain a training image encoding vector; encoding corresponding text features of the training data to obtain a training text encoding vector; and performing an image-based text generation task, an image-text comparison learning task, and an image-text matching task based on the training image encoding vector and the training text encoding vector. The image-based text generation task can be used to generate descriptive text based on the training image encoding vector; the image-text comparison learning task can be used to align the training image encoding vector and the training text encoding vector to achieve matching between the training image encoding vector and the training text encoding vector; and the image-text matching task can be used to determine whether the training image encoding vector and the training text encoding vector match.

[0062] The main purpose of training the connector is to enable the encoding information of the image class to learn from the encoding information of the text class, so that the trainer can obtain an optimized image encoding vector based on the graphic encoding vector output by the graphic encoder, and the optimized image encoding vector can better represent the image.

[0063] Figure 2 is a block diagram of a connector according to some embodiments of the present application. One example of a connector is a query-based connector.

[0064] like Figure 2 As shown in , the connector includes: an image converter and a text converter, wherein the image converter includes: a self-attention module, a cross-attention module and a feedforward network module; the text converter includes: a self-attention module and a feedforward network module.

[0065] The image converter can interact with the input training image encoding vector for image feature extraction. Specifically, the randomly generated query vector to be learned and the training text vector can be input into the self-attention module of the image converter, which shares parameters with the self-attention module of the text converter, wherein the input training text vector corresponds to the input training image encoding vector. The self-attention module of the image converter can model the dependency relationship between the query vector and the input training text vector through the existing attention mechanism function. Then, the query vector output by the self-attention module of the image converter and the input training image encoding vector can be input into the cross-attention module of the image converter, which can use the existing cross-attention mechanism to create a dependency relationship between the input query vector and the input training image encoding vector. The query vector output by the cross-attention module of the image converter is input into the feedforward network module of the image converter. This module can use the existing feedforward network to perform a nonlinear transformation on the input query vector to obtain enhanced image features. The input query vector is an image feature processed by the attention mechanism, which may contain feature associations within the image and cross-modal association information between the image and text. The enhanced image features can be regarded as a deeper image feature representation that is more discriminative and suitable for subsequent tasks (for example, image-based text generation tasks, image-text comparison learning tasks, and image-text matching tasks).

[0066] The text converter can interact with the input training text encoding vector for text feature extraction. Specifically, a randomly generated query vector to be learned and the training text encoding vector can be input into the text converter's self-attention module, which shares parameters with the image converter's self-attention module. The input training text encoding vector corresponds to the input training image encoding vector. The text converter's self-attention module can use existing attention mechanism functions to model the dependency between the input query vector and the input training text encoding vector. The training text encoding vector output by the text converter's self-attention module can then be input into the text converter's feedforward network module, which performs a nonlinear transformation on the input training text encoding vector to generate enhanced text features. The input training text encoding vector is a text feature processed by the attention mechanism and may contain feature correlations within the text or cross-modal associations between the image and text. The enhanced text features can be considered a deeper, more discriminative text feature representation, making them suitable for subsequent tasks (e.g., image-based text generation, image-text comparison learning, and image-text matching).

[0067] Here is an example of the attention mechanism function of the self-attention module:

[0068] .

[0069] In the above function, Q 、 K 、 V is the input matrix X Respectively with the matrix W Q 、 W K 、 W V The result of multiplication, the matrix W Q 、 W K 、 W V They are the main matrices that need to be trained for the self-attention module, Refers to the length of the word vector, where an example of a self-attention module is a neural network. In the above function, the matrix is ​​calculated Q and K The inner product of each row vector. In order to prevent the inner product from being too large, it is divided by d k The square root of . Q Multiply K After the transposition of , the number of rows and columns of the matrix obtained is n , n is the number of words in the sentence. This matrix can represent the attention strength between words. Finally, Softmax is used to calculate the attention coefficient of each word to other words. The Softmax in the above function is to perform Softmax on each row of the matrix, that is, the sum of each row becomes 1. The Softmax formula is:

[0070] .

[0071] In the above formula, Represents the vector input to the Softmax function Z No. i The weight of each category, c represents the index of the category, C represents the total number of categories, Represents a vector Z No. c The weight of each category.

[0072] The image converter and the text converter are used to jointly optimize three tasks with the same input, namely, the image-based text generation task, the image-text comparison learning task, and the image-text matching task. According to some embodiments of the present application, in the joint optimization, the image converter and the text converter share some parameters during training, and simultaneously optimize the loss functions of the three tasks through back propagation, so as to achieve better alignment in the feature representation of images and texts. This joint optimization can enable another large multimodal model that performs these three tasks to share useful image features and text features on these tasks, thereby improving the collaborative learning effect between tasks. The "same input" here refers to the use of the same image and text input samples to train the image-based text generation task, the image-text comparison learning task, and the image-text matching task during the training process. This does not mean that the input of each task is exactly the same, but that the three tasks are learned based on the same pair of images and texts. A brief description of the three tasks is given below:

[0073] Image-based text generation task: input an image and output a text describing the image. The training goal is to generate a natural language description that matches the image content.

[0074] Image-text comparison learning task: Input an image and multiple texts, including correctly matched and unmatched texts, to train the model to distinguish between matching positive and negative sample pairs.

[0075] Image-text matching task: Input an image and a corresponding text, and the task goal is to determine whether the two match.

[0076] The detailed description of these three tasks is as follows. In the following detailed description, an example of another multimodal large model that performs these three tasks is a multimodal large language model.

[0077] The image-based text generation task aims to convert the input image into a text feature vector, and then generate descriptive text using a multimodal large language model. The specific process is as follows.

[0078] The input image is a picture of a cat wearing sunglasses, and the target text description is "A cat wearingsunglasses".

[0079] Image encoding: Input the image (“cat wearing sunglasses”) into an image encoder, such as ViT (VisionTransformer). The image encoder extracts visual features and generates a set of visual embedding vectors, such as V 1,、 V 2.…… V nThese vectors contain spatial and content information of the image, such as "outline of cat", "shape of sunglasses", etc.

[0080] Connector extraction: These visual embedding vectors are cross-attended by query embeddings (e.g., learned queries) inside the connector. Query embeddings (e.g., 32) of Q-Former focus on different visual features and extract representative information. For example, one query may focus on the outline of the cat, and another query focuses on the color and shape of the sunglasses. Finally, Q-Former outputs a set of updated query feature vectors Q 1,、 Q 2, …, Q 32 These vectors represent high-level visual features.

[0081] Feature transformation and input to multi-modal large language model: The output vectors of the connector Q 1,、 Q 2, …, Q 32 will be transformed into a vector form compatible with the multi-modal large language model through a layer of projection, that is, Q ′1,、 Q ′2, …, Q ′ 32 . The transformed image feature vectors are input into the multi-modal large language model. The multi-modal large language model may be, for example, GPT (Generative Pretrained Transformer), BERT (Bidirectional Encoder Representations from Transformers), etc.

[0082] Text generation: The multi-modal large language model receives image feature vectors Q ′1,、 Q ′2, …, Q ′ 32 as input and generates a text description. The output may be "A cat wearing sunglasses" or similar text descriptions.

[0083] The image-text contrastive learning task aims to align the representations of images and texts so that they are closer in the same feature space, thereby achieving the matching of images and texts. The specific process is as follows.

[0084] Suppose: Also use a picture of "a cat wearing sunglasses" and there are multiple text descriptions as candidate descriptions, and only one correct description is "A cat wearing sunglasses".

[0085] Image encoding: Input the image (“cat wearing sunglasses”) into the image encoder to generate a set of visual embedding vectors V 1,、 V 2.…… V n .

[0086] Features: These visual embedding vectors enter the connector and extract the key features of the image through cross attention to form the query feature vector Q 1,、 Q 2.…… Q 32 .

[0087] These query feature vectors pass through the projection layer to finally generate the image representation vector Z i , representing the characteristics of the entire image.

[0088] Text encoding: Suppose there are three text descriptions below, the first one is the correct description:

[0089] "A cat wearing sunglasses";

[0090] "A dog wearing sunglasses"; and

[0091] “A cat wearing a hat”.

[0092] Each text description generates text embedding through a multimodal large language model, and obtains T 1. T 2. T 3, each of which T Vector is a text representation vector.

[0093] Contrastive loss calculation: In the image-text contrast learning task, the multimodal large language model uses contrastive loss to optimize the alignment of images and text.

[0094] Goal: Make the image representation vector Z i and the text representation vector of the correct description T 1 is closer to the text representation vector of the wrong description T 2 and T 3 are further apart.

[0095] Loss function: By comparing the loss function, the correct image-text match scores higher and the incorrect match scores lower.

[0096] Model optimization: Through backpropagation, the parameters of the image encoder and the multimodal large language model are adjusted to better align image and text features in the multimodal space. This optimization process gradually improves the accuracy of the multimodal large language model in matching images and text during inference.

[0097] The image-text matching task is designed as a binary classification task, requiring a multimodal large language model to determine whether given image and text features match. The input to the multimodal large language model is a pair of image encoding vectors and text encoding vectors. The multimodal large language model calculates the similarity or mutual information between the two and uses a classifier to determine whether the pair is a positive example (a matching image feature-text feature pair) or a negative example (an unmatched image feature-text feature pair).

[0098] The result of the task is a classification label (match or mismatch) or a match probability score. Through multiple rounds of training, the multimodal large language model can learn how to more accurately align image and text representations, making the similarity of matching image feature-text feature pairs higher in the feature space.

[0099] The image converter and text converter are optimized by jointly optimizing the loss function of the ITM (Image-Text Matching) task. Specifically, the image converter extracts fine-grained visual features from the image encoding vector, while the text converter extracts semantic features from the text encoding vector. By aligning the representations in the feature space, the two maximize mutual information, placing matching image-text feature pairs closer in the embedding space and mismatching pairs farther apart.

[0100] By simultaneously optimizing the three tasks above, we obtain a query-based connector. Inputting the image encoding vector into the query-based connector yields an optimized image encoding vector of the same dimension as the text encoding vector, enabling subsequent image and text vector fusion. An example of image and text vector fusion is image and text vector concatenation.

[0101] Training a large multimodal model for identifying risky websites

[0102] According to some embodiments of the present application, a multimodal large model for identifying risky websites is trained based on training data. Training the multimodal large model for identifying risky websites includes: inputting a training image encoding vector into a connector to obtain an optimized training image encoding vector; and inputting the optimized training image encoding vector, training text encoding vector, and sample labels into the multimodal large model for identifying risky websites to perform instruction fine-tuning and alignment fine-tuning on the multimodal large model for identifying risky websites.

[0103] According to some embodiments of the present application, the multimodal large model used to identify risky websites is based on a pre-trained multimodal large language model, such as open-source multimodal pre-trained large models such as Qwen2-VL, InternVL, and VisualGLM. Through extensive pre-training on a large web corpus, the pre-trained multimodal large language model has embedded rich general domain knowledge and possesses strong generalization and reasoning capabilities.

[0104] The fused features (e.g., the vector concatenation of the image and text mentioned above) are predicted using a pre-trained multimodal large language model to obtain a prediction result. The loss is calculated based on the prediction result and the sample label. The sample label is obtained, for example, in the following manner. After manual confirmation, a batch of websites are marked as risky websites or low-risk websites. The above prediction operation and loss calculation operation are repeated until the preset convergence conditions are met, thereby obtaining a trained multimodal large model for identifying risky websites. Examples of preset convergence conditions are as follows: the number of iterations reaches the maximum number of iterations, for example, 1000 times; or the loss value is less than a preset threshold.

[0105] In the above process, the multimodal question-answering of general domain data and risk website data is used as training data to fine-tune the instructions and alignment of the pre-trained multimodal large language model. An example of multimodal question-answering is:

[0106] Input: a picture and a text

[0107] Hint: Based on the image and the following key information corresponding to it: the image is a website, the keywords (high returns, get rich quick), the code similarity with the risky website "risky website domain name" is 98%, there has been redirection behavior, and it has existed for 1 year. Please help determine whether this website is a risky website based on the image and provide evidence.

[0108] Output: This website is a risk website, and the field involved is investment. From a technical point of view, the risk website code similarity with "risk website domain name" is 98%, indicating that the website highly imitates "risk website domain name" in design and function, aiming to take advantage of its popularity and user trust; from the business point of view, the page introduces various trading functions and advantages, but lacks specific data and evidence support; from the website's existence time of 1 year, it shows that the website is a newly established website, lacking long-term operation credibility and history.

[0109] When fine-tuning the pre-trained multi-modal large language model, if full fine-tuning is used, a large amount of computing resources and computing time is required for each training iteration. Considering that risk website identification belongs to a relatively narrow subfield, it is not necessary to fine-tune the full parameters. Therefore, a low-rank approximation scheme can be used to fine-tune the pre-trained multi-modal large language model. The input data for fine-tuning includes multi-modal general domain training data and multi-modal risk website training data. The pre-trained matrix can be a weight matrix ∈ , where R represents the real number field, and m and n represent the dimension sizes of the weight matrix . After fine-tuning, the parameters of the multi-modal large language model are set to h, and the formula can be expressed as:

[0110] .

[0111] In the above formula, x represents the input data for fine-tuning, is an incremental matrix that acts on x to supplement or fine-tune . B is an up-sampling matrix, and A is a down-sampling matrix. The product of the up-sampling matrix B and the down-sampling matrix A is consistent with the pre-trained matrix . A∈ , B∈ , where r << min(m, n), that is, the rank r is much smaller than the row and column values of the weight matrix. According to some embodiments of the present application, the value range of the rank r can be greater than 2 and less than 16. For example, the rank r can be set to 4. However, the value range of the rank r is not limited to the above range. During training, the pre-trained multi-modal large language model parameters are fixed, and only the down-sampling matrix A and the up-sampling matrix B are trained. When performing text generation and reasoning tasks, the product BA of the up-sampling matrix B and the down-sampling matrix A is added to the original parameters , without introducing additional reasoning delay. When training the down-sampling matrix A and the up-sampling matrix B, the down-sampling matrix A is initialized with a Gaussian distribution, and the up-sampling matrix B is initialized with a full 0 setting.

[0112] After instruction fine-tuning, the fine-tuned multimodal language model is fine-tuned for alignment. The optimization goal of alignment fine-tuning is to minimize the gap between the output of the fine-tuned multimodal language model and the human preference results. For example, the DPO (Direct Preference Optimization) strategy can be used. This strategy encourages the multimodal language model to assign a higher probability to strong preference outputs than weak preference outputs. The corresponding formula is:

[0113] .

[0114] The meanings of the symbols in the above formula are as follows.

[0115] L( ): represents the target multimodal large language model The loss function is used to calculate the expected RL (Reinforcement Learning) strategies and REF (Reference) Logarithmic ratio of policies to construct optimization objectives in reinforcement learning or contrastive learning.

[0116] : Indicates data distribution D The samples in ( x,y w ,y l ) is the expected value of .

[0117] D : Indicates data distribution or data set, indicating that the sample is from the data distribution D Extracted from .

[0118] β : Usually a hyperparameter used to adjust the importance of a certain item in the loss function. Specifically, β can be used to adjust different strategies (e.g. RL Strategy and REF strategies) contributes to the loss function.

[0119] σ : is a symbol, usually representing the Sigmoid function. The role of σ is to convert the expression in the brackets into a probability value or score.

[0120] : Input data.

[0121] : Strong preference output.

[0122] : Weak preference output.

[0123] : Represents the model Reinforcement learning strategy. Given input When the model The output produced The output of this strategy is the result of model fine-tuning.

[0124] : Indicates the reference strategy. Reference model input The output generated is The result output by this strategy is what humans expect.

[0125] The purpose of the loss function is to make the results of model fine-tuning as close as possible to the results expected by humans.

[0126] Figure 3 This is a block diagram of an apparatus for identifying risky websites according to some embodiments of the present application. As shown in the figure, the apparatus 300 may include: an acquisition module 310 for acquiring multimodal features of a website to be identified, the multimodal features including image features and corresponding text features; an image encoding module 320 for encoding the image features to obtain an image encoding vector; an optimized image encoding module 330 for inputting the image encoding vector into a connector to obtain an optimized image encoding vector; a text encoding module 340 for encoding text features to obtain a text encoding vector; a fusion module 350 for fusing the optimized image encoding vector and the text encoding vector to obtain a fused vector; and a risky website identification module 360 ​​for inputting the fused vector into a multimodal macro model for identifying risky websites to determine whether the website to be identified is a risky website.

[0127] Figure 4 is a block diagram of an apparatus for identifying risky websites according to some embodiments of the present application. As shown in the figure, apparatus 400 may include a processor 402 and a memory 404 storing instructions 406. These instructions 406, when executed by processor 402, cause the processor to perform one or more methods described herein. According to some embodiments of the present application, the apparatus for identifying risky websites may include a processor configured to perform one or more methods described herein.

[0128] According to an embodiment of the present application, the present application also provides a computer-readable storage medium storing instructions, which, when executed by a processor, causes the processor to perform one or more methods described herein.

[0129] According to an embodiment of the present application, the present application also provides a computer program product storing instructions, which, when executed by a processor, causes the processor to perform one or more methods described herein.

[0130] This application iteratively trains a pre-trained multimodal large model using training data including multimodal general domain training data and multimodal risk website training data, resulting in a multimodal large model for identifying risky websites. By fusing multimodal data such as text, images, audio, and video, mapping them into an embedding space, and inputting them into the multimodal large model for identifying risky websites, classification and recognition of multimodal data is achieved. This solution can improve the model's learning ability and enhance recognition accuracy and efficiency.

[0131] Those skilled in the art will appreciate that, for the sake of clarity, the division of the above functional units and modules is merely an example. In actual applications, the above functions can be assigned to different functional units and modules based on actual needs. In other words, the internal structure of the device can be divided into different functional units or modules to achieve all or part of the above functions.

[0132] The present application has been described in detail above by using specific embodiments, but the present application is not limited to the above embodiments. Various modifications and variations may be made to the present application without departing from the scope of the present application. The scope of the present application is defined by the appended claims.

[0133] Example embodiment:

[0134] Exemplary embodiment 1. A method for identifying risky websites, comprising: obtaining multimodal features of a website to be identified, the multimodal features including image features and corresponding text features; encoding the image features to obtain an image encoding vector; inputting the image encoding vector into a connector to obtain an optimized image encoding vector; encoding the text features to obtain a text encoding vector; fusing the optimized image encoding vector and the text encoding vector to obtain a fused vector; and inputting the fused vector into a multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website.

[0135] Example embodiment 2. The method of example embodiment 1, wherein the connector is trained based on training data, the training data comprising multimodal general domain training data and multimodal risky website training data.

[0136] Exemplary embodiment 3. The method according to exemplary embodiment 1 or 2, wherein training the connector includes: encoding image features of the training data to obtain a training image encoding vector; encoding corresponding text features of the training data to obtain a training text encoding vector; and performing image-based text generation tasks, image-text comparison learning tasks, and image-text matching tasks based on the training image encoding vector and the training text encoding vector.

[0137] Exemplary embodiment 4. A method as described in any one of exemplary embodiments 1-3, wherein the image-based text generation task is used to generate descriptive text based on the training image encoding vector; the image-text contrast learning task is used to align the training image encoding vector and the training text encoding vector to achieve matching between the training image encoding vector and the training text encoding vector; and the image-text matching task is used to determine whether the training image encoding vector and the training text encoding vector match.

[0138] Exemplary embodiment 5. The method of any one of exemplary embodiments 1-4, wherein the multimodal large model for identifying risky websites is trained based on the training data.

[0139] Exemplary embodiment 6. A method as described in any one of exemplary embodiments 1-5, wherein training the multimodal big model for identifying risky websites includes: inputting the training image encoding vector into the connector to obtain an optimized training image encoding vector; and performing instruction fine-tuning and alignment fine-tuning on the multimodal big model for identifying risky websites by inputting the optimized training image encoding vector, the training text encoding vector and sample labels into the multimodal big model for identifying risky websites.

[0140] Exemplary embodiment 7. A method as described in any of exemplary embodiments 1-6, wherein the multimodal general domain training data includes coarse-grained multimodal general domain training data and / or fine-grained multimodal general domain training data, the coarse-grained multimodal general domain training data is obtained based on preprocessing the multimodal features of the general domain, and the fine-grained multimodal general domain training data is obtained by processing the coarse-grained multimodal general domain training data using another multimodal large model.

[0141] Exemplary embodiment 8. The method of any one of exemplary embodiments 1-7, wherein the preprocessing includes at least one of the following: image size filtering, image horizontal and vertical filtering, text length filtering, text normalization, and identifying and removing duplicate images.

[0142] Exemplary embodiment 9. The method of any one of exemplary embodiments 1-8, wherein the multimodal risky website training data is obtained by processing multimodal features of risky websites and / or low-risk websites using another multimodal large model.

[0143] Example 10. An apparatus for identifying risky websites, comprising: a processor configured to execute the method according to any one of Example 1-9.

[0144] Example 11. An apparatus for identifying risky websites, comprising: a component for performing the method according to any one of Example 1-9.

[0145] Example embodiment 12. A computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of Example embodiments 1-9.

[0146] Example embodiment 13. A computer program product storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of example embodiments 1-9.

[0147] Exemplary embodiment 14. A device for identifying risky websites, comprising: an acquisition module for acquiring multimodal features of a website to be identified, the multimodal features including image features and corresponding text features; an image encoding module for encoding the image features to obtain an image encoding vector; an optimized image encoding module for inputting the image encoding vector into a connector to obtain an optimized image encoding vector; a text encoding module for encoding the text features to obtain a text encoding vector; a fusion module for fusing the optimized image encoding vector and the text encoding vector to obtain a fused vector; and a risky website identification module for inputting the fused vector into a multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website.

Claims

1. A method for identifying risky websites, comprising: Acquire multimodal features of the website to be identified, wherein the multimodal features include image features and corresponding text features; Encoding the image features to obtain an image encoding vector; Inputting the image coding vector into a connector to obtain an optimized image coding vector; Encoding the text features to obtain a text encoding vector; Fusing the optimized image encoding vector and the text encoding vector to obtain a fused vector; and The fused vector is input into a multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website. The connector is trained based on training data, the training data including multimodal general domain training data and multimodal risk website training data, and the ratio of the general domain training data to the multimodal risk website training data is controlled by a ratio sampling algorithm, wherein training the connector includes: Encoding the image features of the training data to obtain a training image encoding vector; Encoding corresponding text features of the training data to obtain a training text encoding vector; and Performing image-based text generation tasks, image-text comparison learning tasks, and image-text matching tasks based on the training image encoding vector and the training text encoding vector, and The multimodal large model for identifying risky websites is based on a pre-trained open source multimodal large language model and is trained based on the training data. Training the multimodal large model for identifying risky websites includes: Inputting the training image encoding vector into the connector to obtain an optimized training image encoding vector; and The pre-trained open source multimodal large language model is fine-tuned in terms of instruction and alignment by inputting the optimized training image encoding vector, the training text encoding vector, and the sample label into the pre-trained open source multimodal large language model.

2. The method according to claim 1, wherein The image-based text generation task is used to generate descriptive text based on the training image encoding vector; the image-text contrast learning task is used to align the training image encoding vector and the training text encoding vector to achieve matching between the training image encoding vector and the training text encoding vector; and the image-text matching task is used to determine whether the training image encoding vector and the training text encoding vector match.

3. The method according to any one of claims 1 to 2, wherein The multimodal general domain training data includes coarse-grained multimodal general domain training data and / or fine-grained multimodal general domain training data, wherein the coarse-grained multimodal general domain training data is obtained based on preprocessing the multimodal features of the general domain, and the fine-grained multimodal general domain training data is obtained by processing the coarse-grained multimodal general domain training data using another multimodal large model.

4. The method according to claim 3, wherein: The pre-processing includes at least one of the following items: image size filtering, image horizontal and vertical filtering, text length filtering, text normalization processing, and identifying and removing duplicate images.

5. The method according to any one of claims 1 to 2, wherein The multimodal risky website training data is obtained by processing the multimodal features of risky websites and / or low-risk websites using another multimodal large model.

6. A device for identifying risky websites, comprising: A processor configured to execute the method according to any one of claims 1-5.

7. A device for identifying risky websites, comprising: Means for performing the method according to any one of claims 1-5.

8. A computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 5.

9. A computer program product storing instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 5.

10. A device for identifying risky websites, comprising: An acquisition module, configured to acquire multimodal features of a website to be identified, wherein the multimodal features include image features and corresponding text features; An image encoding module, configured to encode the image features to obtain an image encoding vector; An optimized image coding module, configured to input the image coding vector into a connector to obtain an optimized image coding vector; A text encoding module, configured to encode the text features to obtain a text encoding vector; a fusion module, configured to fuse the optimized image encoding vector and the text encoding vector to obtain a fused vector; as well as A risk website identification module is used to input the fused vector into a multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website. The connector is trained based on training data, the training data including multimodal general domain training data and multimodal risk website training data, and the ratio of the general domain training data to the multimodal risk website training data is controlled by a ratio sampling algorithm, wherein training the connector includes: Encoding the image features of the training data to obtain a training image encoding vector; Encoding corresponding text features of the training data to obtain a training text encoding vector; and Performing image-based text generation tasks, image-text comparison learning tasks, and image-text matching tasks based on the training image encoding vector and the training text encoding vector, and The multimodal large model for identifying risky websites is based on a pre-trained open source multimodal large language model and is trained based on the training data. Training the multimodal large model for identifying risky websites includes: Inputting the training image encoding vector into the connector to obtain an optimized training image encoding vector; and The pre-trained open source multimodal large language model is fine-tuned in terms of instruction and alignment by inputting the optimized training image encoding vector, the training text encoding vector, and the sample label into the pre-trained open source multimodal large language model.

Images