Method and device for identifying risk website

Abstract

The invention provides a method and a device for identifying a risk website. The method comprises the steps of obtaining multi-modal features of a to-be-recognized website, wherein the multi-modal features comprise image features and corresponding text features; encoding the image features to obtain an image encoding vector; inputting the image coding vector into a connector to obtain an optimized image coding vector; encoding the text features to obtain a text encoding vector; fusing the optimized image coding vector and the text coding vector to obtain a fused vector; and inputting the fused vector into a multi-modal large model for identifying the risk website to determine whether the website to be identified is the risk website.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and in particular to a method and device for identifying risky websites. Background Art

[0002] With the rapid development of the Internet, the number of risky websites in the fields of virtual currency and e-commerce has increased sharply. These risky websites will not only cause huge economic losses, but also threaten national security and social stability. Therefore, it is very important to identify the risks of website threats.

[0003] Currently, website identification is usually done manually, that is, personnel identify the website's URL, webpage content and other information. However, due to the large number of websites, this method requires a lot of labor costs. Moreover, due to the uneven recognition level of personnel, the recognition efficiency is low and the error rate is high.

[0004] When using traditional machine learning methods to identify risky websites, due to the constant changes and high complexity of the network environment, the machine learning model cannot be trained and updated in a timely manner, so it often has defects such as low accuracy and poor generalization.

[0005] Therefore, there is a need for improved methods and apparatus for identifying risky websites. Summary of the invention

[0006] This application aims to provide an efficient and reliable solution for identifying risky websites.

[0007] According to a first aspect, the present application provides a method for identifying risky websites, comprising: obtaining multimodal features of a website to be identified, the multimodal features comprising image features and corresponding text features; encoding the image features to obtain an image encoding vector; inputting the image encoding vector into a connector to obtain an optimized image encoding vector; encoding the text features to obtain a text encoding vector; fusing the optimized image encoding vector and the text encoding vector to obtain a fused vector; and inputting the fused vector into a multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website.

[0008] According to a second aspect, the present application provides an apparatus for identifying risky websites, comprising: a processor, wherein the processor is configured to execute the method according to the first aspect.

[0009] According to a third aspect, the present application provides an apparatus for identifying risky websites, comprising: a component for executing the method according to the first aspect.

[0010] According to a fourth aspect, the present application provides a computer-readable storage medium storing instructions, which, when executed by a processor, causes the processor to perform the method according to the first aspect.

[0011] According to a fifth aspect, the present application provides a computer program product storing instructions, which, when executed by a processor, cause the processor to perform the method according to the first aspect.

[0012] According to the sixth aspect, the present application provides a device for identifying risky websites, including: an acquisition module for acquiring multimodal features of a website to be identified, the multimodal features including image features and corresponding text features; an image encoding module for encoding image features to obtain an image encoding vector; an optimized image encoding module for inputting the image encoding vector into a connector to obtain an optimized image encoding vector; a text encoding module for encoding text features to obtain a text encoding vector; a fusion module for fusing the optimized image encoding vector and the text encoding vector to obtain a fused vector; and a risky website identification module for inputting the fused vector into a multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website.

[0013] This application utilizes a large multimodal model for identifying risky websites, combined with multiple data inputs such as text and images, to accurately classify and identify websites, thereby improving the accuracy and efficiency of identification and reducing labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] A more thorough understanding of the present application may be obtained by combining the following drawings and referring to the following detailed description of specific embodiments.

[0015] Figure 1 is a flowchart of a method for identifying risky websites according to some embodiments of the present application.

[0016] Figure 2 is a block diagram of a connector according to some embodiments of the present application.

[0017] Figure 3 is a block diagram of an apparatus for identifying risky websites according to some embodiments of the present application.

[0018] Figure 4 is a block diagram of an apparatus for identifying risky websites according to some embodiments of the present application. DETAILED DESCRIPTION

[0019] The concepts of the present application will be described below with reference to the accompanying drawings, in which some embodiments of the concepts of the present application are shown. However, the concepts of the present application can be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein.

[0020] In general, all terms used in this application will be interpreted according to their ordinary meaning in the technical field, unless otherwise explicitly defined herein. All references to "a / an / the element, device, assembly, part, step, etc." will be interpreted openly as referring to at least one instance of the element, device, assembly, part, step, etc., unless otherwise explicitly stated. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless otherwise explicitly stated.

[0021] This application provides an efficient and reliable solution for identifying risky websites. The solution uses a multimodal large model for identifying risky websites, combined with multiple data inputs such as text and images, to accurately classify and identify websites, thereby improving the accuracy and efficiency of identification and reducing labor costs.

[0022] Figure 1 is a flowchart of a method for identifying risky websites according to some embodiments of the present application.

[0023] Step S110: Acquire multimodal features of the website to be identified, where the multimodal features include image features and corresponding text features.

[0024] In one embodiment of the present application, a website information feature extraction method is used to obtain image features and corresponding text features of the website to be identified. The website information feature extraction method extracts information features for websites with specified domain names in an automated manner. For example, information feature extraction can be performed using Internet data crawling technology. The extracted information features mainly include two types of features: the overall page image of the website, which is used as an image feature; and the main core keywords of the website, which are used as text features.

[0025] The website information feature extraction method includes the following steps: analyzing the website address information input by the user, automatically intercepting the code of the website information through the browser simulation access method to obtain the website source code, and automatically saving the screenshot to save the website page as a whole; then the core code of the website is extracted by the keyword extraction algorithm to obtain the high-frequency keywords in the code, which are used as text features to characterize the text information of the website, and the pictures saved by the automatic screenshot are used as image features to characterize the image information of the website. The image features and the corresponding text features constitute the image-text pair of the website information features.

[0026] Step S120: Encode the image features to obtain an image encoding vector. The image encoding vector is a vector that can be operated and can spatially represent the distribution of the image features in a multi-dimensional space.

[0027] In some embodiments of the present application, an image encoder is used to encode image features. An example of an image encoder is a pre-trained image encoding model. The image encoding model can use various neural network architectures, for example, CLIP (Contrastive Language-Image Pre-Training). The input information of the image encoding model is image features, and the output information is a multidimensional vector that can be operated.

[0028] Step S130: Input the image coding vector into the connector to obtain an optimized image coding vector. The optimized image coding vector can better represent the image information of the website.

[0029] In some embodiments of the present application, the connector is trained based on training data, and the training data includes multimodal general domain training data and multimodal risk website training data.

[0030] Step S140: Encode the text features to obtain a text encoding vector. The text encoding vector is a vector that can be operated and can spatially represent the distribution of text features in a multi-dimensional space.

[0031] In some embodiments of the present application, a text encoder is used to encode text features. An example of a text encoder is a pre-trained vocabulary of multidimensional vectors. The vocabulary can be used to convert tokens in a text into multidimensional vectors. The vector length of each token is a fixed number of dimensions. For example, it can be assumed that the vector length of each token is 768 dimensions. It can be understood that the vector length of each token is not limited to 768 dimensions, but can be other values.

[0032] Step S150: Fusing the optimized image coding vector and the text coding vector to obtain a fused vector.

[0033] In some embodiments of the present application, a fused vector is obtained by concatenating an optimized image encoding vector and a text encoding vector.

[0034] Step S160: input the fused vector into a multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website.

[0035] In some embodiments of the present application, a multimodal large model for identifying risky websites is trained based on training data, and the training data includes multimodal general domain training data and multimodal risky website training data.

[0036] According to some embodiments of the present application, the multimodal large model for identifying risky websites is capable of two types of identification: website type identification and website content identification. Website type identification is used to identify whether a website belongs to a high-risk website type or a general website type. If the business scenario of a website involves monetary transactions, such as virtual currency, e-commerce, etc., such websites can be marked as belonging to a high-risk website type. The remaining websites belong to general website types, that is, non-high-risk website types, for example, they can be marked as "others". Website content identification is performed through the image and language understanding capabilities of the multimodal large model for identifying risky websites, for example, identifying misleading sentences, typos, common sense errors in the website, and the design style of the website.

[0037] Based on the identified website types and website contents and the preset judgment rules, the risk type and risk level of the website can be comprehensively evaluated. For example, the identified website types and website contents are weighted and summed to calculate the comprehensive score, and the risk level is determined based on the comprehensive score. Subsequent monitoring and early warning work can be carried out based on the determined risk level.

[0038] From the perspective of accurate classification, the multimodal large model for identifying risky websites in this application can accurately classify websites by comprehensively analyzing the keywords and semantic information contained in the text content, as well as the visual elements and layout features in the image. Whether it is a website containing malicious code text prompts or a website that spreads bad information through images, it can be accurately classified into the corresponding category, greatly improving the accuracy of classification.

[0039] In terms of recognition efficiency, the multimodal large model for identifying risky websites in this application achieves rapid data processing and analysis with its advanced algorithms and architecture. Compared with the traditional single-modal recognition method, it does not need to process different types of data separately, reducing the complexity and time consumption of the processing flow. This means that in the face of a large amount of website data, risky websites can be identified quickly and accurately, effectively saving time costs.

[0040] In addition, in terms of cost control, the multimodal large model for identifying risky websites in this application greatly reduces labor costs. In the past, risky website identification often required a lot of manual participation, and professionals needed to spend a lot of time reviewing texts and analyzing images. The process was cumbersome and error-prone. However, the multimodal large model for identifying risky websites in this application replaces most of the manual work and only requires a small amount of manual supervision and maintenance, thereby significantly reducing labor costs and improving overall economic benefits.

[0041] As described above, according to some embodiments of the present application, the connector and the multimodal large model for identifying risky websites are trained based on training data, and the training data includes multimodal general domain training data and multimodal risky website training data. The construction of training data, the training of the connector, and the training of the multimodal large model for identifying risky websites will be described below.

[0042] Construction of training data

[0043] According to some embodiments of the present application, the training data is used to train a connector and a multimodal large model for identifying risky websites, and may include multimodal general domain training data and multimodal risky website training data.

[0044] The main purpose of multimodal general domain training data is to inject extensive world knowledge into the multimodal large model used to identify risky websites, and to improve the generalization ability, zero-sample or few-sample ability of the multimodal large model used to identify risky websites. It can be divided into two categories: coarse-grained multimodal general domain training data and fine-grained multimodal general domain training data.

[0045] Coarse-grained multimodal general domain training data can be based on Internet data. The amount of data based on the Internet is large, and it can include image data and corresponding text data in various scenarios. Usually, these image data and text data can be directly obtained through Internet data crawling technology, for example, images in videos and corresponding subtitles, pictures and corresponding news descriptions, websites and corresponding website introductions, etc. Coarse-grained multimodal general domain training data can be obtained by preprocessing these image data and corresponding text data.

[0046] According to some embodiments of the present application, preprocessing may include image preprocessing, text preprocessing and / or duplicate filtering. Image preprocessing may include: image size filtering and / or image horizontal and vertical filtering; text preprocessing may include: text length filtering and / or text normalization processing; and duplicate filtering may include: identifying and removing duplicate images.

[0047] In order to make the multimodal big model used to identify risky websites have a deeper understanding of images, fine-grained multimodal general domain training data can be obtained by processing the coarse-grained multimodal general domain training data using another multimodal big model. The other multimodal big model is, for example, a commercial multimodal big model. The processing may include: text expansion and / or accuracy optimization. Text expansion is the process of enriching and expanding the original text without changing the main idea of ​​the original text. Accuracy optimization can sort out the content of the text to make the expression more accurate and clear, and avoid ambiguity and ambiguity.

[0048] According to some embodiments of the present application, fine-grained multimodal general domain training data can be obtained by inputting coarse-grained multimodal general domain training data, "prompt words" and "requirements" into another multimodal large model.

[0049] The main purpose of the multimodal risky website training data is to provide professional knowledge about identifying risky websites to the multimodal big model used to identify risky websites. It mainly contains data of risky websites and low-risk websites, which is used to enhance the ability of the multimodal big model used to identify risky websites to identify risky websites.

[0050] The multimodal risky website training data can be obtained by the following operations: collecting information of risky websites and low-risk websites and performing feature extraction on the collected information to obtain image features and corresponding text features; and processing the obtained image features and corresponding text features using another multimodal large model to obtain the multimodal risky website training data. The other multimodal large model is, for example, a commercial multimodal large model.

[0051] According to some embodiments of the present application, multimodal risk website training data may be obtained by inputting the obtained image features and corresponding text features, "prompt words" and "requirements" into another multimodal large model.

[0052] The data sources for information on risky and low-risk websites include: official anti-fraud website information channels, including but not limited to the police, government, and corporate internal; and public data on the Internet, including but not limited to public welfare organization websites, anti-fraud platforms, self-media, etc.

[0053] An example of "prompt words" for inputting another multimodal large model is as follows: Please refine the text content in combination with the image content. An example of "requirements" for inputting another multimodal large model is as follows: Regenerate the text content, and do not exceed 400 words.

[0054] Another example of "prompt words" input into another multimodal large model is as follows: Please refine and summarize the text information in the picture based on the given picture content. Another example of "requirements" input into another multimodal large model is as follows: The generated text content must be highly consistent with the information in the picture, and cannot be self-compiled or include content unrelated to the picture content; the word count of the text is limited to no more than 400 words, and the sentences must be concise and the expression must be clear; the output content should be mainly summarization and key points to ensure that the key information contained in the picture can be accurately conveyed, and it is suitable for direct use to briefly describe the picture content.

[0055] According to some embodiments of the present application, coarse-grained multimodal general domain training data and / or fine-grained multimodal general domain training data may constitute a multimodal general domain training data set, and multimodal risk website training data may constitute a multimodal risk website training data set. The multimodal general domain training data set and the multimodal risk website training data set may constitute a training data set. By including the multimodal general domain training data set and the multimodal risk website training data set in the training data set, the multimodal large model for identifying risky websites can learn more representation information of various images, thereby improving the generalization ability of the multimodal large model for identifying risky websites.

[0056] In addition, in order to balance the data knowledge during the training process, a model data sampling ratio module can be constructed. This module controls the reasonable ratio of general domain knowledge and risk website domain knowledge through a ratio sampling algorithm, and finally constructs a training data set. This training data set combines rich external world knowledge with risk website domain knowledge, and serves the training of connectors and the training of multimodal large models for identifying risk websites, such as instruction fine-tuning and alignment fine-tuning.

[0057] According to some embodiments of the present application, the ratio sampling algorithm may adopt a random lottery method. For example, sampling may be performed in a 1:1 ratio of general domain training data to risk website training data.

[0058] Connector Training

[0059] According to some embodiments of the present application, the connector is used to connect the image encoder and the multimodal large model for identifying risky websites. As described above, after the image encoder encodes the image features of the website to be identified to obtain the image encoding vector, the image encoding vector is input into the connector to obtain the optimized image encoding vector; the text encoder encodes the corresponding text features of the website to be identified to obtain the text encoding vector; the optimized image encoding vector is fused with the corresponding text encoding vector and then input into the multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website.

[0060] According to some embodiments of the present application, the connector is trained based on training data, and the training data includes multimodal general domain training data and multimodal risk website training data. The multimodal general domain training data includes coarse-grained multimodal general domain training data and / or fine-grained multimodal general domain training data.

[0061] According to some embodiments of the present application, the training connector includes: encoding the image features of the training data to obtain the training image encoding vector; encoding the corresponding text features of the training data to obtain the training text encoding vector; and performing an image-based text generation task, an image-text comparison learning task, and an image-text matching task based on the training image encoding vector and the training text encoding vector. The image-based text generation task can be used to generate descriptive text based on the training image encoding vector; the image-text comparison learning task can be used to align the training image encoding vector and the training text encoding vector to achieve matching between the training image encoding vector and the training text encoding vector; and the image-text matching task can be used to determine whether the training image encoding vector and the training text encoding vector match.

[0062] The main purpose of training the connector is to enable the encoding information of the image class to learn from the encoding information of the text class, so that the trainer can obtain an optimized image encoding vector based on the graphic encoding vector output by the graphic encoder, and the optimized image encoding vector can better represent the image.

[0063] Figure 2 is a block diagram of a connector according to some embodiments of the present application. One example of a connector is a query-based connector.

[0064] like Figure 2 As shown in , the connector includes: an image converter and a text converter, wherein the image converter includes: a self-attention module, a cross-attention module and a feedforward network module; the text converter includes: a self-attention module and a feedforward network module.

[0065] The image converter can interact with the input training image encoding vector for image feature extraction. Specifically, the randomly generated query vector to be learned and the training text vector can be input into the self-attention module of the image converter, which shares parameters with the self-attention module of the text converter, wherein the input training text vector corresponds to the input training image encoding vector. The self-attention module of the image converter can model the dependency relationship between the query vector and the input training text vector through the existing attention mechanism function. Then, the query vector output by the self-attention module of the image converter and the input training image encoding vector can be input into the cross-attention module of the image converter, which can use the existing cross-attention mechanism to make the input query vector and the input training image encoding vector have a dependency relationship. The query vector output by the cross-attention module of the image converter is input into the feedforward network module of the image converter. The module can use the existing feedforward network to perform nonlinear transformation on the input query vector to obtain enhanced image features, wherein the input query vector is an image feature processed by the attention mechanism, which may contain feature associations within the image and cross-modal association information between the image and text; the enhanced image features can be regarded as a deeper image feature representation, which is more discriminative and suitable for subsequent tasks (for example, image-based text generation tasks, image-text comparison learning tasks, and image-text matching tasks).

[0066] The text converter can interact with the input training text encoding vector for text feature extraction. Specifically, the randomly generated query vector to be learned and the training text encoding vector can be input into the self-attention module of the text converter, which shares parameters with the self-attention module of the image converter, wherein the input training text encoding vector corresponds to the input training image encoding vector. The self-attention module of the text converter can model the dependency between the input query vector and the input training text encoding vector through the existing attention mechanism function. Then, the training text encoding vector output by the self-attention module of the text converter can be input into the feedforward network module of the text converter, which performs a nonlinear transformation on the input training text encoding vector to obtain enhanced text features, wherein the input training text encoding vector is a text feature processed by the attention mechanism, which may contain feature associations within the text or cross-modal association information between the image and the text; the enhanced text features can be regarded as a deeper text feature representation, which is more discriminative and suitable for subsequent tasks (for example, image-based text generation tasks, image-text comparison learning tasks, and image-text matching tasks).

[0067] Here is an example of the attention mechanism function of the self-attention module:

[0068]

[0069] In the above function, Q, K, and V are the input matrix X and the matrix W respectively. Q , W K , W V The result of the multiplication, matrix W Q , W K , W V They are the main matrices that need to be trained for the self-attention module, d k Refers to the length of the word vector, where an example of a self-attention module is a neural network. In the above function, the inner product of each row vector of the matrix Q and K is calculated. In order to prevent the inner product from being too large, it is divided by d k The square root of Q multiplied by the transpose of K, the resulting matrix has n rows and columns, where n is the number of words in the sentence. This matrix can represent the intensity of attention between words. Finally, Softmax is used to calculate the attention coefficient of each word to other words. The Softmax in the above function is to perform Softmax on each row of the matrix, that is, the sum of each row becomes 1. The Softmax formula is:

[0070]

[0071] In the above formula, Z i represents the component of the i-th category of the vector Z input to the Softmax function, c represents the index of the category, C represents the total number of categories, and Z c Represents the c-th category component of the vector Z.

[0072] The image converter and the text converter are used to jointly optimize three tasks with the same input, namely, the image-based text generation task, the image-text comparison learning task, and the image-text matching task. According to some embodiments of the present application, in the joint optimization, the image converter and the text converter share some parameters in training, and simultaneously optimize the loss functions of the three tasks through back propagation, so as to achieve better alignment in the feature representation of images and texts. This joint optimization can enable other multimodal large models that perform these three tasks to share useful image features and text features on these tasks, thereby improving the collaborative learning effect between tasks. The "same input" here refers to the use of the same image and text input samples to train the image-based text generation task, the image-text comparison learning task, and the image-text matching task during the training process. This does not mean that the input of each task is exactly the same, but that the three tasks are all learned based on the same pair of images and texts. A brief description of the three tasks is given below:

[0073] Image-based text generation task: Input an image and output a text describing the image. The training goal is to generate a natural language description that matches the image content.

[0074] Image-text comparison learning task: Input an image and multiple texts, including correctly matched and unmatched texts, to train the model to distinguish between matched positive and negative sample pairs.

[0075] Image-text matching task: Input an image and a corresponding text, and the task goal is to determine whether the two match.

[0076] The detailed description of these three tasks is as follows. In the following detailed description, an example of another multimodal large model that performs these three tasks is a multimodal large language model.

[0077] The image-based text generation task aims to convert the input image into a text feature vector, and then generate descriptive text by a multimodal large language model. The specific process is as follows.

[0078] The input image is a picture of a cat wearing sunglasses, and the target text description is "A cat wearingsunglasses".

[0079] Image encoding: Input the image (“cat wearing sunglasses”) to an image encoder, such as ViT (Vision Transformer). The image encoder extracts visual features and generates a set of visual embedding vectors, such as V1, V2, …, V n These vectors contain spatial and content information of the image, such as “the outline of the cat”, “the shape of the sunglasses”, etc.

[0080] Connector Extraction: These visual embeddings are cross-attended with the query embeddings (e.g., learned queries) inside the connector. The query embeddings (e.g., 32) of the Q-Former (Querying Transformer) focus on different visual features and extract representative information. For example, one query may focus on the outline of a cat, while another query focuses on the color and shape of sunglasses. Finally, the Q-Former outputs a set of updated query feature vectors Q1, Q2, ..., Q 32 , these vectors represent high-level visual features.

[0081] Feature conversion and input to the multimodal large language model: the output vector of the connector Q1,, Q2, ..., Q 32 It will be converted into a vector form compatible with the multimodal large language model through a layer of projection, that is, Q′1, Q′2, …, Q′ 32The converted image feature vector is input into a multimodal large language model. The multimodal large language model may be, for example, GPT (Generative Pretrained Transformer), BERT (Bidirectional Encoder Representations from Transformers), etc.

[0082] Text generation: The multimodal large language model receives image feature vectors Q′1, Q′2, …, Q′ 32 As input, and generates a text description. The output might be "A cat wearing sunglasses" or a similar text description.

[0083] The image-text contrast learning task aims to align the representations of images and texts so that they are closer in the same feature space, thereby achieving image and text matching. The specific process is as follows.

[0084] Suppose: Also use a picture of "a cat wearing sunglasses", and there are multiple text descriptions as candidate descriptions, and only one correct description is "A cat wearing sunglasses".

[0085] Image encoding: Input the image (“cat wearing sunglasses”) into the image encoder to generate a set of visual embedding vectors V1, V2, …, V n .

[0086] Features: These visual embedding vectors enter the connector, and the key features of the image are extracted through cross-attention to form the query feature vectors Q1, Q2, ..., Q 32 .

[0087] These query feature vectors are passed through the projection layer to finally generate the image representation vector Z i , representing the characteristics of the entire image.

[0088] Text encoding: Suppose there are three text descriptions below, the first one is the correct description:

[0089] "A cat wearing sunglasses";

[0090] "A dog wearing sunglasses"; and

[0091] “A cat wearing a hat”.

[0092] Each text description generates text embedding through a multimodal large language model to obtain T1, T2, and T3 respectively, where each T vector is a text representation vector.

[0093] Contrastive loss calculation: In the image-text contrast learning task, the multimodal large language model uses contrastive loss to optimize the alignment of images and texts.

[0094] Goal: Make the image representation vector Z i The distance between the text representation vector T1 of the correct description is closer, and the distance between the text representation vectors T2 and T3 of the incorrect description is farther.

[0095] Loss function: By comparing the loss function calculation, the correct image-text match has a higher score and the incorrect match has a lower score.

[0096] Model optimization: Through back propagation, the parameters of the image encoder and the multimodal large language model are adjusted so that the image features and text features are better aligned in the multimodal space. This optimization process gradually enhances the accuracy of the multimodal large language model in matching images and text during inference.

[0097] The image-text matching task is designed as a binary classification task, requiring the multimodal large language model to determine whether the given image features and text features match. The input of the multimodal large language model is a pair of image encoding vectors and text encoding vectors. The multimodal large language model calculates the similarity or mutual information value between the two, and uses the classifier to determine whether the pair is a positive sample (matching image feature-text feature pair) or a negative sample (unmatched image feature-text feature pair).

[0098] The result of the task is a classification label (match or mismatch) or a matching probability score. Through multiple rounds of training, the multimodal large language model can learn how to align image and text representations more accurately, so that the matching image feature-text feature pairs have higher similarity in the feature space.

[0099] The image converter and the text converter are optimized by jointly optimizing the loss function of the ITM (Image-Text Matching) task. Specifically, the image converter extracts fine-grained visual features from the image encoding vector, while the text converter extracts semantic features from the text encoding vector. The two align the representations in the feature space so that the matching image feature-text feature pairs are closer in the embedding space, while the non-matching pairs are farther away, thereby maximizing the mutual information.

[0100] By optimizing the above three tasks simultaneously, a query-based connector is obtained. By inputting the image encoding vector into the query-based connector, an optimized image encoding vector of the same dimension as the text encoding vector can be obtained, so that the subsequent image and text vector fusion can be achieved. An example of the vector fusion of image and text is the vector splicing of image and text.

[0101] Training of a large multimodal model for identifying risky websites

[0102] According to some embodiments of the present application, a multimodal large model for identifying risky websites is trained based on training data. Training the multimodal large model for identifying risky websites includes: inputting a training image encoding vector into a connector to obtain an optimized training image encoding vector; and performing instruction fine-tuning and alignment fine-tuning on the multimodal large model for identifying risky websites by inputting the optimized training image encoding vector, the training text encoding vector, and the sample label into the multimodal large model for identifying risky websites.

[0103] According to some embodiments of the present application, the multimodal large model for identifying risky websites is based on a pre-trained multimodal large language model, such as Qwen2-VL, InternVL, VisualGLM and other open source multimodal pre-trained large models. Through huge pre-training on the network corpus, the pre-trained multimodal large language model has embedded rich general domain knowledge and has strong generalization and reasoning capabilities.

[0104] The prediction results are obtained by predicting the fused features (for example, the vector concatenation of the image and text mentioned above) using a pre-trained multimodal large language model. The loss is calculated based on the prediction results and sample labels. The sample labels are obtained, for example, in the following way. A batch of websites are marked as risky websites or low-risk websites after manual confirmation. For example, websites banned by the police can be directly marked as risky websites. Repeat the above prediction operations and loss calculation operations until the preset convergence conditions are met, thereby obtaining a trained multimodal large model for identifying risky websites. Examples of preset convergence conditions are as follows: the number of iterations reaches the maximum number of iterations, for example, 1000 times; or the loss value is less than a preset threshold.

[0105] In the above process, the multimodal question and answer of general domain data and risk website data is used as training data to perform instruction fine-tuning and alignment fine-tuning on the pre-trained multimodal large language model. An example of multimodal question and answer is:

[0106] Input: a picture and a text

[0107] Prompt words: According to the picture and the following key information corresponding to the picture: the picture is a website, the keywords (high returns, get rich quickly), and "okexin.com"

[0108] The code similarity of the risky website is 98%. Redirection has occurred and has existed for 1 year. Please help determine whether the website is a risky website based on the pictures and provide evidence.

[0109] Output: This website is a risk website, and the field involved is investment. From a technical perspective, the code similarity with the risk website "okexin.com" is 98%, indicating that this website highly imitates "okexin.com" in design and function, with the purpose of taking advantage of its popularity and user trust; from a business perspective, the page introduces a variety of trading functions and advantages, but lacks specific data and evidence support; from the website's existence time of 1 year, it shows that this website is a newly established website, lacking the credibility and history of long-term operation.

[0110] When fine-tuning the pre-trained multimodal large language model, if full fine-tuning is used, each training iteration requires a lot of computing resources and computing time. Considering that risk website identification belongs to a relatively narrow niche, there is no need to fine-tune the full parameters. Therefore, a low-rank approximation solution can be used to fine-tune the pre-trained multimodal large language model. The input data for fine-tuning includes multimodal general domain training data and multimodal risk website training data. The pre-trained matrix can be the weight matrix W0∈R m×n , where R represents the real number domain, and m and n represent the dimensions of the weight matrix W0. The parameter of the multimodal large language model after fine-tuning is set to h, and the formula can be expressed as:

[0111] h=W0x+ΔWx=W0x+BAx=(W0+BA)x.

[0112] In the above formula, x represents the input data for fine-tuning, and ΔW is the incremental matrix, which acts on x to supplement or fine-tune W0. B is the dimension-raising matrix, and A is the dimension-reducing matrix. The product of the dimension-raising matrix B and the dimension-reducing matrix A is consistent with the pre-trained matrix W0. A∈R r×n , B∈R m×r, where r << min(m, n), that is, the rank r is much smaller than the row and column values of the weight matrix. According to some embodiments of the present application, the value range of the rank r can be greater than 2 and less than 16. For example, the rank r can be set to 4. However, the value range of the rank r is not limited to the above range. During training, the parameters of the pre-trained multi-modal large language model are fixed, and only the dimensionality reduction matrix A and the dimensionality increase matrix B are trained. When performing text generation and inference tasks, the product BA of the dimensionality increase matrix B and the dimensionality reduction matrix A is added to the original parameter W0, without introducing additional inference latency. When training the dimensionality reduction matrix A and the dimensionality increase matrix B, the dimensionality reduction matrix A is initialized with a Gaussian distribution, and the dimensionality increase matrix B is initialized with all zeros.

[0113] After instruction fine-tuning, alignment fine-tuning is performed on the fine-tuned multi-modal large language model. The optimization objective of alignment fine-tuning is to minimize the gap between the output result of the fine-tuned multi-modal large language model and the human preference result. For example, the DPO (Direct Preference Optimization) strategy can be adopted. This strategy encourages the multi-modal large language model to assign a higher probability to the strongly preferred output than the weakly preferred output. The corresponding formula is:

[0114]

[0115] The meanings of the symbols in the above formula are as follows.

[0116] represents the loss function of the target multi-modal large language model. This function is used to calculate the logarithmic ratio of the RL (Reinforcement Learning) strategy and the REF (Reference) strategy in expectation, so as to construct the optimization objective in reinforcement learning or contrast learning. The loss function of the target multi-modal large language model is used to calculate the logarithmic ratio of the RL (Reinforcement Learning) strategy and the REF (Reference) strategy in expectation, so as to construct the optimization objective in reinforcement learning or contrast learning.

[0117] represents the expected value of the sample (x, y w , y l ) in the data distribution D.

[0118] D: represents the data distribution or dataset, indicating that the sample is drawn from the data distribution D.

[0119] β: is usually a hyperparameter used to adjust the importance of a certain item in the loss function. Specifically, β can be used to adjust the contribution of the difference between different strategies (such as the RL strategy and the REF strategy) to the loss function.

[0120] σ: is a symbol, usually representing the Sigmoid function. The role of σ is to convert the expression in the parentheses into a probability value or score.

[0121] x: input data.

[0122] y w : Strong preference output.

[0123] y l : Weak preference output.

[0124] Representation Model Reinforcement learning strategy. Given an input x, the model The probability distribution of the output y generated. The result output by this strategy is the result of model fine-tuning.

[0125] π REf (y|x): represents the reference strategy. When the reference model inputs x, the probability distribution of the output y is generated. The result output by this strategy is what humans expect.

[0126] The purpose of the loss function is to make the results of model fine-tuning as close as possible to the results expected by humans.

[0127] Figure 3 It is a block diagram of an apparatus for identifying risky websites according to some embodiments of the present application. As shown in the figure, the apparatus 300 may include: an acquisition module 310, for acquiring multimodal features of a website to be identified, the multimodal features including image features and corresponding text features; an image encoding module 320, for encoding image features to obtain image encoding vectors; an optimized image encoding module 330, for inputting the image encoding vector into a connector to obtain an optimized image encoding vector; a text encoding module 340, for encoding text features to obtain a text encoding vector; a fusion module 350, for fusing the optimized image encoding vector and the text encoding vector to obtain a fused vector; and a risky website identification module 360, for inputting the fused vector into a multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website.

[0128] Figure 4 4 is a block diagram of an apparatus for identifying risky websites according to some embodiments of the present application. As shown in the figure, the apparatus 400 may include a processor 402 and a memory 404 storing instructions 406, which, when executed by the processor 402, cause the processor to perform one or more methods described herein. According to some embodiments of the present application, the apparatus for identifying risky websites may include a processor configured to perform one or more methods described herein.

[0129] According to an embodiment of the present application, the present application also provides a computer-readable storage medium storing instructions, which, when executed by a processor, enables the processor to perform one or more methods described herein.

[0130] According to an embodiment of the present application, the present application also provides a computer program product storing instructions, which, when executed by a processor, causes the processor to perform one or more methods described herein.

[0131] This application iteratively trains a pre-trained multimodal large model through training data including multimodal general field training data and multimodal risk website training data to obtain a multimodal large model for identifying risky websites. By fusing multimodal data such as text, images, audio, and video, mapping them to an embedding space and inputting them into the multimodal large model for identifying risky websites, classification and identification of multimodal data are achieved. This solution can improve the learning ability of the model and can improve the accuracy and efficiency of identification.

[0132] Those skilled in the art will understand that, for the sake of clarity, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned functions can be assigned to different functional units and modules to complete according to actual needs, that is, the internal structure of the device can be divided into different functional units or modules to achieve all or part of the functions described above.

[0133] The present application is described in detail above through specific embodiments, but the present application is not limited to the above embodiments. Various modifications and changes can be made to the present application without departing from the scope of the present application. The scope of the present application is defined by the attached claims.

[0134] Example embodiment:

[0135] Exemplary embodiment 1. A method for identifying risky websites, comprising: obtaining multimodal features of a website to be identified, the multimodal features including image features and corresponding text features; encoding the image features to obtain an image encoding vector; inputting the image encoding vector into a connector to obtain an optimized image encoding vector; encoding the text features to obtain a text encoding vector; fusing the optimized image encoding vector and the text encoding vector to obtain a fused vector; and inputting the fused vector into a multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website.

[0136] Exemplary embodiment 2. The method according to exemplary embodiment 1, wherein the connector is trained based on training data, the training data comprising multimodal general domain training data and multimodal risky website training data.

[0137] Exemplary embodiment 3. A method according to exemplary embodiment 1 or 2, wherein training the connector comprises: encoding image features of the training data to obtain a training image encoding vector; encoding corresponding text features of the training data to obtain a training text encoding vector; and performing image-based text generation tasks, image-text comparison learning tasks, and image-text matching tasks based on the training image encoding vector and the training text encoding vector.

[0138] Exemplary embodiment 4. A method as described in any one of exemplary embodiments 1-3, wherein the image-based text generation task is used to generate descriptive text based on the training image encoding vector; the image-text contrast learning task is used to align the training image encoding vector and the training text encoding vector to achieve matching between the training image encoding vector and the training text encoding vector; and the image-text matching task is used to determine whether the training image encoding vector and the training text encoding vector match.

[0139] Exemplary embodiment 5. The method as described in any of exemplary embodiments 1-4, wherein the multimodal large model for identifying risky websites is trained based on the training data.

[0140] Exemplary embodiment 6. A method as described in any one of exemplary embodiments 1-5, wherein training the multimodal big model for identifying risky websites comprises: inputting the training image encoding vector into the connector to obtain an optimized training image encoding vector; and performing instruction fine-tuning and alignment fine-tuning on the multimodal big model for identifying risky websites by inputting the optimized training image encoding vector, the training text encoding vector and sample labels into the multimodal big model for identifying risky websites.

[0141] Exemplary embodiment 7. A method as described in any of exemplary embodiments 1-6, wherein the multimodal general domain training data includes coarse-grained multimodal general domain training data and / or fine-grained multimodal general domain training data, the coarse-grained multimodal general domain training data is obtained based on preprocessing the multimodal features of the general domain, and the fine-grained multimodal general domain training data is obtained by processing the coarse-grained multimodal general domain training data using another multimodal large model.

[0142] Exemplary embodiment 8. A method as described in any of exemplary embodiments 1-7, wherein the preprocessing includes at least one of the following: image size filtering, image horizontal and vertical filtering, text length filtering, text normalization processing, and identifying and removing duplicate images.

[0143] Exemplary embodiment 9. A method as described in any of exemplary embodiments 1-8, wherein the multimodal risky website training data is obtained by processing multimodal features of risky websites and / or low-risk websites using another multimodal large model.

[0144] Exemplary embodiment 10. An apparatus for identifying risky websites, comprising: a processor configured to execute the method according to any one of exemplary embodiments 1-9.

[0145] Exemplary embodiment 11. An apparatus for identifying risky websites, comprising: a component for executing the method according to any one of exemplary embodiments 1-9.

[0146] Example embodiment 12. A computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of example embodiments 1-9.

[0147] Example embodiment 13. A computer program product storing instructions which, when executed by a processor, cause the processor to perform the method according to any one of example embodiments 1-9.

[0148] Exemplary embodiment 14. A device for identifying risky websites, comprising: an acquisition module, for acquiring multimodal features of a website to be identified, the multimodal features including image features and corresponding text features; an image encoding module, for encoding the image features to obtain an image encoding vector; an optimized image encoding module, for inputting the image encoding vector into a connector to obtain an optimized image encoding vector; a text encoding module, for encoding the text features to obtain a text encoding vector; a fusion module, for fusing the optimized image encoding vector and the text encoding vector to obtain a fused vector; and a risky website identification module, for inputting the fused vector into a multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website.

Claims

1. A method for identifying risky websites, comprising: Acquire multimodal features of the website to be identified, wherein the multimodal features include image features and corresponding text features; Encoding the image features to obtain an image encoding vector; Inputting the image coding vector into a connector to obtain an optimized image coding vector; Encoding the text features to obtain a text encoding vector; Merging the optimized image encoding vector and the text encoding vector to obtain a fused vector; and The fused vector is input into a multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website.

2. The method according to claim 1, wherein: The connector is trained based on training data, and the training data includes multimodal general domain training data and multimodal risky website training data.

3. The method according to claim 2, wherein: Training the connector includes: Encoding the image features of the training data to obtain a training image encoding vector; Encoding corresponding text features of the training data to obtain a training text encoding vector; and Based on the training image encoding vector and the training text encoding vector, an image-based text generation task, an image-text comparison learning task, and an image-text matching task are performed.

4. The method of claim 3, wherein: The image-based text generation task is used to generate descriptive text based on the training image encoding vector; the image-text contrast learning task is used to align the training image encoding vector and the training text encoding vector to achieve matching between the training image encoding vector and the training text encoding vector; and the image-text matching task is used to determine whether the training image encoding vector and the training text encoding vector match.

5. The method of claim 3, wherein: The multimodal large model for identifying risky websites is trained based on the training data.

6. The method of claim 5, wherein: Training the multimodal large model for identifying risky websites includes: Inputting the training image encoding vector into the connector to obtain an optimized training image encoding vector; The multimodal large model for identifying risky websites is instructionally fine-tuned and aligned fine-tuned by inputting the optimized training image encoding vector, the training text encoding vector and the sample label into the multimodal large model for identifying risky websites.

7. The method according to any one of claims 2 to 6, wherein: The multimodal general domain training data includes coarse-grained multimodal general domain training data and / or fine-grained multimodal general domain training data, wherein the coarse-grained multimodal general domain training data is obtained based on preprocessing the multimodal features of the general domain, and the fine-grained multimodal general domain training data is obtained by processing the coarse-grained multimodal general domain training data using another multimodal large model.

8. The method of claim 7, wherein: The preprocessing includes at least one of the following items: image size filtering, image horizontal and vertical filtering, text length filtering, text normalization processing, and identifying and removing duplicate images.

9. The method according to any one of claims 2 to 6, wherein: The multimodal risky website training data is obtained by processing the multimodal features of risky websites and / or low-risk websites using another multimodal large model.

10. A device for identifying risky websites, comprising: A processor configured to execute the method according to any one of claims 1-9.

11. A device for identifying risky websites, comprising: Means for performing the method according to any one of claims 1-9.

12. A computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1-9.

13. A computer program product storing instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 9.

14. A device for identifying risky websites, comprising: An acquisition module, used to acquire multimodal features of a website to be identified, wherein the multimodal features include image features and corresponding text features; An image encoding module, used for encoding the image features to obtain an image encoding vector; An optimized image coding module, used for inputting the image coding vector into a connector to obtain an optimized image coding vector; A text encoding module, used for encoding the text features to obtain a text encoding vector; A fusion module, used for fusing the optimized image coding vector and the text coding vector to obtain a fused vector; as well as The risk website identification module is used to input the fused vector into a multimodal large model for identifying risky websites to determine whether the website to be identified is a risky website.

Images