Multi-modal information fusion site identification method and device
Through the site recognition method of multimodal information fusion, combined with image and text feature extraction, and using the multi-head attention layer and full connection layer, the problem of low recognition efficiency of gambling websites in the existing technology is solved, and efficient recognition of gambling websites is achieved and robustness recognition.
Patent Information
- Application Number
- CN202411034654.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-26
Smart Images

Figure CN119025958A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a site identification method and device for multi-modal information fusion. Background Art
[0002] Gambling site identification is an important network security task that helps users identify and avoid visiting websites that may carry malware, phishing attacks or other threats. Traditionally, gambling site identification technology mainly relies on manual identification by frontline police officers, which is cumbersome and inefficient.
[0003] However, with the comprehensive development of computer technology, hacker technology and the technology of online fraud gangs are also upgrading. Gambling websites are often set up overseas and use the method of changing IP addresses from time to time to prevent tracking. This makes it difficult for public security organs to quickly obtain the actual information of gambling websites, and they cannot always use the method of blocking network addresses to avoid the erosion of gambling websites.
[0004] Most online gambling criminal organizations set up a complicated server hierarchy and use one-way contact to contact gambling participants. At the same time, some criminals have a strong anti-detection awareness and use different hacking techniques to block or divert the sniffing tools of the public security organs. The public security organs face difficulties in obtaining digital evidence, and online evidence collection work is frustrated.
[0005] Therefore, how to improve the recognition efficiency of gambling websites has become a technical problem that needs to be solved urgently. Summary of the invention
[0006] In view of this, the main purpose of this application is to provide a site identification method and device for multimodal information fusion, with the aim of improving the identification efficiency of gambling websites.
[0007] In a first aspect, the present application provides a site identification method for multimodal information fusion, the method comprising:
[0008] Obtain images and texts of the website to be identified;
[0009] Perform image feature extraction on the image through an image feature extraction model to obtain the visual features of the website to be identified;
[0010] The first multi-head attention layer captures the feature relationship of visual features to obtain the visual self-attention weight matrix;
[0011] Extract text features from the text using a text feature extraction model to obtain text features of the website to be identified;
[0012] The feature relationship of text features is captured through the second multi-head attention layer to obtain the text self-attention weight matrix;
[0013] Convert the visual multi-head attention weight matrix and the text word self-attention weight matrix into fusion features;
[0014] The recognition result of the website to be identified is output based on the fusion features through the fully connected layer.
[0015] In some implementations of the first aspect of the present application, obtaining an image and text of a website to be identified includes:
[0016] Initiate a request to the website to be identified through the crawler Python library, and use the corresponding parsing method according to the code source of the website to be identified to parse the text corresponding to the website to be identified;
[0017] The browser is automatically controlled through the selenium library to access the website to be identified, and a screenshot of the website to be identified is taken after the access is successful to obtain the corresponding image of the website to be identified.
[0018] In some implementations of the first aspect of the present application, the image feature extraction model is a deep residual network ResNet50 model in which the average pooling layer and the fully connected layer are replaced with a pooling layer of size 1*1 after pre-training.
[0019] In some implementations of the first aspect of the present application, the ResNet50 model is trained in the following manner:
[0020] Obtain an image dataset and divide the image dataset into a training set and a validation set;
[0021] Perform image enhancement on the training set;
[0022] The image-enhanced training set is used to train the ResNet50 model. During the model training process, the adaptive moment estimation optimizer Adam optimizer is used for parameter optimization, and the cross entropy loss function is used as the loss function.
[0023] When the classification accuracy of the ResNet50 model for the validation set reaches the preset requirement, the trained ResNet50 model is obtained.
[0024] In some implementations of the first aspect of the present application, the text feature extraction model includes: a semantic understanding model and a word segmenter corresponding to the semantic understanding model.
[0025] In some implementations of the first aspect of the present application, extracting text features from text using a text feature extraction model includes:
[0026] Convert the text into a sequence of tokens through a tokenizer;
[0027] The token sequence is converted into the input of the semantic understanding model through the tokenizer, so that the semantic understanding model outputs text features.
[0028] In some implementations of the first aspect of the present application, the identification results include: it is a gambling website, it is not a gambling website, and it is impossible to determine whether it is a gambling website.
[0029] In some implementations of the first aspect of the present application, the input dimension of the first multi-head attention layer is 2048 and the number of attention heads is 16, and the input dimension of the second multi-head attention layer is 768 and the number of attention heads is 16.
[0030] In some implementations of the first aspect of the present application, the dimension of the fused feature is 5632, and the fully connected layer includes:
[0031] The first linear layer has an input dimension of 5632 and an output dimension of 1024;
[0032] A second linear layer with an input dimension of 1024 and an output dimension of 512;
[0033] A third linear layer with input dimension 512 and output dimension 256;
[0034] A fourth linear layer with input dimension 256 and output dimension 128;
[0035] A fifth linear layer with input dimension 128 and output dimension 3.
[0036] A second aspect of the present application provides a site identification device for multimodal information fusion, the device comprising:
[0037] A data acquisition module, used to acquire images and texts of the website to be identified;
[0038] An image feature extraction module is used to extract image features from an image through an image feature extraction model to obtain visual features of a website to be identified;
[0039] A visual attention module, used to capture the feature relationship of visual features through the first multi-head attention layer to obtain a visual self-attention weight matrix;
[0040] A text feature extraction module is used to extract text features from text through a text feature extraction model to obtain text features of the website to be identified;
[0041] The text attention module is used to capture the feature relationship of text features through the second multi-head attention layer to obtain the text self-attention weight matrix;
[0042] Feature fusion module, used to convert the visual multi-head attention weight matrix and the text word self-attention weight matrix into fused features;
[0043] The website recognition module is used to output the recognition result of the website to be recognized based on the fusion features through the fully connected layer.
[0044] The technical solution provided by this application has the following beneficial effects:
[0045] In the technical solution provided in the present application, first, the image and text information of the website to be identified are obtained for subsequent feature extraction; then, the image is feature extracted by the image feature extraction model to capture the visual features of the image, for example, for extracting the visual features of specific images such as gambling game icons and specific color combinations in gambling websites. By using the text feature extraction model to extract features from the text, the semantic information of the text can be captured, which helps to identify the commonly used words, phrases, synonyms, slang or specific text patterns in gambling websites. After that, the first multi-head attention layer and the second multi-head attention layer are introduced to capture the feature relationship between visual features and text features, respectively, so as to focus on key features more accurately. Then, the visual multi-head attention weight matrix and the text self-attention weight matrix are converted into fusion features, which realizes the effective fusion of image and text information. The fusion feature contains both the visual information of the image and the semantic information of the text, providing a richer basis for subsequent recognition. Finally, the fusion feature is processed by the fully connected layer, and the recognition result of the website to be identified is finally output. The fully connected layer can effectively convert the feature map extracted by the previous level into a category probability or numerical prediction, thereby realizing the accurate recognition of the gambling website. In summary, this application achieves efficient identification of gambling websites through feature extraction and attention mechanism, fusion of multimodal data and processing of fully connected layers without manual identification. Even if the IP address of the gambling website keeps changing, the gambling website can be identified through fusion feature recognition of image and text features. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 A schematic diagram of a flow chart of a site identification method for multimodal information fusion provided in an embodiment of the present application;
[0047] Figure 2 A flowchart of another method for identifying a site by fusion of multimodal information provided in an embodiment of the present application;
[0048] Figure 3 A schematic diagram of the performance of neural networks of different depths provided in an embodiment of the present application in the ILSVRC competition;
[0049] Figure 4 A flowchart of another method for identifying a site by fusion of multimodal information provided in an embodiment of the present application;
[0050] Figure 5 A flowchart of another method for identifying a site by fusion of multimodal information provided in an embodiment of the present application;
[0051] Figure 6 A schematic diagram of the structure of a site identification device for multimodal information fusion provided in an embodiment of the present application. DETAILED DESCRIPTION
[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0053] In view of the problems mentioned in the background technology, the inventors found that text features are the key information source for identifying gambling sites. These features include specific keywords or phrases in the website content, such as "gambling", "betting" or "online casino". However, cybercriminals can evade text-based detection by using synonyms, slang or other circumvention techniques. In addition, image features can also provide insights into the visual elements of the website. For example, gambling websites often contain slot machines, playing cards or other gambling-related images. By utilizing image processing and computer vision technology, these features can be extracted from website images and used to identify gambling sites. Based on this, the present application provides a site identification method and device with multimodal information fusion.
[0054] See also Figure 1 As shown, the first embodiment of the present application provides a site identification method for multimodal information fusion, the method comprising:
[0055] S101: Acquire images and texts of a website to be identified.
[0056] In the embodiment of the present application, for each website to be identified, the corresponding image content and text data, such as screenshots of the website and text in the webpage source, are obtained, so as to perform multimodal identification on the website to be identified later. It is understandable that for gambling websites, they usually contain slot machines, playing cards or other gambling-related images, as well as gambling-related text content.
[0057] In some implementations of the embodiments of the present application, the image and text of the website to be identified can be obtained by the following steps:
[0058] S201: Initiate a request to the website to be identified through the crawler Python library, and use a corresponding parsing method according to the code source of the website to be identified to parse out the text corresponding to the website to be identified.
[0059] In the embodiment of the present application, for the image content of the website to be identified, the urllib client and the requests client in the Python library are used to initiate a request to the website to be identified, and different parsing techniques are used according to the source code of different websites to be identified. For example, different XML Path Language (Xpath) is used in combination with the General News Extractor (GNE) module to accurately parse the text content of the website, thereby realizing the automatic acquisition of web page text content.
[0060] S202: Automatically control the browser through the selenium library to access the website to be identified, and take a screenshot of the website to be identified after the access is successful, so as to obtain a picture corresponding to the website to be identified.
[0061] In the embodiment of the present application, for the image content of the website to be identified, the selenium library is used to automatically control the browser to access the website to be identified; once the website to be identified is successfully accessed, the screenshot function of the selenium library will be triggered to take a real-time screenshot of the website page, thereby quickly capturing the visual information of the website page, providing intuitive data for subsequent website content analysis, design review, etc. In addition, through the automated screenshot operation, not only the efficiency of image acquisition is improved, but also the consistency of each screenshot can be ensured, avoiding the errors that may be caused by manual operation.
[0062] In some implementations, since web crawlers may encounter problems such as inconsistent image quality, noise, or inappropriate brightness and contrast when acquiring images, the present application may also clean the data after taking screenshots. For example, the OpenCV image processing library is used to perform denoising, brightness and contrast adjustment on the screenshots to ensure that the image quality meets the requirements.
[0063] S102: extracting image features from the image using an image feature extraction model to obtain visual features of the website to be identified.
[0064] In an embodiment of the present application, the image feature extraction model refers to a deep learning model that is pre-trained through a training set and a test set and is used to extract useful features from an image, and the useful features extracted from the image are visual features, which represent different features of the image such as color, shape, texture, etc.
[0065] In terms of model training of image feature extraction models, the depth of the network is crucial to the performance of the model. When the number of network layers is increased, the network can extract more complex feature patterns, so theoretically better results can be achieved when the model is deeper. However, experimental results show that as the network depth increases, the training problem increases, resulting in performance degradation. To solve this problem, researchers have proposed various strategies, among which the ResNet network stands out for its unique residual learning mechanism. The ResNet network was modified based on the VGG19 network and introduced a short-circuit mechanism and residual units. These innovations enable the network to perform deep training more effectively while avoiding performance degradation.
[0066] See also Figure 3 As shown, Figure 3 The performance of neural networks of different depths in the ILSVRC competition is shown. The horizontal axis represents the ILSVRC competitions of different years, the number of layers of the ResNet network is marked in the figure, and the vertical axis represents the error rate. As can be seen from the figure: as the number of network layers increases, the error rate gradually decreases, and deeper networks can significantly improve model performance and reduce error rates. The ResNet model still maintains a low error rate at 152 layers. In view of the outstanding performance of the ResNet network in deep network training, some implementations in the embodiments of the present application select the ResNet50 model as the image feature extraction model to achieve higher model performance while maintaining the network depth. As an image classification algorithm based on a deep convolutional neural network (CNN), ResNet50 not only inherits the powerful learning ability of the traditional CNN model for image features, but also successfully solves the gradient vanishing problem in deep network training by introducing a unique residual connection mechanism. This design enables ResNet50 to build a deeper network structure, effectively improving the performance of the model.
[0067] Among them, the ResNet50 model specifically implements feature extraction through the following process: The core building unit of the ResNet50 model is a residual block consisting of two main parts: the main path and the jump connection. The convolution layer of the main path is responsible for extracting deep features in the image layer by layer. At the same time, the jump connection directly bypasses the convolution layer in the main path and passes the input information directly to the output of the main path. In this way, the output of the main path is actually the superposition of the original features of the input information and the features extracted by the convolution layer. This superposition method is residual learning. In addition, in order to improve the training efficiency and stability of the model, a batch normalization layer is introduced in each residual block, which helps to speed up the convergence of the model and prevent overfitting. At the same time, the activation function is also used to increase the nonlinearity of the model, so that the model can learn more complex feature representations. Through these designs, ResNet50 can build a deeper network structure and achieve excellent performance in tasks such as image classification.
[0068] In some implementations of the embodiments of the present application, the image feature extraction model is specifically a ResNet50 model in which the average pooling layer and the fully connected layer are replaced with a pooling layer of size 1*1 after pre-training. Specifically, after the ResNet50 model completes model training, the sub-module list of the ResNet50 model is extracted, and the last two layers of the ResNet50 model, namely the average pooling layer and the fully connected layer, are removed. After that, a pooling layer of size 1*1 is added, so that the ResNet50 model is modified to only perform feature extraction without final classification, thereby constituting the ResNet50 model for extracting visual features in the embodiments of the present application.
[0069] In some implementations of the first aspect of the present application, the ResNet50 model is trained in the following manner:
[0070] S401: Obtain an image dataset, and divide the image dataset into a training set and a validation set.
[0071] In an embodiment of the present application, first, an image data set is read from a specified path, and the image data set is randomly cut in proportion to divide the data set into a training set and a validation set for model training and evaluation. Among them, random cutting can also ensure the data distribution and diversity of the training set and the validation set.
[0072] It is understandable that before being used to train the model, the training set is annotated and labeled. Data annotation refers to the process of adding labels to the original data. When the training data is annotated, the corresponding label is called Ground Truth. Among them, in order to ensure that the data set achieves the ideal effect during training, the data target must be accurately labeled, and other interfering information in the labeled area must be avoided as much as possible, while ensuring its integrity. For example, the training samples in the data set are manually labeled separately, and the pictures and text content are clearly marked as to whether they are gambling websites. Ground Truth is defined as "yes" and "no", "yes" is marked as 0, and "no" is marked as 1. Finally, the annotation of the training data set is completed, maintaining a high degree of consistency and accuracy.
[0073] For example, the preparation process before model training stores the data path and labels of the training set in the variables train_paths and train_labels, and stores the data path and labels of the validation set in the variables valid_paths and valid_labels. Then, you need to set up a folder structure to manage the data set, such as Caption, image, labels, and ImageSets folders, which are used to store the original text data set, the original image data set, the labeled txt file, and the classification of the training / test data set. In addition, you can also set up a makeTxt.py folder in the root directory. This file mainly classifies the data set into training data set, test data set, and validation data set. By default, train, val, and text are randomly classified in a ratio of 3:1:1. After running, four files will appear in the ImagesSets folder, which store the image names of the generated training data set and test data set. At the same time, these four files will also appear in the data directory to store the image paths of the training data set and the test data set.
[0074] In some implementations of the embodiments of the present application, by acquiring, preprocessing, labeling, and dividing the data set, a data set containing 3,000 training sets, 1,000 validation sets, and 1,000 test sets, totaling 5,000 images, is finally formed as shown in Table 1. The training set and the validation set contain 1,500 gambling websites and 2,500 other types of websites. In addition, each image or website in the training set and the validation set has a corresponding label.
[0075] Table 1
[0076]
[0077] S402: Perform image enhancement on the training set.
[0078] In the embodiment of the present application, in the process of loading the test set into the ResNet50 model for training, data enhancement techniques such as color jittering, random cropping, etc. are used to increase the diversity and richness of the training data, thereby improving the generalization ability of the model. For example, the image enhancement operation is implemented using the image transformation module in the PaddlePaddle deep learning framework.
[0079] S403: The image-enhanced training set is used to perform model training of the ResNet50 model. During the model training process, the adaptive moment estimation optimizer Adam optimizer is used to perform parameter optimization, and the cross entropy loss function is used as the loss function.
[0080] In the embodiment of the present application, a pre-trained ResNet50 model is selected as the basic model, and the model training is performed on the training data set. During the model training process, the parameters of the model are gradually updated through multiple iterative training to improve the accuracy of the model in the image classification task.
[0081] Specifically, the present application uses the adaptive moment estimation optimizer Adam as the optimizer for parameter optimization, and combines the cross entropy loss function as the loss function for model training. Among them, the Adam (Adaptive Moment Estimation) is an optimization algorithm that dynamically adjusts the learning rate of each parameter by calculating the first-order moment estimation and second-order moment estimation of the gradient, which can converge faster and find a better local optimal solution. The loss function is a measure of the difference between the model prediction results and the actual results. In the classification task, the cross entropy loss function measures the difference between the two distributions based on the probability distribution, that is, the difference between the category distribution predicted by the model and the actual category distribution. During the training process, the ResNet50 model will try to minimize the cross entropy loss function, so that its prediction results are closer to the actual results.
[0082] S404: When the classification accuracy of the ResNet50 model on the validation set reaches the preset requirement, a trained ResNet50 model is obtained.
[0083] In the embodiment of the present application, when the classification accuracy of the ResNet50 model on the validation set reaches the preset requirement, the model training is considered completed. Afterwards, the trained model parameters can be saved in a specified path for use in subsequent image classification prediction tasks.
[0084] S103: Capture the feature relationship of visual features through the first multi-head attention layer to obtain a visual self-attention weight matrix.
[0085] The multi-head attention layer refers to a deep learning mechanism that processes input features in parallel through multiple attention heads to capture the relationship between features, and the first multi-head attention layer specifically refers to the multi-head attention layer used to process visual features. The output of the multi-head attention layer is the self-attention weight matrix, which refers to a matrix that represents the relationship between features. The higher the weight value, the stronger the relationship between features. The visual self-attention weight matrix specifically represents the relationship between visual features.
[0086] In an embodiment of the present application, first, the visual features extracted by the image feature extraction model are input into the multi-head attention layer. Then, the multi-head attention layer calculates the attention scores between the visual features. Specifically, each attention head calculates a query, key, and value matrix. After that, the attention score is calculated by performing a dot product operation on the query matrix and the key matrix. The score is then normalized to obtain the attention weight. Finally, the attention weight is multiplied by the value matrix to obtain a weighted feature representation, and the results of multiple attention heads are spliced to form the final self-attention weight matrix.
[0087] The embodiment of the present application introduces a multi-head attention layer to capture the complex relationship between visual features, improve the model's understanding of image content, and enhance the model's generalization ability.
[0088] In some implementations of the embodiments of the present application, the input dimension of the first multi-head attention layer is 2048 and the number of attention heads is 16.
[0089] S104: extracting text features from the text using a text feature extraction model to obtain text features of the website to be identified.
[0090] In an embodiment of the present application, the text feature extraction model refers to a deep learning model used to extract useful features from text, and the useful features extracted from the image are text features.
[0091] In some implementations of the embodiments of the present application, the text feature extraction model includes: a semantic understanding model and a word segmenter corresponding to the semantic understanding model. In the embodiments of the present application, the text feature extraction model consists of a semantic understanding model and a corresponding word segmenter. The word segmenter is used for the preprocessing step in the text feature extraction process to convert the text into a data form that can be understood by the semantic understanding model, and then the output of the word segmenter is input into the semantic understanding model for feature extraction. Among them, the semantic understanding model can be a representation model ERNIE 2.0 enhanced by knowledge integration, a bidirectional transformer model BERT, etc., which does not affect the implementation of the embodiments of the present application.
[0092] In some implementations of the embodiments of the present application, a semantic understanding pre-training framework with Transformer Encoder as the skeleton is mainly used to extract text features, such as the ERNIE model, the BERT model, etc. This framework is a deep neural network structure suitable for processing sequence data such as text. Tasks can be continuously added without reducing the accuracy of previous tasks, so that lexical, syntactic, and semantic expressions can be obtained more effectively.
[0093] Specifically, the tasks of the language model can be classified into the following three categories, which enable the model to continuously learn new tasks: word-level task sentences, structure-level tasks, and semantic-level tasks.
[0094] For the text data in the dataset, word embedding technology is used to convert it into vector representation to facilitate neural network processing; then it is encoded using the text convolutional neural network TextCNN to capture local features in the text; finally, the encoded features are sent to the semantic understanding pre-training framework with Transformer Encoder as the skeleton for subsequent data analysis. During the training process, the framework learns multiple representations of text, such as lexical, syntactic and semantic information, through multiple pre-training tasks.
[0095] For example, do the English terms such as Knowledge Masking Task and Capitalization Prediction Task have professional Chinese names?
[0096] The above-mentioned pre-training tasks include but are not limited to:
[0097] Knowledge Masking Task: Use this task for preparation and training to help the model learn the dependencies between local and global contexts, thereby obtaining the initial version of the model.
[0098] Capitalization Prediction Task: This task is used to predict whether an English word is capitalized or lowercase, which helps the model learn the grammatical role of words and named entity recognition.
[0099] Token-Document Relation Prediction Task: This task is used to predict whether a word will appear multiple times in a text, or whether the word is a keyword. This task can improve the model's ability to capture keywords.
[0100] Sentence Reordering Task: This task is used to solve multi-classification problems. Simply put, after a paragraph is split into m segments and their order is disrupted, the model needs to predict the original order of these m segments.
[0101] Sentence Distance Task: This task allows the model to learn the distance between sentences through document-level information, with "0" representing that two sentences belong to the same document and are adjacent, "1" representing that two sentences belong to the same document but are not adjacent, and "2" representing that two sentences come from different documents.
[0102] Discourse Relation Task: This task is used to determine the semantic relationship between sentences and predict the semantic or modification relationship between two sentences.
[0103] IR Relevance Task: This task is used to predict the relationship between the query and the web page title, which allows the model to learn the relationship between short texts in information retrieval. The query text is the first sentence, and the title is the second sentence. The model needs to predict the relationship between the two sentences. "0" represents a strong association, that is, the user entered the query text and clicked on the title. "1" represents a weak association, that is, the user entered the query and the title appeared in the search results, but the user did not click on it. "2" means that the query text and the title are random in semantic information and are completely unrelated.
[0104] The newly constructed pre-training task type can be seamlessly added to the training framework to continuously carry out semantic understanding learning. Through the newly added semantic tasks such as entity prediction, sentence causal relationship judgment, and article sentence structure reconstruction, the semantic understanding pre-training framework with TransformerEncoder as the skeleton obtains natural language information in multiple dimensions such as lexical, syntactic, and semantic from the training data, greatly enhancing the general semantic representation ability.
[0105] In some implementations of the embodiments of the present application, text feature extraction is performed on the text using a text feature extraction model, including:
[0106] S501: Convert the text into a token sequence through a tokenizer.
[0107] In an embodiment of the present application, a word segmenter corresponding to a semantic understanding model is loaded. After the text is input into the word segmenter, the word segmenter decomposes the text to output a group of word sequences. Each word or phrase in the word sequence is called a token, so the output of the word segmenter is a token sequence.
[0108] S502: Convert the token sequence into the input of the semantic understanding model through the word segmenter, so that the semantic understanding model outputs text features.
[0109] In an embodiment of the present application, the output of the word segmenter is input into a semantic understanding model for text feature extraction. For example, the output is input into the ERNIE 2.0 semantic understanding model with Transformer Encoder as the skeleton, and the input token sequence is processed through multi-level pre-training tasks to extract the deep semantic features of the text.
[0110] S105: Capture the feature relationship of text features through the second multi-head attention layer to obtain a text self-attention weight matrix.
[0111] In the embodiment of the present application, the first multi-head attention layer specifically refers to a multi-head attention layer for processing text features. The text visual self-attention weight matrix specifically represents a matrix of relationships between text features. Here, the present application introduces a self-attention mechanism after the text feature extraction model extracts text features to capture long-term dependencies in the text, thereby being able to extract rich text features.
[0112] In some implementations of the embodiments of the present application, the input dimension of the second multi-head attention layer is 768 and the number of attention heads is 16.
[0113] In addition, the specific implementation method can be found in the description of S103, and this application will not elaborate on it here.
[0114] S106: Convert the visual multi-head attention weight matrix and the text word self-attention weight matrix into fusion features.
[0115] In an embodiment of the present application, multimodal information is combined by combining the visual multi-head attention weight matrix and the text word self-attention weight matrix, and the combined features become fusion features.
[0116] For example, assuming that the PaddlePaddle framework is used, the paddle.concat() function can be used to concatenate the adjusted weight matrices in a specific dimension. The concatenation operation can be stacked in the feature dimension so that the subsequent model can consider both visual and textual information.
[0117] In some implementations of the embodiments of the present application, the input dimension of the first multi-head attention layer is 2048 and the number of attention heads is 16, and the input dimension of the second multi-head attention layer is 768 and the number of attention heads is 16. At this time, the dimension of the fused feature is 2×(768+2048)=5632.
[0118] S107: Outputting the recognition result of the website to be recognized based on the fusion features through the fully connected layer.
[0119] In an embodiment of the present application, the fully connected layer is used to map the fused features of different modalities to the final output result through cross-modal connection. The fused features already contain the deep semantic information of the text and image data, and the role of the fully connected layer is to integrate this information and convert it into classification results.
[0120] The main responsibility of the fully connected layer is to further process the features extracted by previous layers of the network, such as convolution and pooling layers, and map these features to specific category probabilities or numerical predictions through multiplication of weight matrices and addition of bias terms. In the task of identifying gambling websites, the fully connected layer can effectively integrate data of different modalities, such as text and images, to predict whether a website is a gambling website.
[0121] Specifically, the fully connected layer receives fused features as input, which may contain tens of thousands of feature vectors. Then, through a series of weight matrix operations and bias adjustments, the fully connected layer converts these feature vectors into the final classification result, which, for example, outputs a two-dimensional vector, in which one element represents the probability that the website is a "gambling website" and the other element represents the probability that the website is a "not a gambling website".
[0122] In the embodiment of the present application, the dimension of the fused feature is 5632. At this time, the fully connected layer consists of five linear layers, including:
[0123] The first linear layer has an input dimension of 5632 and an output dimension of 1024;
[0124] A second linear layer with an input dimension of 1024 and an output dimension of 512;
[0125] A third linear layer with input dimension 512 and output dimension 256;
[0126] A fourth linear layer with input dimension 256 and output dimension 128;
[0127] A fifth linear layer with input dimension 128 and output dimension 3.
[0128] Through the first linear layer to the fifth linear layer, the fused features are mapped to a three-dimensional result.
[0129] In some implementations of the embodiments of the present application, the recognition results include: it is a gambling website, it is not a gambling website, and it is impossible to determine whether it is a gambling website. Considering that in some cases, the characteristics of the website may not be obvious or there is ambiguity, directly judging it as "yes" or "no" may lead to misjudgment, so adding the "unable to identify" option can avoid this situation and improve the accuracy of the overall recognition results. At the same time, when the model cannot determine the nature of the website, selecting "unable to identify" can reduce the probability of misjudging legitimate websites as gambling websites and misjudging gambling websites as legitimate websites, further improving the overall reliability.
[0130] exist Figure 1 In the process shown, first, the image and text information of the website to be identified are obtained for subsequent feature extraction; then, the image is feature extracted by the image feature extraction model to capture the visual features of the image, for example, to extract the visual features of specific images such as gambling game icons and specific color combinations in gambling websites. By using the text feature extraction model to extract features from the text, the semantic information of the text can be captured, which is helpful to identify the commonly used words, phrases, synonyms, slang or specific text patterns in gambling websites. After that, the first multi-head attention layer and the second multi-head attention layer are introduced to capture the feature relationship between visual features and text features, respectively, so as to focus on key features more accurately. Then, the visual multi-head attention weight matrix and the text self-attention weight matrix are converted into fusion features, which realizes the effective fusion of image and text information. The fusion feature contains both the visual information of the image and the semantic information of the text, providing a richer basis for subsequent recognition. Finally, the fusion feature is processed by the fully connected layer, and the recognition result of the website to be identified is finally output. The fully connected layer can effectively convert the feature map extracted by the previous layer into a category probability or numerical prediction, so as to realize the accurate recognition of the gambling website.
[0131] In summary, this application achieves efficient identification of gambling websites without manual identification through the fusion of multimodal data, feature extraction and attention mechanism, generation of fusion features and processing of fully connected layers.
[0132] In addition, this application also has the following effects:
[0133] Improve recognition accuracy: This application realizes the progress of deep learning in multimodal data sources, comprehensively reflects the real characteristics of the website through the multimodal information of the website to be identified, applies the multimodal data technology of deep learning, and incorporates a variety of different data types and sources into the recognition process, which can obtain more comprehensive and accurate data, effectively reduce false positives and false negatives, and ensure that users are protected from threats related to gambling websites, thereby improving the accuracy and reliability of website identification.
[0134] Enhanced robustness: Even if cybercriminals continue to develop new technologies to circumvent detection methods based on a single information source, or try to circumvent detection methods, other information streams can still provide the evidence needed to identify gambling websites. It can be seen that the robustness of gambling site identification can be enhanced by utilizing multiple information streams through multimodal information fusion technology.
[0135] Improved resistance to evasion techniques: Even if cybercriminals use synonyms, slang, or other evasion techniques to evade text-based gambling website detection, multimodal information fusion techniques can be used to improve resistance to these evasion techniques by leveraging image features.
[0136] See also Figure 6 As shown, an embodiment of the present application provides a site identification device for multimodal information fusion, the device comprising:
[0137] Data acquisition module 601, used to acquire images and texts of the website to be identified;
[0138] An image feature extraction module 602 is used to extract image features from an image using an image feature extraction model to obtain visual features of a website to be identified;
[0139] A visual attention module 603, configured to capture the feature relationship of visual features through a first multi-head attention layer to obtain a visual self-attention weight matrix;
[0140] A text feature extraction module 604 is used to extract text features from text using a text feature extraction model to obtain text features of the website to be identified;
[0141] A text attention module 605 is used to capture the feature relationship of text features through a second multi-head attention layer to obtain a text self-attention weight matrix;
[0142] A feature fusion module 606, used to convert the visual multi-head attention weight matrix and the text word self-attention weight matrix into a fusion feature;
[0143] The website identification module 607 is used to output the identification result of the website to be identified based on the fusion features through the fully connected layer.
[0144] In some implementations of the embodiments of the present application, obtaining an image and text of a website to be identified includes:
[0145] Initiate a request to the website to be identified through the crawler Python library, and use the corresponding parsing method according to the code source of the website to be identified to parse the text corresponding to the website to be identified;
[0146] The browser is automatically controlled through the selenium library to access the website to be identified, and a screenshot of the website to be identified is taken after the access is successful to obtain the corresponding image of the website to be identified.
[0147] In some implementations of the embodiments of the present application, the image feature extraction model is a deep residual network ResNet50 model in which the average pooling layer and the fully connected layer are replaced with a pooling layer of size 1*1 after pre-training.
[0148] In some implementations of the embodiments of the present application, the ResNet50 model is trained in the following manner:
[0149] Obtain an image dataset and divide the image dataset into a training set and a validation set;
[0150] Perform image enhancement on the training set;
[0151] The image-enhanced training set is used to train the ResNet50 model. During the model training process, the adaptive moment estimation optimizer Adam optimizer is used for parameter optimization, and the cross entropy loss function is used as the loss function.
[0152] When the classification accuracy of the ResNet50 model for the validation set reaches the preset requirement, the trained ResNet50 model is obtained.
[0153] In some implementations of the embodiments of the present application, the text feature extraction model includes: a semantic understanding model and a word segmenter corresponding to the semantic understanding model.
[0154] In some implementations of the embodiments of the present application, text feature extraction is performed on the text using a text feature extraction model, including:
[0155] Convert the text into a sequence of tokens through a tokenizer;
[0156] The token sequence is converted into the input of the semantic understanding model through the tokenizer, so that the semantic understanding model outputs text features.
[0157] In some implementations of the embodiments of the present application, the identification results include: it is a gambling website, it is not a gambling website, and it is impossible to determine whether it is a gambling website.
[0158] In some implementations of the embodiments of the present application, the input dimension of the first multi-head attention layer is 2048 and the number of attention heads is 16, and the input dimension of the second multi-head attention layer is 768 and the number of attention heads is 16.
[0159] In some implementations of the embodiments of the present application, the dimension of the fused feature is 5632, and the fully connected layer includes:
[0160] The first linear layer has an input dimension of 5632 and an output dimension of 1024;
[0161] A second linear layer with an input dimension of 1024 and an output dimension of 512;
[0162] A third linear layer with input dimension 512 and output dimension 256;
[0163] A fourth linear layer with input dimension 256 and output dimension 128;
[0164] A fifth linear layer with input dimension 128 and output dimension 3.
[0165] Finally, it should be noted that, in the embodiments of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the statement "comprise one..." do not exclude the presence of other identical elements in the process, method, article or equipment including the elements.
[0166] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A site identification method based on multimodal information fusion, characterized in that: The method comprises: Obtain images and texts of the website to be identified; Extracting image features from the image using an image feature extraction model to obtain visual features of the website to be identified; Capturing the feature relationship of the visual features through a first multi-head attention layer to obtain a visual self-attention weight matrix; Performing text feature extraction on the text by using a text feature extraction model to obtain text features of the website to be identified; Capturing the feature relationship of the text features through a second multi-head attention layer to obtain a text self-attention weight matrix; Converting the visual multi-head attention weight matrix and the text word self-attention weight matrix into fusion features; The recognition result of the website to be recognized based on the fusion feature is output through a fully connected layer.
2. The method according to claim 1, characterized in that The step of obtaining the image and text of the website to be identified includes: Initiate a request to the website to be identified through the crawler Python library, and use a corresponding parsing method according to the code source of the website to be identified to parse out the text corresponding to the website to be identified; The browser is automatically controlled through the selenium library to access the website to be identified, and a screenshot of the website to be identified is taken after the access is successful to obtain the picture corresponding to the website to be identified.
3. The method according to claim 1, characterized in that The image feature extraction model is a deep residual network ResNet50 model in which the average pooling layer and the fully connected layer are replaced with a pooling layer of size 1*1 after pre-training.
4. The method according to claim 3, characterized in that The ResNet50 model is trained in the following way: Acquire an image dataset, and divide the image dataset into a training set and a validation set; Performing image enhancement on the training set; The image-enhanced training set is used to perform model training of the ResNet50 model, and during the model training process, an adaptive moment estimation optimizer Adam optimizer is used to perform parameter optimization, and a cross entropy loss function is used as a loss function; When the classification accuracy of the ResNet50 model for the validation set reaches the preset requirement, a trained ResNet50 model is obtained.
5. The method according to claim 1, characterized in that The text feature extraction model includes: a semantic understanding model and a word segmenter corresponding to the semantic understanding model.
6. The method according to claim 5, characterized in that The step of extracting text features from the text using a text feature extraction model includes: Converting the text into a sequence of tokens by the tokenizer; The token sequence is converted into the input of the semantic understanding model through the word segmenter, so that the semantic understanding model outputs the text feature.
7. The method according to claim 1, characterized in that The identification results include: it is a gambling website, it is not a gambling website, and it is impossible to determine whether it is a gambling website.
8. The method according to claim 1, characterized in that: The input dimension of the first multi-head attention layer is 2048 and the number of attention heads is 16, and the input dimension of the second multi-head attention layer is 768 and the number of attention heads is 16.
9. The method according to claim 8, characterized in that The dimension of the fusion feature is 5632, and the fully connected layer includes: The first linear layer has an input dimension of 5632 and an output dimension of 1024; A second linear layer with an input dimension of 1024 and an output dimension of 512; A third linear layer with input dimension 512 and output dimension 256; A fourth linear layer with input dimension 256 and output dimension 128; A fifth linear layer with input dimension 128 and output dimension 3.
10. A site identification device for multi-modal information fusion, characterized in that: The device comprises: A data acquisition module, used to acquire images and texts of the website to be identified; An image feature extraction module, used to extract image features from the image using an image feature extraction model to obtain visual features of the website to be identified; A visual attention module, configured to capture the feature relationship of the visual features through a first multi-head attention layer to obtain a visual self-attention weight matrix; A text feature extraction module, used to extract text features from the text using a text feature extraction model to obtain text features of the website to be identified; A text attention module, used for capturing the feature relationship of the text features through a second multi-head attention layer to obtain a text self-attention weight matrix; A feature fusion module, used for converting the visual multi-head attention weight matrix and the text word self-attention weight matrix into a fusion feature; The website identification module is used to output the identification result of the website to be identified based on the fusion feature through a fully connected layer.
Citation Information
Patent Citations
Text and image-based multi-modal pornography and gambling domain name automatic detection method
CN115277211A
Cited By
Method and device for identifying risk website
CN119720089A
Method and apparatus for identifying a risky website
CN119720089B
Phishing website detection method and system based on multi-modal fusion
CN119835069A
A phishing website detection method and system based on multimodal fusion
CN119835069B
Multi-modal large model-based ship and ocean engineering aided design method
CN120070673A
