Website Sensitive Content Classification Methods, Systems, Devices, and Media Based on Multimodal Feature Fusion

Abstract

A method, system, device, and medium for classifying sensitive website content based on multimodal feature fusion are described. The method involves: first, acquiring a multimodal dataset of website content; second, constructing a SEResNet-Transformer-based classification model for sensitive website image content using the acquired dataset; third, constructing a website sensitive content classification model based on multimodal feature fusion; and finally, obtaining the classification result of sensitive website content based on the multimodal feature fusion model. The system, device, and medium classify sensitive website content based on multimodal feature fusion. This invention utilizes the local features extracted by SEResNet in the preliminary features, reducing the computational parameters of the overall model. Furthermore, the Transformer module employs a self-attention mechanism to learn global image information, improving the accuracy of illegal image classification. Multimodal feature fusion combines textual and image information features from illegal websites into a global feature vector, further improving classification accuracy.

Description

Technical Field

[0001] This invention belongs to the field of image processing and analysis technology, and in particular relates to a method, system, device and medium for classifying sensitive website content based on multimodal feature fusion. Background Technology

[0002] Users access the internet primarily through mobile applications and web pages. Mobile internet applications are booming, with 2.6 million active apps monitored in the domestic market, further covering users' daily learning, work, and life, and continuously strengthening the foundation for the Internet of Things. On the web page side, my country currently has approximately 3.87 million websites. Although this number is declining year by year, many online services are still provided through websites. The declining number of websites reflects a decrease in the number of management and maintenance personnel, and some websites have even been abandoned but not deregistered. This has led to a large number of poorly maintained websites being illegally used and transformed into illegitimate sites. Simultaneously, a large number of illegal overseas websites exist, and because their servers are located overseas, this further increases the difficulty of dealing with them and the harm they cause to society.

[0003] During routine maintenance of the campus website, the following phenomenon has been observed: Existing legitimate domain name links have been widely published on the campus intranet. When these legitimate domains, such as international conference domains like www.****.org, cease to be renewed after fulfilling their purpose or providing website services, they are illegally re-registered by some individuals for use on illegal websites. This exploitation of the university website to promote illegal sites damages the positive image of educational institutions and has a very negative impact. Once a domain is used by an illegal website, the telecommunications security department detects it and demands immediate removal; otherwise, the school's telecommunications outage will be cut off, leaving administrators in a very passive position. Currently, administrators can only manually handle each illegal domain discovered by the telecommunications department. To more thoroughly and proactively maintain the campus network environment, it is necessary to propose a method that can accurately detect illegal websites and proactively identify those existing on the school's website pages.

[0004] The closest implementation to this invention is a sensitive information detection method based on a Graph Convolutional Network (GCN) model proposed by Ying Liu et al. in 2021. This method uses a pre-trained model to encode keywords in a given document and corpus. Secondly, it applies a graph attention network to extract structural and textual information, which is then used to form a classification. This method only performs sensitive information detection on textual information within websites.

[0005] Other implementations include a deep single-classification algorithm (DOCAPorn) based on visual attention mechanisms proposed by Junren Chen et al. in 2020 for identifying illegal images. This deep learning-based single-classification avoids the problem of an infinite number of negative sample types caused by only recognizing the target object, thus avoiding the problem of insufficient negative samples. Furthermore, introducing visual attention mechanisms into deep single-classification makes the neural network more focused on the target object. The proposed method improves the accuracy of identifying illegal information in images. However, this method only shows good classification results for illegal and sensitive content images, and simply using attention mechanisms lacks attention to local information.

[0006] Patent application CN107862050A, entitled "A Website Content Security Detection System and Method," uses image information crawled from web pages as a classification basis and employs an FPGA hardware accelerator to accelerate the training of an image classification model, achieving fast and accurate website content security detection. However, this method has limitations as it only classifies image content within websites. Summary of the Invention

[0007] To overcome the shortcomings of the existing technologies, the present invention aims to propose a website sensitive content classification method, system, device, and medium based on multimodal feature fusion. The method utilizes the preliminary features extracted by SEResNet, which include local features, reducing the computational parameters of the overall model. Then, through the Transformer module, a self-attention mechanism is used to learn global information of the image, improving the accuracy of illegal image classification. Furthermore, multimodal feature fusion combines textual and image information features from illegal websites into a global feature vector, further enhancing classification accuracy.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A website sensitive content classification method based on multimodal feature fusion specifically includes the following steps:

[0010] Step 1: Obtain the multimodal dataset of website content;

[0011] Step 2: Construct a website sensitive image content classification model based on SEResNet-Transformer using the dataset obtained in Step 1;

[0012] Step 3: Based on the multimodal dataset obtained in Step 1 and the SEResNet-Transformer-based website sensitive image content classification model in Step 2, construct a website sensitive content classification model based on multimodal feature fusion.

[0013] Step 4: Input the multimodal dataset of website content constructed in Step 1 into the website sensitive content classification model based on multimodal feature fusion constructed in Step 3 to obtain the website sensitive content classification results.

[0014] The specific method for step 1 is as follows:

[0015] Step 1.1 Data Acquisition

[0016] Step 1.1a Chinese text content collection

[0017] First, use automated testing tools to simulate the process of a browser accessing the webpage. While scrolling down the webpage, wait for the webpage to finish rendering and the script to finish loading. Then, collect the webpage's HTML source code to obtain the HTML source code of the content actually displayed on the webpage. Next, use regular expression matching to collect the Chinese text content.

[0018] Step 1.1b Image Data Acquisition

[0019] According to the HTML source code The `src` or `data-origin` attribute in the `` tag can be used to extract image content from the page.

[0020] Step 1.2 Data Processing

[0021] Step 1.2a: Data processing of text content:

[0022] The jieba word segmentation library in Python is used to segment web page text data. Stop word mining is performed on the segmented results to obtain a stop word list that is unique to the web page text data but is not related to the actual content of the web page. After obtaining the stop word list unique to the web page text data, the stop words in the segmented results are deleted.

[0023] Step 1.2b: Data processing of image content:

[0024] First, filter out image files ending with JPG, JPEG, PNG, and GIF, and remove other image files with incorrect formats;

[0025] Secondly, images with both height and width below the pixel threshold are selected and removed; GIF animations are first decoded into a series of frames, then converted frame by frame into JPG files, and the images are scaled to a uniform size; thus, the datasets of the website's text and image modalities are obtained.

[0026] The process of mining stop words from web page text data is as follows: First, each word after segmenting the web page text data using Python's jieba word segmentation library is encoded using TF-IDF (Term Frequency-Inverse Document Frequency). TF-IDF encoding is used to evaluate the importance of a word in a text dataset; that is, the more times a word appears in a document, the more important it is. However, the more documents a word appears in a dataset, the less important it is. Based on this, the TF-IDF encoding values ​​of each word are sorted from low to high, that is, arranged according to the importance of each word in the dataset from low to high. The top n (n≥100) words are extracted to obtain a stop word list that is high-frequency but irrelevant to the actual content of the web page text data. After obtaining the stop word list that is unique to the web page text data, the stop words in the word segmentation results are deleted.

[0027] Step 2 specifically includes the following steps:

[0028] Step 2.1, Construct a preliminary feature extraction module based on the SEResNet model:

[0029] First, the ResNet-50 backbone network structure is improved to adjust the dimension of the output feature vector;

[0030] Then, the compression operation is performed using the SE module, which includes compression and excitation components: Let the feature vector size be represented by H*W, and the number of feature vector channels be C. The formula for the compression part of the SE module is as follows:

[0031]

[0032] Among them, u c For each feature vector channel of size H*W, a global average pooling operation is performed to obtain a z-axis of size 1*1. c Pixel; A feature vector of H*W*C is compressed to obtain a feature vector of 1*1*C.

[0033] The SE module's activation part operates as follows: The compressed 1*1*C feature vector is first reduced to a single dimension using a fully connected layer. The vector is then input into a ReLU activation layer; then passed through a fully connected layer... The vector is redimensionalized into a 1*1*C vector, and then input into a Sigmoid activation layer; the formula for the activation part of the SE module is:

[0034] F ex(z,W)=σ(g(z,W))=Sigmoid(W2*ReLU(W1z))

[0035] Finally, the channel attention vector is multiplied by the feature vector to calibrate the feature vector, resulting in a feature vector with added channel attention:

[0036] F scale (u c ,s c ) = s c u c

[0037] Step 2.2: Use the global feature focusing module based on the Transformer module to focus the global features of the website image:

[0038] Based on the preliminary feature extraction module of the SEResNet model constructed in step 2.1, the extracted preliminary image features are first added with a class token, and then with a positional encoding. Subsequently, they are processed by a global feature focusing module composed of multiple Transformer Encoder modules. The Transformer Encoder modules do not change the input and output sizes, the input features and output features have the same dimension, and multiple superimposed output features remain unchanged. Finally, the class token is extracted from the output feature vector and passed to the MLP (Multilayer Perceptron) and Softmax function, which are tools for converting feature vectors into classification results.

[0039] The method for improving the ResNet-50 backbone network structure is as follows: Stage-1 in the ResNet-50 network is stacked 3 times, Stage-2 is stacked 4 times, and Stage-3 is stacked 9 times, with the downsampling rate set to 16.

[0040] The specific method for step 3 is as follows:

[0041] After removing stop words from the text modal data in the multimodal dataset obtained in step 1, the text modal feature vectors are input into the ALBERT text classification model to obtain text modal feature vectors. The feature vectors transformed in step 2.2 are input into the preliminary feature extraction module and the Transformer Encoder module of the SEResNet model. After removing the Transformer Encoder module, the MLP multilayer perceptron and Softmax function are used to extract image modal feature vectors. Finally, the obtained text modal feature vectors and image modal feature vectors are concatenated and input into the fully connected layer and the Softmax function to output the multimodal (image + text) classification results, thus constructing a website sensitive content classification model based on multimodal feature fusion.

[0042] The specific method for step 4 is as follows:

[0043] Input the multimodal dataset of website content constructed in step 1 into the website sensitive content classification model based on multimodal feature fusion proposed in step 3, and output the multimodal (image + text) classification results. Obtain the website sensitive content classification results from the output multimodal (image + text) classification results; use accuracy, precision, recall and F1 score to comprehensively evaluate the website sensitive content classification results.

[0044] A website sensitive content classification system based on multimodal feature fusion includes:

[0045] The image acquisition and preprocessing module is used in step 1 to perform stop word mining on the web page text data, remove stop words unique to the web page text data and common Chinese stop words; filter the image content data to remove image files with incorrect formats, obtain image data that meets the requirements, and perform uniform scaling; obtain the dataset of the website's text modality and image modality.

[0046] The preliminary feature extraction module of the SEResNet model is used to extract preliminary image features in step 2.1. It can obtain preliminary feature information of the image to be classified after compression and activation operations on the input image to the SEResNet network.

[0047] The global feature focusing module based on the Transformer module is used to gather global image features in step 2.2, and learn global information of the image using a self-attention mechanism to improve the accuracy of illegal image classification.

[0048] The multimodal feature fusion module is used to fuse image feature vectors and text feature vectors in step 3. It can combine text information features and image information features from illegal websites into a global feature vector, thereby complementing the feature information between different modalities, obtaining more comprehensive global features, and thus improving the accuracy of classification.

[0049] An electronic device for classifying sensitive website content based on multimodal feature fusion includes: a memory and a processor, wherein the memory stores a computer program that causes the processor to execute the method for classifying sensitive website content based on multimodal feature fusion in any one of steps 1 to 4.

[0050] A user input program storage medium, wherein the stored computer program, when executed by a processor, is capable of classifying sensitive website content based on the multimodal feature fusion method described in any one of claims 1 to 7, thereby classifying sensitive website content based on multimodal feature fusion.

[0051] Compared with the prior art, the present invention has the following advantages:

[0052] (1) This invention uses image classification technology, extracts primary features of images through the SEResNet model, learns global information of images through the Transformer module, and finally outputs classification results through a fully connected neural network and the Softmax function, which solves the problem that the key regions of illegal sensitive images have a large spatial distribution scale and lack regularity, thereby improving the accuracy of illegal sensitive image classification.

[0053] (2) This method utilizes the local features extracted by SEResNet in the preliminary features and reduces the sequence length of the subsequent input Transformer module, thereby reducing the computational parameters of the overall model.

[0054] (3) This invention uses the Transformer module to learn global information of images through a self-attention mechanism, thereby improving the accuracy of illegal image classification.

[0055] (4) The present invention uses a multimodal feature fusion method to combine the text information features and image information features on illegal websites into a global feature vector, thereby complementing the feature information between different modalities, obtaining more comprehensive global features, and thus improving the accuracy of classification.

[0056] (5) Compared with the sensitive information detection method based on graph convolutional network model, the present invention analyzes the information contained in the website more comprehensively. In addition to text information, it also incorporates image information into the classification process. The method of multimodal feature fusion is used to fuse text content and image content, thereby improving the accuracy of illegal website classification.

[0057] (6) Compared with deep single-classification methods based on visual attention mechanisms, this method only has a good classification effect on illegal and sensitive content images, and the use of attention mechanism alone lacks attention to local information. In this invention, the key regions of illegal and sensitive images have a large spatial distribution scale and lack regularity. The invention proposes the SEResNet-Transformer model. First, the image to be classified is input into the SEResNet network to obtain the preliminary feature information of the image. Then, the feature vector is input into the Transformer module to learn the global information of the image using the self-attention mechanism. Finally, the classification result is output through a fully connected neural network and the Softmax function, which more comprehensively analyzes the information contained in the website. In addition to image information, text information is also included in the classification process. The multimodal feature fusion method is used to fuse text content and image content, thereby improving the accuracy of illegal website classification. Attached Figure Description

[0058] Figure 1 This is a flowchart of the present invention.

[0059] Figure 2This is the structural diagram of the multi-modal feature fusion module of the present invention.

[0060] Figure 3 This is the schematic structural diagram of the SE module of the present invention.

[0061] Figure 4 This is the structural diagram of the preliminary feature extraction module based on the SEResNet model of the present invention.

[0062] Figure 5 This is the structural diagram of the global feature focusing module based on the Transformer module of the present invention. Detailed implementation manners

[0063] The following further describes the present invention in detail with reference to the accompanying drawings and specific embodiments.

[0064] See Figure 1 、 Figure 2 A website sensitive content classification method based on multi-modal feature fusion specifically includes the following steps:

[0065] Step 1, obtain the multi-modal data set of the website content;

[0066] The present invention uses web crawler technology and data processing technology to obtain the multi-modal data set of the website content. The adopted crawler framework is the WebMagic framework implemented in Java language.

[0067] 1.1 Data collection

[0068] 1.1a Collection of Chinese text content

[0069] First, use the automated testing tool selenium to simulate the process of the browser accessing the web page. During the process of scrolling down the web page, wait for the web page to complete rendering and the JavaScript script to be loaded completely, and then collect the HTML source code of the web page to obtain the HTML source code of the content actually displayed on the web page. Then, use the method of regular expression matching to collect Chinese text content. The Unicode encoding range of Chinese characters is [\u4E00-\u9FA5], where \u4E00 represents the first Chinese character "one", and \u9FA5 represents the last Chinese character "龥". If the text is encoded in other formats, first convert it to Unicode encoding, and then match the characters within the range of [\u4E00-\u9FA5] through regular expressions to obtain all Chinese text content in the HTML source code.

[0070] 1.1b Collection of image data

[0071] The method for collecting image data is as follows: according to the The src or data-origin attribute in the tag can be used to extract image content from the page.

[0072] 1.2 Data Processing

[0073] 1.2a Data processing of text content:

[0074] The text content data processing involves first segmenting the text data into words, and then removing stop words from the segmentation results. The stop word mining process for webpage text data involves first using Python's jieba segmentation library to segment the webpage text data into words. Then, each word is encoded using TF-IDF (Term Frequency-Inverse Document Frequency). TF-IDF encoding is primarily used to evaluate the importance of a word in a text dataset. The idea is that the more frequently a word appears in a document, the more important it is; however, the more documents a word appears in a dataset, the less important it becomes. Therefore, the TF-IDF encoding values ​​of each word are sorted from low to high, i.e., arranged according to the importance of each word in the dataset from low to high. The top 100 words are then used to obtain a preliminary stop word list. After manual screening, a final stop word list is obtained that represents frequently occurring words that are irrelevant to the actual content of the webpage text data. After obtaining the stop word list unique to the webpage text data, the stop words unique to the webpage text data and common Chinese stop words are removed from the results of text data segmentation to obtain the website's text modality dataset. After processing, the irrelevant information in the text data is reduced, thereby improving the quality of the webpage text data.

[0075] 1.2b Data processing of image content:

[0076] The image content data processing procedure involves first filtering image files ending in JPG, JPEG, PNG, and GIF, and discarding other image files with incorrect formats. Secondly, illegal websites often contain icons or decorative images, which are typically small in height and width; images with both height and width less than 50 pixels are then removed. GIF animations are prevalent on illegal websites, often used in banner ads, attracting users to click through images that display exaggerated colors or text in rapid succession. Therefore, GIF animations need to be decoded into a series of frames, then converted frame by frame to JPG format files, and the images scaled to 244*244 pixels to obtain a dataset of the website's image modalities.

[0077] Step 2: Construct a website sensitive image content classification model S based on SEResNet-Transformer. This includes the following steps:

[0078] Step 2.1: Construct a preliminary feature extraction module based on the SEResNet model.

[0079] First, using ResNet-50 as the backbone network requires adjustments to its structure to change the dimension of the output feature vector. In the original ResNet-50 network, Stage-1 was stacked 3 times, Stage-2 was stacked 4 times, Stage-3 was stacked 6 times, and Stage-4 was stacked 3 times, with a downsampling rate of 32. In the improved ResNet-50, three blocks from Stage-4 were moved to Stage-3, and the downsampling rate was adjusted to 16, resulting in an output feature vector dimension of 14*14*1024.

[0080] Then, the compression operation is performed using the SE module, which includes compression and excitation sections (see...). Figure 3 Let the feature map size be represented by H*W, and the number of feature map channels be C. Then the formula for the compression part of the SE module is as follows:

[0081]

[0082] Among them, u c For each feature map channel of size H*W, a global average pooling operation is performed to obtain a z-axis of size 1*1. c Pixels. Overall, an H*W*C feature map is compressed to obtain a 1*1*C feature vector.

[0083] The SE module's activation part operates as follows: The compressed 1*1*C feature vector is first reduced to a single dimension using a fully connected layer. The vector is then input into a ReLU activation layer; then passed through a fully connected layer... The vector is redimensionalized into a 1*1*C vector, and then input into a Sigmoid activation layer; the formula for the activation part of the SE module is:

[0084] F ex (z,W)=σ(g(z,W))=Sigmoid(W2*ReLU(W1z))

[0085] Finally, the channel attention vector is multiplied by the feature map to calibrate the feature vector, resulting in a feature vector with added channel attention:

[0086] F scale (u c ,s c ) = s c u c

[0087] Step 2.2: Use the global feature focusing module based on the Transformer module to focus the global features of the website image:

[0088] Based on the preliminary feature extraction module of the SEResNet model constructed in step 2.1, the preliminary image features extracted are shown in (see...). Figure 4 First, a class token with a dimension of 1*768 is added, followed by positional encoding. Then, a global feature focusing module consisting of 12 Transformer Encoder modules is used. The input and output features of each Transformer Encoder module have the same dimension; therefore, the input features are 197*768, and the output features from multiple stacked modules remain 197*768. Finally, the class token is extracted from the output feature slices and fed into the MLP (Multilayer Perceptron) and Softmax function, which convert feature vectors into classification results, outputting the image classification result (see [link to documentation]). Figure 5 ).

[0089] Step 3, construct a website sensitive content classification model S based on multimodal feature fusion:

[0090] After removing stop words from the text data in the multimodal dataset obtained in step 1, input it into the ALBERT text classification model to obtain text feature vectors. Then, input the feature vectors transformed in step 2.2 into the preliminary feature extraction module and Transformer Encoder module of the SEResNet model. After removing the Transformer Encoder module, use the MLP multilayer perceptron and Softmax function to extract image feature vectors. Finally, concatenate the obtained text modal feature vectors and image modal feature vectors and input them into the fully connected layer and Softmax function to output the multimodal (image + text) classification result, thus constructing a website sensitive content classification model based on multimodal feature fusion.

[0091] Step 4, obtain the website's sensitive content classification results:

[0092] Input the multimodal dataset of website content constructed in step 1 into the website sensitive content classification model based on multimodal feature fusion proposed in step 3, and output the multimodal (image + text) classification results. Obtain the website sensitive content classification results from the output multimodal (image + text) classification results; use accuracy, precision, recall and F1 score to comprehensively evaluate the website sensitive content classification results.

[0093] The following experimental results further illustrate the technical effects of this invention.

[0094] The results of the text modality experiment are shown in Table 1.1 below:

[0095] Table 1.1 Experimental results of different text modal models

[0096]

[0097] The experimental results show that the ALBERT model outperforms other text classification models in terms of accuracy, precision, recall, and F1 score. Therefore, the ALBERT model will be used as the text modality classification model in subsequent multimodal fusion experiments.

[0098] The results of the image modality experiment are shown in Table 1.2 below:

[0099] To verify the effectiveness of the method in this invention, VGGNet, ResNet, SEResNet, EfficientNet, and Vision Transformer models were selected as comparison objects. Specifically, VGGNet and ResNet were selected as VGG16 and ResNet50 models, respectively, as classic convolutional neural networks and benchmark models for many tasks. SEResNet is based on ResNet with channel attention; this invention verifies how adding channel attention to convolutional neural networks improves model classification performance. EfficientNet is a convolutional neural network that searches and scales parameters in the model. Vision Transformer is an image classification model that uses a global attention mechanism. Therefore, this invention uses the above models as comparison objects, and the specific experimental results are shown in Table 1.2 below:

[0100] Table 1.2 Experimental results of different image modal models

[0101]

[0102] Experimental results show that the SEResNet model improves the F1 score by 3.11%, 2.38%, and 0.99% compared to VGG, ResNet, and EfficientNet models, respectively. This demonstrates that the SEResNet model outperforms the other three models in classifying website image modal datasets. The proposed SEResNet-Transformer model uses SEResNet to extract local image features, achieving a 1.66% improvement in F1 score compared to the Vit model, which also uses multi-head self-attention. This indicates that using SEResNet to extract local image features can reduce interference from irrelevant information to some extent, slightly improving classification performance, ultimately validating the effectiveness of the proposed model. Therefore, the SEResNet-Transformer model will be used as the image modality classification model in subsequent multimodal fusion experiments.

[0103] Multimodal fusion experiment

[0104] The specific experimental results are shown in Table 1.3 below:

[0105] Table 1.3 Experimental results of different multimodal models

[0106]

[0107]

[0108] The experimental results from the combined text modality model, image modality model, and multimodal model show that the text modality model outperforms the image modality model in all cases, and the multimodal model outperforms the unimodal model in all cases. Comparing different multimodal fusion models, the multimodal feature fusion model slightly outperforms the multimodal decision fusion model, with an accuracy improvement of 0.44% and an F1 score improvement of 0.42%. Therefore, the multimodal feature fusion model proposed in this invention outperforms other models in classifying website content datasets, ultimately validating the effectiveness of the proposed model.

Claims

1. A website sensitive content classification method based on multimodal feature fusion, characterized in that, Specifically, the following steps are included: Step 1: Obtain the multimodal dataset of website content; Step 2: Construct a website sensitive image content classification model based on SEResNet-Transformer using the dataset obtained in Step 1; Step 2.1, Construct a preliminary feature extraction module based on the SEResNet model: First, the ResNet-50 backbone network structure is improved to adjust the dimension of the output feature vector; Then, compression is performed using the SE module, which includes compression and excitation components: the feature vector size is reduced using... This indicates that the number of feature vector channels is The formula for the compression part of the SE module is as follows: in, For each size The feature vector channels are processed by global average pooling to obtain a size of [value]. of pixel; one The feature vectors are obtained after compression. eigenvectors; SE module excitation section operation: The compressed result... The feature vectors are first processed through a fully connected layer. Dimensionality reduction of feature vectors into a The vector is then input into a ReLU activation layer; then passed through a fully connected layer... The vector is redimensionalized into a single dimension. The vector is then input into a sigmoid activation layer; the formula for the activation part of the SE module is: Finally, the channel attention vector is multiplied by the feature vector to calibrate the feature vector, resulting in a feature vector with added channel attention: Step 2.2: Use the global feature focusing module based on the Transformer module to focus the global features of the website image: Based on the preliminary feature extraction module of the SEResNet model constructed in step 2.1, the extracted preliminary image features are first added with a class token, and then with a positional encoding. Subsequently, they are processed by a global feature focusing module composed of multiple Transformer Encoder modules. The Transformer Encoder modules do not change the input and output sizes, the input features and output features have the same dimension, and multiple superimposed output features remain unchanged. Finally, the class token is extracted from the output feature vector and passed to the MLP (Multilayer Perceptron) and Softmax function, which are tools for converting feature vectors into classification results. Step 3: Based on the multimodal dataset obtained in Step 1 and the SEResNet-Transformer-based website sensitive image content classification model in Step 2, construct a website sensitive content classification model based on multimodal feature fusion; specifically: After removing stop words from the text modal data in the multimodal dataset obtained in step 1, the text modal feature vectors are input into the ALBERT text classification model to obtain text modal feature vectors. The feature vectors transformed in step 2.2 are input into the preliminary feature extraction module and the Transformer Encoder module of the SEResNet model. After removing the Transformer Encoder module, the MLP multilayer perceptron and Softmax function are used to extract image modal feature vectors. Finally, the obtained text modal feature vectors and image modal feature vectors are concatenated and input into the fully connected layer and the Softmax function to output the multimodal classification result of image + text, thus constructing a website sensitive content classification model based on multimodal feature fusion. Step 4: Input the multimodal dataset of website content obtained in Step 1 into the website sensitive content classification model based on multimodal feature fusion constructed in Step 3 to obtain the website sensitive content classification results.

2. The website sensitive content classification method based on multimodal feature fusion according to claim 1, characterized in that, The specific method for step 1 is as follows: Step 1.1 Data Acquisition Step 1.1a Chinese text content collection First, use automated testing tools to simulate the process of a browser accessing a webpage. While scrolling down the webpage, wait for the webpage to finish rendering and the script to finish loading. Then, collect the webpage's HTML source code to obtain the HTML source code of the content actually displayed on the webpage. Then, regular expression matching is used to collect Chinese text content; Step 1.1b Image Data Acquisition According to the HTML source code The `src` or `data-origin` attribute in the `` tag can be used to extract image content from the page. Step 1.2 Data Processing Step 1.2a: Data processing of text content: The jieba word segmentation library in Python is used to segment web page text data. Stop word mining is performed on the segmented results to obtain a stop word list that is unique to web page text data that appears frequently but is not related to the actual content of the web page. After obtaining the stop word list unique to the webpage text data, the stop words are removed from the word segmentation results to obtain the website's text modality dataset; 1.2b Data processing of image content: First, filter out image files ending with JPG, JPEG, PNG, and GIF, and remove other image files with incorrect formats; Secondly, images whose height and width are both below the pixel threshold are selected and removed; GIF animations are first decoded into a series of frames, then converted into JPG files frame by frame, and the images are scaled to a uniform size to obtain the image modality dataset of the website.

3. The website sensitive content classification method based on multimodal feature fusion according to claim 2, characterized in that, The process of mining stop words from web page text data is as follows: First, each word after segmenting the web page text data using Python's jieba word segmentation library is encoded using TF-IDF. TF-IDF encoding is used to evaluate the importance of a word in a text dataset; that is, the more times a word appears in a document, the more important it is. However, the more documents a word appears in a dataset, the less important it is. Accordingly, the TF-IDF encoding values ​​of each word are sorted from low to high, that is, arranged according to the importance of each word in the dataset from low to high. The top n words are extracted, where n≥100, thus obtaining a stop word list unique to the web page text data that appears frequently but is irrelevant to the actual content of the web page. After obtaining the stop word list unique to the web page text data, the stop words in the word segmentation results are deleted.

4. The website sensitive content classification method based on multimodal feature fusion according to claim 1, characterized in that, The method for improving the ResNet-50 backbone network structure is as follows: Stage-1 in the ResNet-50 network is stacked 3 times, Stage-2 is stacked 4 times, and Stage-3 is stacked 9 times, with the downsampling rate set to 16.

5. The website sensitive content classification method based on multimodal feature fusion according to claim 1, characterized in that, The specific method for step 4 is as follows: The multimodal dataset of website content constructed in step 1 is input into the website sensitive content classification model based on multimodal feature fusion proposed in step 3, and the multimodal classification result is output. The website sensitive content classification result is obtained from the output multimodal classification result. The accuracy, precision, recall and F1 score are used to comprehensively evaluate the output website sensitive content classification result.

6. A website sensitive content classification system based on multimodal feature fusion, used to implement the website sensitive content classification method based on multimodal feature fusion as described in any one of claims 1 to 5, characterized in that, include: The image acquisition and preprocessing module is used in step 1 to mine stop words in the web page text data and remove stop words unique to the web page text data and common Chinese stop words. The image content data is filtered to remove image files with incorrect formats, and image data that meets the requirements is obtained and then uniformly scaled. Obtain the datasets of the website's text and image modalities; The preliminary feature extraction module of the SEResNet model is used to extract preliminary image features in step 2.

1. It can obtain preliminary feature information of the image to be classified after compression and activation operations on the input image to the SEResNet network. The global feature focusing module based on the Transformer module is used to gather global image features in step 2.2, and learn global information of the image using a self-attention mechanism to improve the accuracy of illegal image classification. The multimodal feature fusion module is used to fuse image feature vectors and text feature vectors in step 3. It can combine text information features and image information features from illegal websites into a global feature vector, thereby complementing the feature information between different modalities, obtaining more comprehensive global features, and thus improving the accuracy of classification.

7. An electronic device for classifying sensitive website content based on multimodal feature fusion, characterized in that, include: A memory and a processor, the memory storing a computer program that causes the processor to execute the method for classifying sensitive website content by multimodal feature fusion as described in any one of claims 1 to 5.

8. A storage medium for receiving user input programs, characterized in that, When the stored computer program is executed by a processor, it is able to classify sensitive website content based on the multimodal feature fusion method described in any one of claims 1 to 5.

Images