A multi-modal fake news detection method based on unsupervised clustering and frequency domain information

Abstract

The application provides a multi-modal fake news detection method based on unsupervised clustering and frequency domain information, and relates to the technical field of image and text data. The method comprises the following steps: based on multi-modal sample data, text features, visual features and image text features are extracted respectively, text feature clustering labels are obtained according to the global semantic correlation of the text features, the text features with labels are input into an unsupervised clustering learning network, the semantic consistency is enhanced through a bidirectional gate recurrent unit and a multi-head attention mechanism, and enhanced text features are obtained; frequency domain features are extracted based on the visual features, and visual joint features are obtained by fusing the frequency domain features and spatial domain features; the image text features are also enhanced through the unsupervised clustering learning network, and are used as a bridge to fuse the text features and the visual features through an attention fusion module, multi-modal fusion features are obtained, and the multi-modal fusion features are input into a full connection layer to complete fake news classification. The method effectively improves the accuracy of fake news detection.

Description

Technical Field

[0001] This application relates to the field of image and text data technology, and in particular to a multimodal fake news detection method based on unsupervised clustering and frequency domain information. Background Technology

[0002] With the rapid development of the internet, social media platforms have become indispensable tools for information sharing and acquisition in daily life. Their openness allows users to independently publish information and personal opinions. However, this convenience of information freedom has also brought about a problem that cannot be ignored: a large amount of unverified and biased fake news appears and spreads rapidly on social media platforms, which can easily mislead readers and even pose a serious threat to social stability and development. If the authenticity of news information published on social media platforms can be verified in real time to ensure its accuracy and credibility, it will be of great significance for maintaining social stability and long-term development.

[0003] Previous fake news detection techniques primarily focused on a single modality—either the textual content of the news or its image representation. However, for fake news information circulating on social media platforms today, a single modality presents limited information content. Although the field of fake news detection has made some progress in multimodal fusion, it mainly studies image-text features from a global semantic perspective, neglecting inconsistencies between local semantics. This results in models failing to effectively reduce the differences between images and text. Furthermore, most studies directly represent image information, ignoring details such as texture and edges, as well as textual information, affecting semantic alignment between images and text. Therefore, a multimodal fake news detection method based on unsupervised clustering and frequency domain information is needed. Summary of the Invention

[0004] This application provides a multimodal fake news detection method based on unsupervised clustering and frequency domain information to address three main problems existing in current multimodal fake news detection methods: 1. Insufficient modality fusion and weak interactivity due to semantic feature differences between different modalities in multimodal fake news detection; 2. Directly representing image information while ignoring the texture, edge and other details and text information contained in the image, resulting in insufficient utilization of image information; 3. The semantic gap between images and text.

[0005] This application provides a multimodal fake news detection method based on unsupervised clustering and frequency domain information, the method comprising:

[0006] Obtain sample data from the fake news dataset and use feature vectors. =[ , ] indicates that, among them matrix T Represents the text feature matrix. matrix V The visual feature matrix is ​​represented; the feature vector is input into the feature extraction layer, and the text feature vector is obtained through a pre-trained model and a self-attention method, respectively. Visual feature vectors and image text feature vectors ;

[0007] Based on the K-means clustering method, the category labels are obtained according to the correlation between global semantics in the text feature vector, resulting in text feature vectors with category labels. ;

[0008] The text feature vectors with category labels are input into a first unsupervised clustering learning network. Contextual and temporal information of the text is acquired through bidirectional gated recurrent units and multi-head attention methods, enhancing key information of the text content. The unsupervised clustering learning network is trained by increasing the distance between different categories and enhancing intra-class aggregation, resulting in enhanced text feature vectors. ;

[0009] The visual feature matrix The input is fed into the frequency domain information fusion module, where frequency domain features of the image information are extracted using two-dimensional wavelet transform to enhance the information utilization rate of image modalities. Then, the visual feature vectors and frequency domain features extracted by the pre-trained model are input into the attention fusion module to achieve interaction between spatial and frequency domain features, resulting in joint spatial-frequency domain features rich in semantic relevance. ;

[0010] Based on the visual feature matrix Clustering is performed using the K-means clustering method to obtain image feature vectors with category labels. ;

[0011] The image feature vector with category label Joint features of spatial and frequency domains The input is fed into a second unsupervised clustering learning network, which uses bidirectional gated recurrent units and multi-head attention to obtain the temporal dependencies and contextual relationships of the images. Local contextual features of the news images are learned using triplet learning to obtain the image feature vector. ;

[0012] The image text feature vector The input is fed into the first unsupervised clustering learning network to obtain information-enhanced image and text features. ,Will , and The input is fed into the first attention fusion module, to... The intermediate bridge is fused with both textual and visual features to obtain text-image text fusion features. Visual-image-text fusion features ,Will and The input is fed into the second attention fusion module for cross-modal fusion to obtain multimodal fusion features. ;

[0013] Based on the multimodal fusion features A fully connected layer is used to integrate intermodal interaction features and intramodal features. The integrated feature vector is then input into the Softmax function for multimodal fusion feature classification of fake news.

[0014] Furthermore, the feature vector is input into the feature extraction layer, and text feature vectors are obtained through a pre-trained model and a self-attention method, respectively. Visual feature vectors and image text feature vectors ,include:

[0015] Text feature extraction is performed using a BERT pre-trained model. The BERT pre-trained model performs word segmentation in response to the input text feature vector. After obtaining the segmented sequence, the text feature vector is obtained by querying the segmentation number and its position in the embedding matrix. ;

[0016] Visual feature vectors in images are learned using a VGG-19 pre-trained model. ;

[0017] For each image-text data point, a BLIP-2 pre-trained model is used to extract features from the generated text. For text embedded in the image, RapidOCR is used for feature extraction. The generated text and the embedded text are concatenated and input into a self-attention layer to obtain the image-text feature vector. .

[0018] Furthermore, based on the K-means clustering method, the category labels are obtained according to the correlation between global semantics in the text feature vector, resulting in text feature vectors with category labels. ,include:

[0019] Extract the global semantic features of the text feature vector, represented as follows:

[0020] ;

[0021] in Represents global semantic features. This indicates the number of text samples in the dataset. , and These represent the global semantic features of each paragraph in the text;

[0022] Using the K-means algorithm to Clustering is performed to obtain a set of labels. ;in, k i For the added category label number, k This is the maximum number of labels set for unsupervised clustering.

[0023] Furthermore, the text feature vectors with category labels are input into a first unsupervised clustering learning network. Contextual temporal information of the text and enhancement of key text content information are obtained through bidirectional gated recurrent units and multi-head attention methods. The unsupervised clustering learning network is trained by increasing the distance between different categories and enhancing intra-class aggregation, resulting in enhanced text feature vectors. ,include:

[0024] The text feature vector with category labels The input is fed into a BiGRU network to obtain the bidirectional hidden state of each text feature segmentation, and a multi-head attention mechanism is introduced to dynamically focus on key contextual features, resulting in an enhanced text feature vector. The process of the BiGRU network and multi-head attention processing is as follows:

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] in Local semantic features H T-cls The bidirectional temporal coding vector enhanced by the BiGRU network, where M is the number of attention heads. For the dimensions of text features, , , They are respectively , , The learnable parameter matrix, , , These are the query vector, key vector, and value vector, respectively. a i Let i be the attention head matrix. softmax For normalization function, Q i Let i be the query projection matrix of the i-th parallel head. T This is a matrix transpose operation. V i M is the projection matrix of the values ​​of the i-th parallel head. a This is the merged vector of multiple attention heads. Concat For join operations, LN For layer normalization operation;

[0031] The sum of the contrast loss and the attention sparsity loss is used as the loss function of the first unsupervised clustering learning network, and the loss function is expressed as:

[0032] ;

[0033] ;

[0034] ;

[0035] in, B This indicates the size of each batch of samples during unsupervised training. Represents the similarity function. Indicates positive samples of the same type. Indicates a negative sample from the outlier category. Indicates the temperature coefficient. Indicates the number of negative samples. Indicates the number of attention heads. Indicates the word segmentation length of the sample. h Indicates the number of attention heads. i Indicates the size of each batch. j Indicates text length. L contrast Latt represents the contrast loss, while Latt represents the attention sparsity loss. L unsup Represents the unsupervised loss function. A h,i,t,j This represents the attention weight matrix, where log represents the logarithm operation with base 10. x i Represents the i-th vector. e Represents the natural constant.

[0036] Furthermore, the visual feature matrix The input is fed into the frequency domain information fusion module, where frequency domain features of the image information are extracted using two-dimensional wavelet transform to enhance the information utilization rate of image modalities. Then, the visual feature vectors and frequency domain features extracted by the pre-trained model are input into the attention fusion module to achieve interaction between spatial and frequency domain features, resulting in joint spatial-frequency domain features rich in semantic relevance. ,include:

[0037] Visual feature matrix By using Four convolutional kernels perform a combined convolution operation with a stride of 2 in both the row and column dimensions, where... For low-pass convolution kernels, The specific forms of three high-pass convolution kernels and four convolution kernels are as follows:

[0038] ;

[0039] Through depthwise convolution with four kernels, the output for each channel is:

[0040] ;

[0041] in, Represented as low-frequency components, they contain overall structural information of the image. Represented as high-frequency components in the horizontal, vertical, and diagonal directions, containing detailed texture information of the image. conv Indicates the convolution operation;

[0042] The VGG-19 network is used to extract visual features from images, and spatial and frequency domain features are fused through local-local and global approaches. In this process, for visual features, local textures, edges, and local structures represented by two convolutional layers in the VGG-19 network are extracted as local visual features. The deep semantics represented by the sub-layers serve as global visual features. For frequency domain features, the local frequency domain features are derived from... Composition, frequency domain global features are composed of In the fusion process, convolutions with kernels of different sizes are performed. The outputs of these convolutions at different scales are then used by the CBAM module to redistribute attention weights across channels and space. These weights are then stitched together along the channels to form a multi-scale joint representation of the image. The channel dimensions are adjusted through unit convolutions, and the multi-scale joint features from global and local branches are stitched together to form a spatial-frequency domain joint feature. .

[0043] Furthermore, the VGG-19 network is used to extract visual features from the image, and spatial and frequency domain features are fused through local-local and global methods. The specific process is as follows:

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] in, Indicates the concatenation of features. This represents the features of Conv3-4 layers in the VGG network. This is represented as a feature of the Conv4-4 layer in the VGG network. Indicates size is N × M convolution kernel, This is represented as a convolution operation. Indicates the CBAM module. These are represented as a multi-scale joint representation of local branches and global branches, respectively. BN Indicates batch normalization, X Indicates input features, f LV X is a horizontal low-frequency characteristic. ~ For the normalized features, f V For frequency domain characteristics, H V These are image features.

[0049] Furthermore, the image feature vector with category labels Joint features of spatial and frequency domains The input is fed into a second unsupervised clustering learning network, which uses bidirectional gated recurrent units and multi-head attention to obtain the temporal dependencies and contextual relationships of the images. Local contextual features of the news images are learned using triplet learning to obtain the image feature vector. ,include:

[0050] The image feature vector with category label Joint features of spatial and frequency domains The input is fed into a second unsupervised clustering learning network for images, which learns a bidirectional GRU network for the images. Through triplet learning, images with similar contextual information are made closer in semantic space, resulting in image feature vectors. .

[0051] Furthermore, a bidirectional GRU network is trained on the images, and triplet learning is used to make images with similar contextual information more similar in semantic space, thus obtaining image feature vectors. The calculation process is expressed as follows:

[0052] ;

[0053] ;

[0054] ;

[0055] in This indicates the currently selected anchor sample. This indicates samples that have the same category label. This represents samples with different category labels. N Indicates the number of samples. L tri Let $\mathbf{t}$ represent the triple loss function. $USCLM$ represents an unsupervised contextual learning network that uses triple loss as its loss function.

[0056] Furthermore, the image text feature vector The input is fed into the first unsupervised clustering learning network to obtain information-enhanced image and text features. ,Will , and The input is fed into the first attention fusion module, to... The intermediate bridge is fused with both textual and visual features to obtain text-image text fusion features. Visual-image-text fusion features ,Will and The input is fed into the second attention fusion module for cross-modal fusion to obtain multimodal fusion features. ,include:

[0057] Image text features Through the attention fusion module and respectively and Attention fusion is performed; among them, during the process and During attention fusion, with As an auxiliary mode , For target mode , Modality and The modal fusion representation is shown below:

[0058] ;

[0059] ;

[0060] in, This indicates the attention-based operator, which enhances the query. = From the target mode Enhanced key = and the enhanced value = From auxiliary mode , , , These are the enhanced query, key, and value weight matrices. Indicates auxiliary mode Softmax represents normalization, T represents matrix transpose, Y represents output features, and d k This represents the Kth attention head;

[0061] Text-image text fusion feature fusion using attention fusion method Visual-image-text fusion features The fusion yields multimodal fusion features. .

[0062] Furthermore, based on the aforementioned multimodal fusion features The process of using fully connected layers to integrate intermodal interaction features and intramodal features, and then inputting the integrated feature vector into the Softmax function for multimodal fusion feature classification of fake news is represented as follows:

[0063] ;

[0064] ;

[0065] in, and These are the initial weights and biases for the ReLU activation function. For the final classification result, F 1 represents the feature vector after processing by the ReLU function.

[0066] The multimodal fake news detection method based on unsupervised clustering and frequency domain information provided in this application has at least the following beneficial effects:

[0067] 1) This application extracts multimodal features of text, visual and image text, and uses image text features as a bridge to achieve cross-modal fusion, which fully integrates information from different modalities, solves the problem of one-sided information in single-modal detection, and greatly improves the comprehensiveness and accuracy of fake news detection.

[0068] 2) This application utilizes an unsupervised clustering learning network combined with bidirectional gated recurrent units and multi-head attention mechanisms to enhance the semantic consistency of text and image text features. By fusing visual spatial and frequency domain features through wavelet transform, it effectively strengthens the expressive power of each modality feature and makes the features more discriminative.

[0069] 3) The synergistic effect of multimodal fusion and feature enhancement enables this application to perform well in the task of classifying fake news, and can more accurately identify fake news. It provides an efficient and reliable technical means for news authenticity detection, and has important practical application value and market prospects. Attached Figure Description

[0070] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0071] Figure 1 A flowchart illustrating a multimodal fake news detection method based on unsupervised clustering and frequency domain information, provided in this application embodiment;

[0072] Figure 2 A schematic diagram of the architecture of an unsupervised clustering learning network (text) provided in an embodiment of this application;

[0073] Figure 3 A schematic diagram of the architecture of the frequency domain information fusion module provided in an embodiment of this application;

[0074] Figure 4 A schematic diagram of triple learning in an unsupervised clustering learning network (image) provided in an embodiment of this application;

[0075] Figure 5 A schematic diagram of the structure of a multimodal fake news detection model based on unsupervised clustering and frequency domain information provided in this application embodiment;

[0076] Figure 6 The figure shows the experimental results of unsupervised clustering quantity analysis provided in the embodiments of this application.

[0077] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0078] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0079] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0080] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0081] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0082] This application provides a multimodal fake news detection method based on unsupervised clustering and frequency domain information. Figure 1 A flowchart illustrating a multimodal fake news detection method based on unsupervised clustering and frequency domain information, provided for embodiments of this application, is shown below. Figure 1 As shown, the multimodal fake news detection method includes the following steps 1 to 8.

[0083] Step 1: Obtain the feature vectors of text, visual, and image-generated text from the dataset samples.

[0084] In step 1, sample data is obtained from a fake news dataset. Each sample data contains two sequence modalities: text (T) and visual (V), and feature vectors are used to analyze the data. =[ , The text and visual feature matrices for each fake news sample were obtained. = , The feature vectors are input into the feature extraction layer, and text, image text, and visual feature sequences are obtained through a pre-trained model and a self-attention method, respectively, and used as text feature vectors. Visual feature vectors and image text feature vectors .

[0085] In some embodiments, step 1 specifically includes using pre-trained models such as BERT, VGG19, and BLIP-2 to obtain single-modal features in the sample data, which is achieved through the following steps 1.1 to 1.3.

[0086] Step 1.1: For text data, use the BERT pre-trained model to extract text features. The BERT pre-trained model will segment the input raw text into words. After obtaining the segmented sequence, the text feature vector can be obtained by querying the position of the segment number in the embedding matrix. ;

[0087] Step 1.2: For the visual features contained in the image data, use the VGG-19 pre-trained model to learn rich visual feature vectors in the image. ;

[0088] Step 1.3: For image text data contained in the image data, the first step is to generate descriptive text for the original content of the image. For each image data, a BLIP-2 pre-trained model is used to extract features of the generated text. Then, RapidOCR is used to extract features of the embedded text in the image. Finally, the generated text and the embedded text are concatenated and input into a self-attention layer to obtain the final image text feature vector. .

[0089] In this embodiment, the dimensions of text, visual, and image text features obtained from the fake news sample data are as follows: =768, =4096, =768.

[0090] Step 2: Obtain the text feature vector from Step 1 and label each text feature with a category label.

[0091] In this embodiment, the text feature vector of each fake news sample is obtained in step 1. The K-means clustering method is used to obtain the category labels based on the correlation between global semantics in the text feature vectors, and finally outputs text feature vectors with category labels. .

[0092] In some embodiments, step 2 specifically includes:

[0093] Step 2.1: Extract the global semantic features of the text features extracted from the BERT pre-trained model, represented as:

[0094] ;

[0095] in This represents the global semantic features that define the text features. This indicates the number of text samples in the dataset.

[0096] Step 2.2: Apply the K-means algorithm to... Clustering is performed to group texts with high global semantic similarity into the same category, resulting in a set of labels. , k i For the added category label number, k This is the maximum number of labels set for unsupervised clustering.

[0097] Step 3: Obtain the text feature vectors with completed category labeling in Step 2 and input them into the unsupervised context network for training.

[0098] In this embodiment, the text feature vectors with category labels obtained in step 2 are input into an unsupervised clustering learning network (text). The network uses bidirectional gated recurrent units and multi-head attention methods to acquire contextual temporal information and enhance key information of the text content. Furthermore, by increasing the distance between different categories and enhancing intra-class aggregation, the unsupervised clustering learning network is trained. Finally, it outputs text feature vectors with high semantic consistency within the same category and enhanced modal information. .

[0099] In some embodiments, Figure 2 This is a schematic diagram of the architecture of an unsupervised clustering learning network (text) provided in an embodiment of this application. Combined with... Figure 2 As shown, step 3 specifically includes:

[0100] Step 3.1: Due to the wide range of content contained in news text data and the differences in local semantics between different categories, in order to enhance the consistency of local semantics within each category, the output data from Step 2 is input into the BiGRU network in this step to obtain the bidirectional hidden state of each text feature segmentation. Finally, a multi-head attention mechanism is introduced to dynamically focus on key contextual features. The input local text features are represented as follows: The text features after multi-head attention learning are represented as follows: The learning and multi-head attention process of the BiGRU network is as follows:

[0101] ;

[0102] ;

[0103] ;

[0104] ;

[0105] ;

[0106] in Local semantic features H T-cls go through The enhanced bidirectional temporal coding vector, where M is the number of attention heads. For the dimensions of text features, , , They are respectively , , The learnable parameter matrix is ​​obtained. The outputs of each attention head are concatenated to obtain the complete output result. To ensure the integrity of the information, residual connections are introduced, and layer normalization (LN) is used to process the concatenated output result and the bidirectional temporal coding vector.

[0107] Step 3.2: To achieve semantic aggregation within the same category, semantic distancing between different categories, and enhancement of key elements within categories. The weights are used as the sum of the contrastive loss and the attention sparsity loss as the loss function of the unsupervised network, as shown below:

[0108] ;

[0109] ;

[0110] ;

[0111] Where B represents the size of each batch of samples during unsupervised training. Represents the similarity function. Indicates positive samples of the same type. Indicates a negative sample from the outlier category. Indicates the temperature coefficient. Indicates the number of negative samples. Indicates the number of attention heads. Indicates the word segmentation length of the sample. This represents the attention weight matrix.

[0112] Step 4: Obtain the visual feature vector from Step 1, extract frequency domain information, and fuse spatial and frequency domain information.

[0113] In this embodiment, the visual feature matrix in step 1 The input is fed into the frequency domain information fusion module, where frequency domain features of the image information are extracted using two-dimensional wavelet transform to enhance the information utilization rate of image modalities. Then, the visual feature vectors and frequency domain features extracted by the pre-trained model are input into the attention fusion module to achieve interaction between spatial and frequency domain features. Finally, the output is a joint spatial-frequency domain feature rich in semantic relevance. .

[0114] In some embodiments, Figure 3 This is a schematic diagram of the architecture of the frequency domain information fusion module provided in the embodiments of this application. Figure 3 In this context, the input image refers to the original visual feature matrix. , f 4096 This represents the features extracted by the VGG-19 network from the input image. Conv 1*1、 Conv 1*3、 Conv 3*1 and Conv 3*3 represents convolution operations of 1*1, 1*3, 3*1, and 3*3 respectively; comba represents feature concatenation operation; the output features are semantically relevant spatial-frequency joint features. , combined Figure 3 As shown, step 4 specifically includes:

[0115] Step 4.1: Convert the original visual feature matrix Perform Haar wavelet decomposition by using ( Four convolutional kernels perform a combined convolution operation with a stride of 2 in both the row and column dimensions. For low-pass convolution kernels, ( This consists of three high-pass convolution kernels. The specific form of each kernel is as follows:

[0116] ;

[0117] Input image V, through depthwise convolution with four convolution kernels, each channel yields the following convolution output:

[0118] ;

[0119] in, Represented as low-frequency components, these mainly contain overall structural information of the image, while It is represented as high-frequency components in the horizontal, vertical, and diagonal directions, and mainly contains the detailed texture information of the image.

[0120] Step 4.2: Representing images solely using frequency domain features or spatial domain features has limitations in terms of information from a single domain. Therefore, the VGG-19 network is used to extract visual features from the image, and spatial and frequency domain features are fused using a local-local and global-global approach. In this process, for visual features: the local texture, edges, and local structure of the image represented by the Conv3-4 and Conv4-4 convolutional layers in the VGG-19 network are extracted as local features. The deep semantics represented by the sub-layer serve as its global features. For frequency domain features: local features are... Composition, global features are composed of The process involves convolution with kernels of different sizes, followed by a redistribution of channel and spatial attention weights using the CBAM module. These weights are then stitched together along the channels to form a multi-scale joint representation of the image. Finally, a 1×1 convolution kernel is used to adjust the channel dimensions. The multi-scale joint features from both global and local branches are then combined to form the final image representation. The specific process is as follows:

[0121] ;

[0122] ;

[0123] ;

[0124] ;

[0125] in, Indicates the concatenation of features. This represents the features of Conv3-4 layers in the VGG network. This is represented as a feature of the Conv4-4 layer in the VGG network. This represents a convolution kernel of size N×M. This is represented as a convolution operation. Indicates the CBAM module. These are represented as a multi-scale joint representation of local branches and global branches, respectively.

[0126] Step 5: Obtain the image feature vector from Step 1 and label each image feature with a category label.

[0127] In this embodiment, the original visual feature matrix from step 1 is extracted. Image features are clustered using the K-means clustering method. The final output is an image feature vector with category labels. .

[0128] In some embodiments, step 5 specifically includes:

[0129] Step 5.1: Extract the image feature matrix after feature extraction from the VGG pre-trained model. The K-means algorithm is used for clustering to divide the data into K classes. The process is as follows: [Resulting label set] .

[0130] Step 6: Obtain the image feature vectors with completed category labeling in Step 5 and input them into the unsupervised context network for training.

[0131] In this embodiment, the image feature vector with category labels in step 5 is... Joint features of spatial and frequency domains The input is fed into an unsupervised clustering learning network (image). Temporal dependencies and contextual relationships of the image are obtained through bidirectional gated recurrent units and multi-head attention methods. Local contextual features of the news image are learned using triplet learning, ultimately yielding the image feature vector. .

[0132] In some embodiments, step 6 specifically includes:

[0133] Step 6.1: Extract image feature vectors for category labels Joint features of spatial and frequency domains The input is fed into an unsupervised clustering learning network for images;

[0134] Step 6.2: First, learn a bidirectional GRU network for the image. Then, use triplet learning to make images with similar contextual information more semantically similar. Finally, obtain the image feature vector. The specific process is as follows:

[0135] ;

[0136] ;

[0137] ;

[0138] in This indicates the currently selected anchor sample. This indicates samples that have the same category label. This represents samples with different category labels. USCLM is an unsupervised contextual learning network that uses triple loss as its loss function.

[0139] In some embodiments, Figure 4 This is a schematic diagram of triple learning in an unsupervised clustering learning network (image) provided in an embodiment of this application. Figure 4 As shown, Figure 4In this diagram, P1, P2, and P3 represent positive samples, while N, N1, N2, and N3 represent negative samples. A represents the anchor sample. After obtaining the feature matrix of the image with classification labels, the coordinates of the cluster center for each category are first calculated. Then, the sample closest to the center coordinates is selected as the anchor sample for that category. In triplet learning, samples belonging to the same category as the anchor sample are positive samples, while those belonging to different categories are negative samples. For example, if the anchor sample's label is 10, then samples with labels other than 10 are negative samples. Finally, learning is achieved by aggregating similar samples and excluding negative samples.

[0140] Step 7: Input the image and text from Step 1 into the unsupervised context network in Step 3, and perform feature fusion with the text and image respectively using the image and text as auxiliary tools to finally obtain the multimodal feature vector.

[0141] In this embodiment, the image text feature vector generated in step 1 is obtained. Then repeat step 3 to obtain the information-enhanced image-text features. Then enhance the modal semantic consistency , and modal characterization enhancement The input is fed into the attention fusion module, and then... To obtain text-image text fusion features with small semantic differences, the intermediate bridge is fused with both textual and visual features. Visual-image-text fusion features Finally , The input is fed into the attention fusion module for cross-modal fusion to obtain multimodal fusion features. .

[0142] In some embodiments, step 7 specifically includes:

[0143] Step 7.1: Extract the image text from Step 1 Then, the unsupervised context network trained in step 3 is used for intramodal information augmentation, and the augmented results are then used to perform intramodal information augmentation. The attention fusion module enhances text consistency. Multi-scale joint representation of images Perform attention fusion. and Taking attention fusion as an example, As an auxiliary mode , For target mode , Modality and The modal fusion representation is shown below:

[0144] ;

[0145] ;

[0146] in, This represents an attention-based operator, enhancing the query. = From auxiliary mode key = Sum of values = From modality , , , These are the corresponding weight matrices and source modes. Using low-order signals for target modes The feature information is reconstructed, and key / value interactions are used to capture important correlation information across modalities. Conversely, then splice them together. , The final weighted cross-modal interaction features of image, text, and visual features are obtained. .

[0147] Step 7.2: Implement text-image text fusion features using the same attention fusion method. Visual-image-text fusion features The fusion of these features ultimately yields multimodal fusion characteristics. .

[0148] Step 8: Process the multimodal feature vectors obtained in Step 7 using the softmax function to achieve the task of identifying fake news.

[0149] In this embodiment, the obtained multimodal fusion features Fully connected layers are used to integrate intermodal interaction features and intramodal features. Finally, the integrated feature vector is input into the Softmax function for multimodal fusion feature classification of fake news.

[0150] In some embodiments, step 8 specifically includes: obtaining multimodal fusion features Fully connected layers are used to integrate intermodal interaction features and intramodal features. Finally, the integrated feature vector is input into the Softmax function for multimodal fusion feature classification of fake news. The calculation process is as follows:

[0151] ;

[0152] ;

[0153] in, and These are the initial weights and biases for the ReLU activation function. This is the final classification result.

[0154] Figure 5 This is a schematic diagram illustrating the structure of a multimodal fake news detection model based on unsupervised clustering and frequency domain information, provided in an embodiment of this application. This multimodal fake news detection model based on unsupervised clustering and frequency domain information can be used to implement the method provided in the above embodiment. The multimodal fake news detection model based on unsupervised clustering and frequency domain information includes:

[0155] (1) Input layer: Extract primary modal features. For text data in text information, the 768-dimensional output of the BERT pre-trained model is used as text features. For image data in image information, the 4096-dimensional output of the VGG-19 pre-trained model is used as visual features. For image text features in image information, the BLIP-2 and RapidOCR pre-trained models are used to extract the descriptive text and embedded text features of the image, respectively.

[0156] (2) Unsupervised context training network (USCLM): It uses bidirectional gated recurrent network and multi-head attention mechanism to enhance the semantic consistency of uniform class data within the modality and assign high weights to key information in the sample data.

[0157] (3) Frequency domain fusion module: Haar wavelet transform is used to capture the frequency domain information contained in the image, and the feature data of the image spatial domain and frequency domain are fused in a global-global and local-local manner to enhance the representation ability of image information.

[0158] (4) Multimodal fusion layer: The enhanced text features, image features and image-text features are fused by cross attention to obtain cross-modal interaction features.

[0159] (5) Output layer: The cross-modal interaction features are sent to Softmax after passing through a fully connected layer for the final fake news classification task.

[0160] The following embodiment will further illustrate the feasibility and advancement of the method proposed in this application by combining comparative experiments, model ablation experiments, and unsupervised clustering quantitative analysis.

[0161] This embodiment compares the SCFI-MFD model used to implement the method proposed in this application with other existing models on the same dataset. The results are shown in Table 1.

[0162] Table 1 Comparative Experimental Results

[0163]

[0164] The relevant abbreviations in Table 1 are explained as follows:

[0165] Acc: Accuracy, which measures the proportion of samples correctly predicted by the model out of the total number of samples;

[0166] Precision: This measures how many samples the model predicts to be positive are actually positive.

[0167] Recall: Recall rate measures how many of all true positive samples are correctly predicted as positive by the model;

[0168] F1: The F1 score is the harmonic mean of precision and recall, which takes into account both precision and recall.

[0169] BERT: A text preprocessing model.

[0170] VGG19: An image preprocessing model.

[0171] Att-RNN: A model that uses a Long Short-Term Memory (LSTM) network to model news text, extracts image features using the VGG model, and fuses textual and visual features through an attention mechanism.

[0172] SpotFake: A comparative model that uses the BERT and VGG models to extract unimodal features from text and images and concatenates them as multimodal fusion features for news.

[0173] EANN: A model that uses a text convolutional neural network (TextCNN) and a VGG model to extract textual and visual features, concatenates the two types of features, and inputs them into a fully connected layer to classify news as true or false.

[0174] CARMN: A model that integrates textual and visual features through cross-attention residuals and multi-channel convolutional neural networks to detect fake news, while simultaneously extracting feature representations of both original and fused information to reduce the impact of noise.

[0175] MVAE: This model uses LSTM and VGG models to extract textual and visual features from news articles. The features are concatenated and encoded into multimodal features to reconstruct the input features and classify the news as true or false.

[0176] CAFE: A model that adaptively aggregates cross-modal correlations by combining the strength of ambiguity between modes.

[0177] MCAN: A model that extracts spatial and frequency domain features from an image and fuses them with text features using multiple cross-modal co-attention modules.

[0178] Weibo: Chinese dataset.

[0179] Twitter: English dataset.

[0180] As shown in Table 1, the experimental comparison demonstrates that the proposed SCFI-MFD model outperforms other baseline models in most metrics across both datasets. On the two different datasets, among multimodal methods, att-RNN, MAVE, and EANN, using simple concatenation, failed to uncover deep semantic relationships and exhibited low cross-modal interactivity, resulting in their detection performance ranking at the bottom among various multimodal methods. CAFÉ's detection performance is insufficient because its cross-modal alignment approach only captures semantic consistency from a holistic perspective, neglecting fine-grained semantic matching. CARMN and MCAN show significant differences of over 9% across the two datasets, likely due to the shorter text in the Twitter dataset, which hinders effective information alignment and complementary learning with images. In terms of accuracy, the proposed method outperforms all baseline models on the Weibo and Twitter datasets, achieving 92.2% and 84.9% respectively, exceeding the best baseline results by 2.3% and 4%, confirming that the proposed model effectively improves the performance of fake news detection.

[0181] This embodiment also included a model ablation experiment, and the results are shown in Table 2.

[0182] Table 2 Comparison Results of Model Ablation Experiments

[0183]

[0184] It should be noted that in Table 2:

[0185] SCFI-MFD(-U): Based on the SCFI-MFD model, the unsupervised clustering learning network is removed, and no text semantic information is processed; the data is directly fused.

[0186] SCFI-MFD(-F): Based on the SCFI-MFD model, the Haar wavelet frequency domain transform of image information is removed, and only the visual information extracted by VGG19 is fused with the text information.

[0187] SCFI-MFD (-OCR): Based on the SCFI-MFD model, OCR text generation is removed, and image OCR text enhancement is not used to enhance the semantic information of the image.

[0188] SCFI-MFD(-A): Based on the SCFI-MFD model, the attention fusion module is removed, and only a fully connected layer is used for feature fusion of image and text.

[0189] SCFI-MFD(-VT): Based on the SCFI-MFD model, it removes image description text generation in addition to OCR text generation, and only uses the attention fusion module to fuse the features of the image and text.

[0190] As shown in Table 2, experimental results demonstrate that the complete model outperforms all variant models on both the Weibo and Twitter datasets. Removing any crucial component of the model negatively impacts its performance to varying degrees, highlighting the importance of each component to the overall effectiveness of the model. Based on the degree of accuracy degradation, removing frequency domain features of image modalities has the most significant impact on model performance, indicating that utilizing fine-grained information contained in the frequency domain of images helps improve the accuracy of fake news detection. Ablation experiments after removing the unsupervised clustering learning network reveal that enhancing the semantic consistency between local semantic features of news text and the context helps the model uncover deeper semantic connections within the text and improves the model's interpretability. It is noteworthy that when using image-text as auxiliary information for cross-modal fusion, initially removing the OCR text data contained in the image disrupts the semantic integrity of the news image, weakening the model's performance. When the image description text features are removed, and cross-modal fusion relies solely on image and text features, the model's performance declines again. This indicates that introducing image OCR text and image description text as auxiliary information not only enhances the semantic expression of the image but also helps improve the model's ability to identify fake news. In conclusion, ablation experiments demonstrate that each component in SCFI-MFD makes a positive contribution to the model's multimodal fake news detection task.

[0191] This embodiment also conducted an unsupervised clustering quantity analysis. In the unsupervised clustering learning network, context division was performed using semantic clustering. To explore the impact of different cluster sizes on model performance, this embodiment set different cluster sizes for quantitative analysis, using accuracy (ACC) as the evaluation metric. The experimental results are as follows: Figure 6 As shown, both too small and too large a number of clusters negatively impact the performance of the fake news detection model. The model performs best when the number of clusters is 10.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A multimodal fake news detection method based on unsupervised clustering and frequency domain information, characterized in that, The method includes: Obtain sample data from the fake news dataset and use feature vectors. =[ , ] indicates that, among them matrix T Represents the text feature matrix. matrix V The visual feature matrix is ​​represented; the feature vector is input into the feature extraction layer, and the text feature vector is obtained through a pre-trained model and a self-attention method, respectively. Visual feature vectors Image and text feature vectors ; Based on the K-means clustering method, the category labels are obtained according to the correlation between global semantics in the text feature vector, resulting in text feature vectors with category labels. ; The text feature vectors with category labels are input into a first unsupervised clustering learning network. Contextual and temporal information of the text is acquired through bidirectional gated recurrent units and multi-head attention methods, enhancing key information of the text content. The unsupervised clustering learning network is trained by increasing the distance between different categories and enhancing intra-class aggregation, resulting in enhanced text feature vectors. ; The visual feature matrix The input is fed into the frequency domain information fusion module, where frequency domain features of the image information are extracted using two-dimensional wavelet transform to enhance the information utilization rate of image modalities. Then, the visual feature vectors and frequency domain features extracted by the pre-trained model are input into the attention fusion module to achieve interaction between spatial and frequency domain features, resulting in joint spatial-frequency domain features rich in semantic relevance. ; Based on the visual feature matrix Clustering is performed using the K-means clustering method to obtain image feature vectors with category labels. ; The image feature vector with category label Joint features of spatial and frequency domains The input is fed into a second unsupervised clustering learning network, which uses bidirectional gated recurrent units and multi-head attention to obtain the temporal dependencies and contextual relationships of the images. Local contextual features of the news images are learned using triplet learning to obtain the image feature vector. ; The image text feature vector The input is fed into the first unsupervised clustering learning network to obtain information-enhanced image and text features. ,Will , and The input is fed into the first attention fusion module, to... The intermediate bridge is fused with both textual and visual features to obtain text-image text fusion features. Visual-image-text fusion features ,Will and The input is fed into the second attention fusion module for cross-modal fusion to obtain multimodal fusion features. ; Based on the multimodal fusion features A fully connected layer is used to integrate intermodal interaction features and intramodal features. The integrated feature vector is then input into the Softmax function for multimodal fusion feature classification of fake news.

2. The multimodal fake news detection method based on unsupervised clustering and frequency domain information according to claim 1, characterized in that, The feature vector is input into the feature extraction layer, and text feature vectors are obtained through a pre-trained model and a self-attention method, respectively. Visual feature vectors Image and text feature vectors ,include: Text feature extraction is performed using a BERT pre-trained model. The BERT pre-trained model performs word segmentation in response to the input text feature vector. After obtaining the segmented sequence, the text feature vector is obtained by querying the segmentation number and its position in the embedding matrix. ; Visual feature vectors in images are learned using a VGG-19 pre-trained model. ; For each image-text data point, a BLIP-2 pre-trained model is used to extract features from the generated text. For text embedded in the image, RapidOCR is used for feature extraction. The generated text and the embedded text are concatenated and input into a self-attention layer to obtain the image-text feature vector. .

3. The multimodal fake news detection method based on unsupervised clustering and frequency domain information according to claim 1, characterized in that, Based on the K-means clustering method, the category labels are obtained according to the correlation between global semantics in the text feature vector, resulting in text feature vectors with category labels. ,include: Extract the global semantic features of the text feature vector, represented as follows: ； in Represents global semantic features. This indicates the number of text samples in the dataset. , and These represent text features that embody the global semantic information of a text segment; Using the K-means algorithm to Clustering is performed to obtain a set of labels. ;in, k i For the added category label number, k This is the maximum number of labels set for unsupervised clustering.

4. The multimodal fake news detection method based on unsupervised clustering and frequency domain information according to claim 1, characterized in that, The text feature vectors with category labels are input into a first unsupervised clustering learning network. Contextual and temporal information of the text is acquired through bidirectional gated recurrent units and multi-head attention methods, enhancing key information of the text content. The unsupervised clustering learning network is trained by increasing the distance between different categories and enhancing intra-class aggregation, resulting in enhanced text feature vectors. ,include: The text feature vector with category labels The input is fed into a BiGRU network to obtain the bidirectional hidden state of each text feature segmentation, and a multi-head attention mechanism is introduced to dynamically focus on key contextual features, resulting in an enhanced text feature vector. The process of the BiGRU network and multi-head attention processing is as follows: ； ； ； ； ； in Local semantic features H T-cls The bidirectional temporal coding vector enhanced by the BiGRU network, where M is the number of attention heads. For the dimensions of text features, , , They are respectively , , The learnable parameter matrix, , , These are the query vector, key vector, and value vector, respectively. a i Let i be the attention head matrix. softmax For normalization function, Q i Let i be the query projection matrix of the i-th parallel head. T This is a matrix transpose operation. V i M is the projection matrix of the values ​​of the i-th parallel head. a This is the merged vector of multiple attention heads. Concat For join operations, LN For layer normalization operation; The sum of the contrast loss and the attention sparsity loss is used as the loss function of the first unsupervised clustering learning network, and the loss function is expressed as: ； ； ； in, B This indicates the size of each batch of samples during unsupervised training. Represents the similarity function. Indicates positive samples of the same type. Indicates a negative sample from the outlier category. Indicates the temperature coefficient. Indicates the number of negative samples. Indicates the number of attention heads. Indicates the word segmentation length of the sample. h Indicates the number of attention heads. i Indicates the size of each batch. j Indicates text length. L contrast Latt represents the contrast loss, while Latt represents the attention sparsity loss. L unsup Represents the unsupervised loss function. A h,i,t,j This represents the attention weight matrix, where log represents the logarithm operation with base 10. x i Represents the i-th vector. e Represents the natural constant.

5. The multimodal fake news detection method based on unsupervised clustering and frequency domain information according to claim 1, characterized in that, The visual feature matrix The input is fed into the frequency domain information fusion module, where frequency domain features of the image information are extracted using two-dimensional wavelet transform to enhance the information utilization rate of image modalities. Then, the visual feature vectors and frequency domain features extracted by the pre-trained model are input into the attention fusion module to achieve interaction between spatial and frequency domain features, resulting in joint spatial-frequency domain features rich in semantic relevance. ,include: Visual feature matrix By using Four convolutional kernels perform a combined convolution operation with a stride of 2 in both the row and column dimensions, where... For low-pass convolution kernels, The specific forms of three high-pass convolution kernels and four convolution kernels are as follows: ； Through depthwise convolution with four kernels, the output for each channel is: ； in, Represented as low-frequency components, they contain overall structural information of the image. Represented as high-frequency components in the horizontal, vertical, and diagonal directions, containing detailed texture information of the image. conv Indicates the convolution operation; The VGG-19 network is used to extract visual features from images, and spatial and frequency domain features are fused through local-local and global approaches. In this process, for visual features, local textures, edges, and local structures represented by two convolutional layers in the VGG-19 network are extracted as local visual features. The deep semantics represented by the sub-layers serve as global visual features. For frequency domain features, the local frequency domain features are derived from... Composition, frequency domain global features are composed of In the fusion process, convolutions with kernels of different sizes are performed. The outputs of these convolutions at different scales are then used by the CBAM module to redistribute attention weights across channels and space. These weights are then stitched together along the channels to form a multi-scale joint representation of the image. The channel dimensions are adjusted through unit convolutions, and the multi-scale joint features from global and local branches are stitched together to form a spatial-frequency domain joint feature. .

6. The multimodal fake news detection method based on unsupervised clustering and frequency domain information according to claim 5, characterized in that, The VGG-19 network is used to extract visual features from images, and spatial and frequency domain features are fused through local-local and global methods, as shown in the following process: ； ； ； ； in, Indicates the concatenation of features. This represents the features of Conv3-4 layers in the VGG network. This is represented as a feature of the Conv4-4 layer in the VGG network. Indicates size is N × M convolution kernel, This is represented as a convolution operation. Indicates the CBAM module. These are represented as a multi-scale joint representation of local branches and global branches, respectively. BN Indicates batch normalization, X Indicates input features, f LV X is a horizontal low-frequency characteristic. ~ For the normalized features, f V For frequency domain characteristics, H V These are image features.

7. The multimodal fake news detection method based on unsupervised clustering and frequency domain information according to claim 1, characterized in that, The image feature vector with category label Joint features of spatial and frequency domains The input is fed into a second unsupervised clustering learning network, which uses bidirectional gated recurrent units and multi-head attention to obtain the temporal dependencies and contextual relationships of the images. Local contextual features of the news images are learned using triplet learning to obtain the image feature vector. ,include: The image feature vector with category label Joint features of spatial and frequency domains The input is fed into a second unsupervised clustering learning network for images, which learns a bidirectional GRU network for the images. Through triplet learning, images with similar contextual information are made closer in semantic space, resulting in image feature vectors. .

8. The multimodal fake news detection method based on unsupervised clustering and frequency domain information according to claim 7, characterized in that, The image is trained using a bidirectional GRU network, and triplet learning is used to make images with similar contextual information more semantically closer, thus obtaining the image feature vector. The process is represented as: ； ； ； in This indicates the currently selected anchor sample. This indicates samples that have the same category label. This represents samples with different category labels. N This represents the total number of clusters. L tri Let $\int \ ...

9. The multimodal fake news detection method based on unsupervised clustering and frequency domain information according to claim 1, characterized in that, The image text feature vector The input is fed into the first unsupervised clustering learning network to obtain information-enhanced image and text features. ,Will , and The input is fed into the first attention fusion module, to... The intermediate bridge is fused with both textual and visual features to obtain text-image text fusion features. Visual-image-text fusion features ,Will and The input is fed into the second attention fusion module for cross-modal fusion to obtain multimodal fusion features. ,include: Image text features Through the attention fusion module and respectively and Perform attention fusion; among which, in performing and During attention fusion, with As an auxiliary mode , For target mode , Modality and The modal fusion representation is shown below: ； ； in, This indicates the attention-based operator, enhancing the query. = From the target mode Enhanced key = and the enhanced value = From auxiliary mode , , , These are the enhanced query, key, and value weight matrices. Indicates auxiliary mode Softmax represents normalization, T represents matrix transpose, Y represents output features, and d k This represents the Kth attention head; Text-image text fusion feature fusion using attention fusion method Visual-image-text fusion features The fusion yields multimodal fusion features. .

10. The multimodal fake news detection method based on unsupervised clustering and frequency domain information according to claim 1, characterized in that, Based on the multimodal fusion features The process of using fully connected layers to integrate intermodal interaction features and intramodal features, and then inputting the integrated feature vector into the Softmax function for multimodal fusion feature classification of fake news is represented as follows: ； ； in, and These are the initial weights and biases for the ReLU activation function. For the final classification result, F 1 represents the feature vector after processing by the ReLU function.

Images