False news detection method based on multi-modal feature enhancement and semantic alignment

Abstract

The invention provides a false news detection method based on multi-modal feature enhancement and semantic alignment. The method comprises the following steps: preprocessing acquired text data and image data; extracting multi-granularity text features, and extracting multi-granularity image features; outputting the enhanced text features, and outputting the enhanced image features; the enhanced text features and the enhanced image features are aligned through a multi-head self-attention mechanism in a shared Transform model, and the aligned text features and the image features are spliced and fused to obtain fused multi-modal features; inputting the fused multi-modal features into a classifier to obtain a detection result; training a false news detection model based on multi-modal feature enhancement and semantic alignment to obtain an optimized model; performing real-time false news detection; according to the method, the accuracy and the reliability of false news detection can be remarkably improved, and a new thought and a new solution are provided for research and application in related fields.

Description

Technical Field

[0001] This invention relates to a method for detecting fake news based on multimodal feature enhancement and semantic alignment, belonging to the field of news detection technology. Background Technology

[0002] News on social media and other platforms typically comprises multiple information formats, including news content, social context information, and external information. News content usually refers to multimodal presentations such as articles, images, or videos, while social context and external information involve the dissemination structure of the news and related objective facts. Because news content plays a central role in information dissemination, researchers have increasingly focused on the characteristics and structure of misinformation. Therefore, multimodal supervised detection methods based on news content have become a major research direction.

[0003] The key to multimodal fake news detection lies in adjusting the joint representation of textual and visual information. News in existing social networks typically contains both textual and image information, and combining this information helps improve detection results. Currently, there are three main directions surrounding multimodal fake news detection.

[0004] First, there's multimodal representation learning. Some studies treat visual information as a complement to fake news text. These studies use text encoders to extract textual features, visual encoders to extract visual features, and then simply concatenate the two as the news features. A commonly used multimodal complementarity framework in fake news detection first obtains image features from a pre-trained VGG19 model, then simply concatenates these visual features with textual features. Within this framework, Wang et al. improved the model's generalization ability by introducing event classification as an additional task to guide the learning of event-invariant multimodal features. Dhruv and Khattar et al. modified this general complementarity framework into a multimodal variational autoencoder to obtain multimodal representations for fake news detection. While these studies have achieved some success, they all fail to consider the complex cross-modal relationships in fake news, limiting the effectiveness of multimodal content detection.

[0005] Secondly, there is multimodal consistency learning. Irrelevant images are one of the features of multimodal fake news; therefore, some studies have begun to focus on multimodal consistency in fake news detection. Zhou et al. used an image captioning model to convert images into sentences and then evaluated the sentence similarity between the original news text and the generated image captions to compute multimodal inconsistency. However, the performance of the model is limited due to the difference between the training dataset of the image captioning model and the actual news corpus. Xue et al. used a weight-shared encoder to project visual features and text features into a common feature space and then computed the similarity of the transformed multimodal features; however, capturing multimodal inconsistency remains challenging due to the semantic gap between visual features and text features.

[0006] Finally, there is multimodal enhancement fusion. Directly and coarsely modeling images is not the optimal choice when fusing information. News text and images are semantically related at a high level, and their alignment often indicates important features of the news. Therefore, some studies focus on extracting features from images and text and enhancing them together to better detect fake news. Zhang et al. used a multi-channel CNN with an attention mechanism to fuse multimodal information, emphasizing important image regions guided by text. MCAN also developed a novel fusion method that employs multiple co-attention layers to learn cross-modal relationships, first fusing visual features, then textual features. The fused representation obtained from the last co-attention layer is used for fake news detection. Although existing multimodal fake news detection methods have made some progress in handling the joint representation of text and image information, these methods still have some shortcomings, especially in the context of complex and diverse news content on social networks. Most current research focuses on single multimodal complementarity or consistency measures, failing to fully utilize the complex cross-modal relationships between text and images, resulting in insufficient generalization ability of the models in real-world scenarios. Furthermore, existing methods often rely on simple feature concatenation or consistency evaluation, failing to delve into the deep semantic associations of multimodal information. Summary of the Invention

[0007] The purpose of this invention is to provide a fake news detection method based on multimodal feature enhancement and semantic alignment to address the problem that existing technologies fail to fully utilize the complex cross-modal relationships between text and images, and the effectiveness of multimodal content detection needs to be improved.

[0008] The technical solution of this invention is:

[0009] A method for detecting fake news based on multimodal feature enhancement and semantic alignment includes the following steps:

[0010] S1. The data preprocessing module is used to preprocess the acquired text data and image data to obtain preprocessed text data and preprocessed image data.

[0011] S2. Apply Bert to the preprocessed text data. base The model extracts multi-granularity text features, including coarse-grained text features, medium-grained text features, and fine-grained text features. The VGG19 model is applied to the preprocessed image data to extract multi-granularity image features, including coarse-grained image features, medium-grained image features, and fine-grained image features.

[0012] S3. The text feature enhancement module performs feature aggregation and feature selection on multi-granularity text features and outputs the enhanced text features; the image feature enhancement module performs feature aggregation and feature selection on multi-granularity image features and outputs the enhanced image features.

[0013] S4. Align the enhanced text features and enhanced image features by sharing the multi-head self-attention mechanism in the Transformer model, and then concatenate and fuse the aligned text features and image features to obtain fused multimodal features.

[0014] S5. Input the fused multimodal features obtained in step S4 into the classifier to obtain the detection results;

[0015] S6, by the data preprocessing module, Bert base The model, VGG19 model, text feature enhancement module, image feature enhancement module, shared Transformer model and classifier constitute a fake news detection model based on multimodal feature enhancement and semantic alignment. After training the fake news detection model based on multimodal feature enhancement and semantic alignment with a composite loss consisting of classification loss and contrast loss, the optimized model is obtained.

[0016] S7. After deploying the optimized model, perform real-time fake news detection.

[0017] Furthermore, in step S1, the acquired text data and image data are preprocessed, specifically by segmenting the text data, removing stop words, and standardizing it; performing geometric transformation on the image data; and cleaning the text data and image data to remove data with missing modalities, i.e., data samples without corresponding images or with empty text content.

[0018] Further, step S2 specifically involves,

[0019] S21. For the preprocessed text data, a pre-trained BERT dataset is used. baseThe model extracts text features by obtaining the embedding representation of each token output from the top-level Transformer block as fine-grained text features, the mean-pooled representation output from the intermediate Transformer block as medium-grained text features, and the global semantic vector output from the bottom layer. As a coarse-grained text feature;

[0020] S22. For the preprocessed image data, the pre-trained VGG19 model is used to extract image features. The high-resolution local feature map output by the shallow convolutional network at the front end is obtained as fine-grained image features, the feature map containing texture and shape information output by the convolutional network at the middle end is obtained as medium-grained image features, and the feature vector of the deep fully connected layer at the back end is obtained as coarse-grained image features.

[0021] Furthermore, in step S3, a text feature enhancement module is used to perform feature aggregation and feature selection on multi-granularity text features, and outputs the enhanced text features, specifically,

[0022] S31. Randomly shuffle the multi-granularity text features and evenly divide the shuffled features into two subsets, called the source subset src and the target subset dst, respectively.

[0023] S32. For each feature in the source subset src, calculate its cosine similarity with the features in the target subset dst, and select the feature with the most similarity. Sort all matching pairs in descending order of similarity, and retain the top k pairs with the highest similarity to ensure the relevance of the selected features. Generate a new feature representation as an aggregate feature by averaging the matching pairs in each dimension.

[0024] S33. After obtaining the aggregated features of each modality, the maximum value in each feature dimension is selected as the discrimination parameter to select the most discriminative feature in each dimension. These most discriminative features are then reorganized in the original feature dimension order into a feature vector that has the highest activation response in each semantic dimension, thus obtaining the enhanced text features.

[0025] Furthermore, in step S3, the image feature enhancement module and the text feature enhancement module use the same structure and processing method to obtain the enhanced image features.

[0026] Furthermore, in step S6, the composite loss L:

[0027]

[0028] Among them, L c To compare the losses, L mixup For comparison of losses, θ represents model parameters. This is the partial derivative of the loss function with respect to the model parameters.

[0029] Furthermore, comparing the loss L mixup :

[0030] ,

[0031] ,

[0032] ,

[0033] in, This indicates that the summation operation is performed on all samples in the current training batch; β1 and β2 are two hyperparameters that represent the distance interval between positive and negative sample pairs to ensure that positive and negative samples can be effectively distinguished during training. To calculate cosine similarity; and These represent the visual feature vector and text feature vector of the current input news sample after enhancement processing, respectively, i.e., the positive sample pair; and These are the hard negative visual samples and text samples in each mini-batch; negative samples generated from visual features. Negative samples generated from text features ; Set α1 and α2 to be sampled from the Beta distribution to enhance the diversity of negative samples; and These represent the generated hard negative visual samples and hard negative text samples, respectively. The hinge loss operator is defined as follows: ; This represents the function for selecting the maximum value; This represents the index optimization function, used to search and lock in values ​​that maximize cosine similarity. The index of the negative sample that reaches the maximum value; and These represent the visual feature candidate variables and text feature candidate variables used for searching hard negative samples in the mini-batch, respectively. This indicates exclusion constraints.

[0034] Further, in step S7, the optimized model is deployed. Specifically, after building the news detection system using the lightweight Flask framework, the optimized model is deployed on the news detection system.

[0035] The beneficial effects of this invention are:

[0036] First, this fake news detection method based on multimodal feature enhancement and semantic alignment can significantly improve the accuracy and reliability of fake news detection. Compared with existing methods, the proposed model has significantly improved the performance indicators in fake news detection tasks, demonstrating strong practicality and effectiveness, and providing new ideas and solutions for research and application in related fields.

[0037] II. This fake news detection method based on multimodal feature enhancement and semantic alignment first performs multimodal preprocessing on the input data to improve data quality and ensure the consistency of the input data; then it utilizes... The model, along with the VGG19 model, extracts multimodal features at three granular levels: coarse, medium, and fine. This enables a more comprehensive capture of the semantic and visual representations of fake news at different levels, enriching feature expression. Building upon this, text and image feature enhancement modules are introduced. Through similarity score aggregation and dimensional feature selection, redundant features are automatically eliminated, and key discriminative information is strengthened, thereby improving the discriminative power of fake news features. Subsequently, a shared Transformer and a hard negative sample contrast learning strategy based on Mixup are used to achieve accurate alignment of text and images in the shared space, effectively narrowing the cross-modal semantic gap. Furthermore, a joint gradient optimization mechanism is employed, enabling the model to improve classification accuracy while possessing stronger cross-modal generalization ability and robustness. Finally, the trained model is deployed on a lightweight framework, achieving real-time fake news detection and enhancing the model's practical application value. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the fake news detection method based on multimodal feature enhancement and semantic alignment according to an embodiment of the present invention.

[0039] Figure 2 This is an illustrative diagram illustrating the fake news detection model based on multimodal feature enhancement and semantic alignment in the embodiment.

[0040] Figure 3 This is an illustrative diagram illustrating feature aggregation and feature selection for multi-granularity text features or multi-granularity image features in the embodiments;

[0041] Figure 4 The implementation introduces a shared Transformer and a contrastive loss function. A schematic diagram comparing the sample distribution before and after, where (a) shows the distribution with the shared Transformer and contrastive loss function. The previous sample distribution comparison diagram (b) shows the introduction of a shared Transformer and a contrastive loss function. A diagram showing the comparison of the sample distribution after the comparison. Detailed Implementation

[0042] To enhance understanding of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. These embodiments are only used to explain the invention and do not limit the scope of protection of the invention.

[0043] The embodiment provides a fake news detection method based on multimodal feature enhancement and semantic alignment, such as Figure 1 and Figure 2 This includes the following steps:

[0044] S1. The data preprocessing module is used to preprocess the acquired text data and image data to obtain preprocessed text data and preprocessed image data.

[0045] In step S1, the acquired text and image data are preprocessed. Specifically, the text data is segmented, stop words are removed, and normalization is performed; the image data undergoes geometric transformation; and the text and image data are cleaned to remove data with missing modalities, i.e., data samples without corresponding images or with empty text content. This ensures the integrity and consistency of the model's multimodal input and improves the model's detection accuracy.

[0046] S2. Apply Bert to the preprocessed text data. base The model extracts multi-granularity text features, including coarse-grained text features, medium-grained text features, and fine-grained text features. The VGG19 model is applied to the preprocessed image data to extract multi-granularity image features, including coarse-grained image features, medium-grained image features, and fine-grained image features.

[0047] S21. For the preprocessed text data, a pre-trained BERT dataset is used. base The model extracts text features by obtaining the embedding representation of each token output from the top-level Transformer block as fine-grained text features, the mean-pooled representation output from the intermediate Transformer block as medium-grained text features, and the global semantic vector output from the bottom layer. As a coarse-grained text feature;

[0048] S22. For the preprocessed image data, the pre-trained VGG19 model is used to extract image features. The high-resolution local feature map output by the shallow convolutional network at the front end is obtained as fine-grained image features, the feature map containing texture and shape information output by the convolutional network at the middle end is obtained as medium-grained image features, and the feature vector of the deep fully connected layer at the back end is obtained as coarse-grained image features.

[0049] S3. The text feature enhancement module performs feature aggregation and feature selection on multi-granularity text features and outputs the enhanced text features; the image feature enhancement module performs feature aggregation and feature selection on multi-granularity image features and outputs the enhanced image features.

[0050] In step S3, the text feature enhancement module is used to perform feature aggregation and feature selection on multi-granularity text features, and outputs the enhanced text features, specifically, as follows: Figure 3 :

[0051] S31. Randomly shuffle the multi-granularity text features and evenly divide the shuffled features into two subsets, called the source subset src and the target subset dst, respectively.

[0052] S32. For each feature in the source subset src, calculate its cosine similarity with the features in the target subset dst, and select the feature with the most similarity. Sort all matching pairs in descending order of similarity, and retain the top k pairs with the highest similarity to ensure the relevance of the selected features. Generate a new feature representation as an aggregate feature by averaging the matching pairs in each dimension.

[0053] In step S32, the above aggregation method enables the model to utilize information of different granularities at the same time, capture semantic information at different levels more comprehensively, and thus improve the expressive power of features.

[0054] S33. After obtaining the aggregated features of each modality, the maximum value in each feature dimension is selected as the discrimination parameter to select the most discriminative feature in each dimension. These most discriminative features are then reorganized in the original feature dimension order into a feature vector that has the highest activation response in each semantic dimension, thus obtaining the enhanced text features.

[0055] In step S3, the image feature enhancement module and the text feature enhancement module use the same structure and processing method to obtain enhanced image features. By designing two feature enhancement modules with identical internal algorithm structures, one for text and one for image branches, the most discriminative features in each modality are highlighted, improving the model's sensitivity to fake news features. Feature enhancement mainly consists of two steps: multi-granularity feature aggregation and feature selection, to obtain rich text and visual feature representations. First, the extracted single-modal features are randomly shuffled to break potential order dependencies and ensure randomness in subsequent processing. Then, the shuffled features are evenly divided into two subsets to facilitate subsequent similarity calculations. After obtaining the aggregated features for each modality, the maximum value is used as the discrimination parameter to select the most discriminative features in each dimension, enhancing the model's sensitivity to key information and providing strong support for subsequent comparative learning.

[0056] S4. Align the enhanced text features and enhanced image features by sharing the multi-head self-attention mechanism in the Transformer model, and then concatenate and fuse the aligned text features and image features to obtain fused multimodal features.

[0057] In step S4, the enhanced text features and enhanced image features are input into a shared Transformer. The shared Transformer employs a Transformer architecture based on a multi-head self-attention mechanism and a feedforward neural network in its internal hierarchical structure. The multi-head self-attention mechanism aligns text and image features in a unified embedding space to eliminate semantic gaps. Through a weight-sharing mechanism, text and image branch features are processed under the same set of Transformer parameters, thereby mapping features from different modalities to a unified shared embedding space. This shared embedding space captures deep correlations between modalities, enhancing the semantic association between the two modalities.

[0058] S5. Input the fused multimodal features obtained in step S4 into the classifier to obtain the detection results.

[0059] In step S5, the classifier includes an inversion layer and two fully connected classification layers. The news authenticity classification label and classification loss are obtained through non-linear mapping. .

[0060] S6, by the data preprocessing module, Bert base The model, VGG19 model, text feature enhancement module, image feature enhancement module, shared Transformer model, and classifier constitute a fake news detection model based on multimodal feature enhancement and semantic alignment. After training the fake news detection model based on multimodal feature enhancement and semantic alignment with a composite loss consisting of classification loss and contrast loss, the optimized model is obtained.

[0061] In step S6, the composite loss L is:

[0062]

[0063] Among them, L c To compare the losses, L mixup For comparison of losses, θ represents model parameters. This is the partial derivative of the loss function with respect to the model parameters.

[0064] Comparison loss L mixup :

[0065] ,

[0066] ,

[0067] ,

[0068] in, This indicates that the summation operation is performed on all samples in the current training batch; β1 and β2 are two hyperparameters that represent the distance interval between positive and negative sample pairs to ensure that positive and negative samples can be effectively distinguished during training. To calculate cosine similarity; and These represent the visual feature vector and text feature vector of the current input news sample after enhancement processing, respectively, i.e., the positive sample pair; and These are the hard negative visual samples and text samples in each mini-batch; negative samples generated from visual features. Negative samples generated from text features ; Set α1 and α2 to be sampled from the Beta distribution to enhance the diversity of negative samples; and These represent the generated hard negative visual samples and hard negative text samples, respectively. The hinge loss operator is defined as follows: ; This represents the function for selecting the maximum value; This represents the index optimization function, used to search and lock in values ​​that maximize cosine similarity. The index of negative samples that reaches the maximum value is used to accurately extract the most intrusive hard negative samples; and These represent the visual feature candidate variables and text feature candidate variables used for searching hard negative samples in the mini-batch, respectively. This indicates exclusion constraints.

[0069] In step S6, a new contrastive loss function is designed using a contrastive learning mechanism. The design aims to introduce more challenging hard negative samples to improve the model's detection performance. The novel contrastive loss function consists of multiple parts, primarily optimizing the model by maximizing the similarity between positive sample pairs and minimizing the distance to negative sample pairs. Each term in the loss function is designed to focus on different sample pairs, particularly the introduced hard negative samples and the newly constructed, more challenging hard negative samples. Specifically, a hard negative sample construction strategy based on the Mixup technique is adopted, generating new negative sample pairs through linear combination of existing samples, expressed as... A novel loss function is used to perform contrastive learning on text-image features, thereby reducing the semantic gap between cross-modal features and effectively improving the model's detection performance. The fused features are input into a classifier containing inversion and fully connected layers to obtain the classification loss. Combining and According to the formula The model is jointly optimized to balance the two objectives of news classification accuracy and semantic alignment learning. This is achieved by adjusting the weight parameters. and By balancing news classification and semantic alignment learning tasks, we can optimize model performance and improve the accuracy and robustness of the model in detecting fake news.

[0070] S7. After deploying the optimized model, perform real-time fake news detection.

[0071] In step S7, the optimized model is deployed. Specifically, after building a news detection system using the lightweight Flask framework, the optimized model is deployed within that system. This enables the model to have multimodal input capabilities, supporting real-time fake news detection in real-world scenarios. The model is deployed on the lightweight framework. The trained model is then converted to a format suitable for the lightweight framework to reduce memory usage and improve inference speed.

[0072] This fake news detection method based on multimodal feature enhancement and semantic alignment can significantly improve the accuracy and reliability of fake news detection. Compared with existing methods, the proposed model has significantly improved the performance indicators of fake news detection tasks, demonstrating strong practicality and effectiveness, and providing new ideas and solutions for research and application in related fields.

[0073] This fake news detection method based on multimodal feature enhancement and semantic alignment first performs multimodal preprocessing on the input data to improve data quality and ensure the consistency of the input data, effectively reducing detection errors caused by incomplete data and improving the robustness and consistency of detection; then it utilizes... The model, along with the VGG19 model, extracts multimodal features at three granular levels: coarse, medium, and fine. This enables a more comprehensive capture of the semantic and visual representations of fake news at different levels, enriching feature expression. Building upon this, text and image feature enhancement modules are introduced. Through similarity score aggregation and dimensional feature selection, redundant features are automatically eliminated, and key discriminative information is strengthened, thereby improving the discriminative power of fake news features. Subsequently, a shared Transformer and a hard negative sample contrast learning strategy based on Mixup are used to achieve accurate alignment of text and images in the shared space, effectively narrowing the cross-modal semantic gap. Furthermore, a gradient inversion layer and a joint gradient optimization mechanism are used in the classifier, enabling the model to improve classification accuracy while possessing stronger cross-modal generalization ability and robustness. Finally, the trained model is deployed on a lightweight framework, achieving real-time fake news detection and enhancing the model's practical application value.

[0074] This fake news detection method based on multimodal feature enhancement and semantic alignment can improve the multimodal feature expression capability. In view of the difficulty of traditional detection methods in fully utilizing the detailed information of text and images in feature expression, this invention obtains coarse-grained, medium-grained and fine-grained features of text and images, and uses image feature enhancement module and text feature enhancement module to aggregate key information, thereby improving the discriminative power of multimodal input.

[0075] The method of this invention can enhance semantic alignment capabilities. Since the text and images of fake news are often semantically related at a high level, this invention achieves semantic alignment of multimodal features by introducing a shared Transformer and a contrastive learning module, combined with classification loss. Compared with loss Joint gradient optimization enables the model to more accurately capture cross-modal correlation information.

[0076] To address the common issue of modal missing data in fake news, which leads to incomplete multimodal information, this invention introduces a modal missing handling mechanism during the data preprocessing stage to ensure the integrity of multimodal data. Considering practical application needs, this invention deploys the optimized news detection model within a lightweight framework, enabling real-time response capabilities for fake news detection, improving the deployment adaptability of real-time detection, and enhancing the system's versatility and robustness.

[0077] Figure 4 The implementation introduces a shared Transformer and a contrastive loss function. A diagram showing the comparison of sample distribution before and after. Figure 4 This presentation showcases the visualization comparison results after dimensionality reduction and projection of the high-dimensional multimodal features extracted from the model onto a two-dimensional plane using t-SNE technology. It should be noted that... Figure 4 The horizontal coordinates (t-SNE dimension 1) and vertical coordinates (t-SNE dimension 2) in the dimensionality reduction are the embedding space components. Their values ​​themselves have no specific physical meaning, but the relative distance and clustering state of sample points on this two-dimensional plane intuitively reflect their semantic similarity in the high-dimensional feature space. Figure 4 In (a) and (b), different colors represent the authenticity category of the news (blue represents real news, orange represents fake news), and different shapes represent data modalities (circles represent text features, squares represent image features).

[0078] like Figure 4As shown in (a) of the sub-image before alignment, the sample distribution in the feature space exhibits a clearly scattered state when the core solution of this application is not applied. For the same news event, the corresponding text feature points and image feature points are far apart, failing to form a close correspondence, intuitively revealing a serious cross-modal semantic gap problem. At the same time, sample points of real news category and fake news category overlap, lacking a clear classification boundary. This messy distribution indicates that the feature representation under the existing technology lacks sufficient discriminative power, making it difficult to effectively distinguish between real and fake news, thus limiting the final detection performance.

[0079] and Figure 4 In stark contrast to (a), Figure 4 (b) clearly demonstrates the effectiveness of the invention by showing the aligned subgraph. This is achieved by introducing the shared Transformer structure and the contrastive loss function L, which includes hard negative sample mining, as proposed in this application. mixup Subsequently, the feature distribution underwent fundamental optimization. On the one hand, the text and image features of the same news story clustered tightly, proving that the shared Transformer successfully mapped different modalities to a unified semantic space, achieving deep alignment. On the other hand, the clusters of real news and fake news were significantly separated, forming a clear classification interface. This is due to the contrastive loss function L... mixup By aggressively pushing away the most deceptive hard negative samples, the model's ability to identify and its robustness against fake news is significantly enhanced. In summary, Figure 4 This intuitively demonstrates the significant progress made by this invention in improving the quality of feature representation.

[0080] This invention addresses the problem that fake news detection methods based on multimodal feature enhancement and semantic alignment primarily rely on traditional text analysis techniques in social media. These techniques focus on single multimodal representations or consistency learning, failing to fully utilize the complex cross-modal relationships between text and images. Data imbalance and feature noise issues lead to insufficient generalization ability and low accuracy in real-world scenarios. This invention provides an efficient, accurate, and robust fake news detection method. First, a novel feature enhancement module is designed to dynamically aggregate and filter multimodal features using similarity scoring, focusing on extracting feature information at different granularities to reduce redundant data and enhance key content. Image and text feature enhancement modules effectively aggregate key information from text and visual features, improving the model's feature representation ability for multimodal inputs, thereby more accurately identifying fake news. Second, based on a shared Transformer architecture, text and image features are processed simultaneously, fully leveraging the advantages of multi-head self-attention mechanisms to effectively capture the complex relationships between text and images, reducing the semantic gap in cross-modal features. Finally, a novel contrastive learning loss function was designed, specifically optimized for the embedding spaces of text and image features, ensuring consistency of multimodal inputs within the shared embedding space, thereby achieving accurate image-text semantic alignment. By combining multimodal feature enhancement with contrastive learning techniques, efficient detection and accurate classification of fake news were achieved.

[0081] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting fake news based on multimodal feature enhancement and semantic alignment, characterized in that: Includes the following steps, S1. The data preprocessing module is used to preprocess the acquired text data and image data to obtain preprocessed text data and preprocessed image data. S2. Apply Bert to the preprocessed text data. base The model extracts multi-granularity text features, including coarse-grained text features, medium-grained text features, and fine-grained text features. The VGG19 model is applied to the preprocessed image data to extract multi-granularity image features, including coarse-grained image features, medium-grained image features, and fine-grained image features. S3. The text feature enhancement module performs feature aggregation and feature selection on multi-granularity text features and outputs the enhanced text features; the image feature enhancement module performs feature aggregation and feature selection on multi-granularity image features and outputs the enhanced image features. S4. Align the enhanced text features and enhanced image features by sharing the multi-head self-attention mechanism in the Transformer model, and then concatenate and fuse the aligned text features and image features to obtain fused multimodal features. S5. Input the fused multimodal features obtained in step S4 into the classifier to obtain the detection results; S6, by the data preprocessing module, Bert base The model, VGG19 model, text feature enhancement module, image feature enhancement module, shared Transformer model and classifier constitute a fake news detection model based on multimodal feature enhancement and semantic alignment. After training the fake news detection model based on multimodal feature enhancement and semantic alignment with a composite loss consisting of classification loss and contrast loss, the optimized model is obtained. S7. After deploying the optimized model, perform real-time fake news detection.

2. The fake news detection method based on multimodal feature enhancement and semantic alignment as described in claim 1, characterized in that: In step S1, the acquired text data and image data are preprocessed. Specifically, the text data is segmented, stop words are removed, and the data is standardized. The image data is subjected to geometric transformation. The text data and image data are cleaned to remove data with missing modalities, i.e., data samples with no corresponding image or empty text content.

3. The fake news detection method based on multimodal feature enhancement and semantic alignment as described in claim 1, characterized in that: Step S2, specifically, S21. For the preprocessed text data, a pre-trained BERT dataset is used. base The model extracts text features by obtaining the embedding representation of each token output from the top-level Transformer block as fine-grained text features, the mean-pooled representation output from the intermediate Transformer block as medium-grained text features, and the global semantic vector output from the bottom layer. As a coarse-grained text feature; S22. For the preprocessed image data, the pre-trained VGG19 model is used to extract image features. The high-resolution local feature map output by the shallow convolutional network at the front end is obtained as fine-grained image features, the feature map containing texture and shape information output by the convolutional network at the middle end is obtained as medium-grained image features, and the feature vector of the deep fully connected layer at the back end is obtained as coarse-grained image features.

4. The fake news detection method based on multimodal feature enhancement and semantic alignment as described in any one of claims 1-3, characterized in that: In step S3, a text feature enhancement module is used to perform feature aggregation and feature selection on multi-granularity text features, and outputs the enhanced text features. Specifically, S31. Randomly shuffle the multi-granularity text features and evenly divide the shuffled features into two subsets, called the source subset src and the target subset dst, respectively. S32. For each feature in the source subset src, calculate its cosine similarity with the features in the target subset dst, and select the feature with the most similarity. Sort all matching pairs in descending order of similarity, and retain the top k pairs with the highest similarity to ensure the relevance of the selected features. Generate a new feature representation as an aggregate feature by averaging the matching pairs in each dimension. S33. After obtaining the aggregated features of each modality, the maximum value in each feature dimension is selected as the discrimination parameter to select the most discriminative feature in each dimension. These most discriminative features are then reorganized in the original feature dimension order into a feature vector that has the highest activation response in each semantic dimension, thus obtaining the enhanced text features.

5. The fake news detection method based on multimodal feature enhancement and semantic alignment as described in claim 4, characterized in that: In step S3, the image feature enhancement module and the text feature enhancement module use the same structure and processing method to obtain the enhanced image features.

6. The method for detecting fake news based on multimodal feature enhancement and semantic alignment as described in any one of claims 1-3, characterized in that: In step S6, the composite loss L: ， Among them, L c To compare the losses, L mixup For comparison of losses, θ represents model parameters. This is the partial derivative of the loss function with respect to the model parameters.

7. The fake news detection method based on multimodal feature enhancement and semantic alignment as described in claim 6, characterized in that: Comparison loss L mixup : ， ， ， in, This indicates that the summation operation is performed on all samples in the current training batch; β1 and β2 are two hyperparameters that represent the distance interval between positive and negative sample pairs to ensure that positive and negative samples can be effectively distinguished during training. To calculate cosine similarity; and These represent the visual feature vector and text feature vector of the current input news sample after enhancement processing, respectively, i.e., the positive sample pair; and These are the hard negative visual samples and text samples in each mini-batch; negative samples generated from visual features. Negative samples generated from text features ; Set α1 and α2 to be sampled from the Beta distribution to enhance the diversity of negative samples; and These represent the generated hard negative visual samples and hard negative text samples, respectively. The hinge loss operator is defined as follows: ; This represents the function for selecting the maximum value; This represents the index optimization function, used to search and lock in values ​​that maximize cosine similarity. The index of the negative sample that reaches the maximum value; and These represent the visual feature candidate variables and text feature candidate variables used for searching hard negative samples in the mini-batch, respectively. This indicates exclusion constraints.

8. The fake news detection method based on multimodal feature enhancement and semantic alignment as described in any one of claims 1-3, characterized in that: In step S7, the optimized model is deployed. Specifically, after building the news detection system using the lightweight Flask framework, the optimized model is deployed on the news detection system.

Images