Multimodal fake news detection method based on adaptive fusion

Abstract

The present invention relates to a field, and more specifically to a multimodal fake news detection method based on adaptive fusion, comprising the following steps: extracting unimodal features using text and image encoders respectively; achieving feature complementarity between modalities using cross-attention; projecting the extracted text features into a low-dimensional space for single-view prediction; interactively fusing multimodal features using an adaptive fusion module; performing classification using the fused multimodal features; and training the network using a combined classification loss and a single-view loss. The present invention utilizes the multi-layer semantics of text to capture rich text information, utilizes text view prediction to reduce the impact of multimodal feature complexity on classification results, utilizes a cross-attention module to address heterogeneity between multimodal features, and solves the problems of traditional multimodal feature fusion through an adaptive fusion module.

Description

Technical Field

[0001] The present invention relates to the intersection of natural language processing and computer vision, and in particular to a multimodal fake news detection method based on adaptive fusion. Background Art

[0002] Multimodal content can increase the credibility and spread of fake news, but it also poses greater challenges to detection. Current multimodal fake news detection faces the following challenges:

[0003] (1) Diversity of multimodal clues. In different types of multimodal fake news detection, the relationship between images and text may vary, such as inconsistency between images and text, mutual enhancement between images and text, and text embedded in images. These relationships provide different detection clues, which require the design of appropriate models to capture and integrate.

[0004] (2) The heterogeneity of multimodal data. Images and text are two different data types with significant differences in semantics. For example, object recognition in images and object recognition in text are two different tasks, each requiring its own model. How to effectively transform images and text into a common semantic space and perform effective reasoning is a challenge.

[0005] (3) Fusion of multimodal data. Data from different modalities may have different formats, distributions, dimensions, and densities, which poses challenges to data representation, alignment, and fusion. For example, text is a discrete sequence of symbols, while images are continuous pixel matrices. How to transform them into a common semantic space while preserving their respective features and relationships is a challenge.

[0006] Therefore, in response to the above problems, a multimodal fake news detection method based on adaptive fusion is proposed to solve the above problems. Summary of the Invention

[0007] In view of the shortcomings of the existing technology, the present invention proposes a multimodal fake news detection method based on adaptive fusion.

[0008] The technical solution of the present invention to solve the technical problem is:

[0009] A multimodal fake news detection method based on adaptive fusion, comprising the following steps:

[0010] S1. Use text and image encoders to extract unimodal features respectively, and input text and image into the encoders respectively;

[0011] For text, the pre-trained model BERT is used as the text encoder. The English and Chinese languages ​​are respectively trained using the English pre-trained model BERT-base-uncased and the Chinese pre-trained model BERT-base-chinese. The 12 hidden layer features of BERT are extracted and added together to obtain the feature representation of the text.

[0012] For images, the pre-trained model ResNet50 is used to extract the semantic features of the image. The fourth block is used to obtain the feature representation of the image, and its features are adjusted to 3 dimensions to be consistent with the text feature dimension.

[0013] S2. Use cross-attention to achieve inter-modal feature complementarity. This is achieved through the multimodal pre-trained model ViLBERT, effectively fusing the textual and visual features of news and addressing the heterogeneity between multimodal features.

[0014] S3. Project the extracted text features into a low-dimensional space for single-view prediction. Project the text representation obtained by the text encoder into the low-dimensional space and perform a loss calculation with the true label to obtain the text classification loss and the impact of the single-view classification decision.

[0015] S4. Use adaptive fusion module to perform interactive fusion of multimodal features;

[0016] The extracted text and image feature representations are used as input and compressed in the spatial dimension to obtain a one-dimensional vector. An activation function is used to calculate the weight of each channel. For text, the weight represents the importance of each word to the entire text. For images, the weight represents the importance of each region to the entire image. The global representation of the text and image is obtained through weighted summation, namely the text enhancement feature and image enhancement feature.

[0017] Multiply the text enhancement feature and the image enhancement feature to obtain the visual fusion feature, which represents the projection of the text information in the visual space. Then, the visual fusion feature is normalized and adaptively averaged to obtain the final visual fusion vector.

[0018] Multiply the image enhancement features and text features to obtain text fusion features, which represent the projection of visual information in the text space. Then, perform normalization and adaptive average pooling operations on the text fusion features to obtain the final text fusion vector.

[0019] The text fusion vector and the visual fusion vector are concatenated to obtain the fused multimodal features. At this time, the multimodal channel becomes twice the size of the unimodal channel. The fused multimodal features are fed into two gating units to regulate the information flow of the vectors.

[0020] Channel information is used to emphasize the multimodal information features and suppress the noise impact after multimodal fusion. Specifically, the multimodal vector adjusted by the gated unit is divided into a visual attention vector and a text attention vector, which are added to the original encoding vector respectively. The multimodal features are projected to the same dimension and then spliced ​​to obtain the final multimodal feature fusion vector.

[0021] S5. Use the fused multimodal features for classification, and combine the classification loss and single-view loss for network training. Use the cross-entropy function to calculate the classification loss, and alleviate the impact of category imbalance by amplifying the classification loss. Combined with the text single-view classification loss, the classifier maps the feature vector output by the network into a probability distribution to indicate the probability that the news belongs to each category.

[0022] The formula used in S1 is as follows:

[0023] ,

[0024] ,

[0025] in, Indicates the text content corresponding to the news. Represents the semantic representation of each word after concatenating 12 hidden layer features. Indicates the number of words in the text, Indicates the image content corresponding to the news, Represents the image semantic features extracted by the ResNet50 pre-training model, represents the embedding dimension of the image, Represents an image, Represents text.

[0026] S2 is as follows:

[0027] Given an image and text Indicates that, in the standard Transformer model, 、 、 , each modal 、 This is passed as input to another Transformer’s multi-head attention block, which generates attention pooling features for each modality, i.e., image-to-text attention is performed in the visual stream, and text-to-image attention is performed in the text stream. The rest of the Transformer block continues as before, including the residual layer with the initial representation to produce multimodal features.

[0028] The formula used in S2 is as follows:

[0029] ,

[0030] ,

[0031] and Represents two input modes, 、 、 Represents a certain mode Query, key, value, represents the feature representation after feature complementation, 、 、 represents the weight matrix, represents the embedding dimension, represents the normalization function, The query value representing the text, Represents the query value of the image, The key value representing the text, Indicates the key value of the image, Indicates the value of the text. Indicates the value of the image.

[0032] S3 is as follows: use convolution layer for dimensionality reduction, use BN normalization layer for normalizing feature distribution, use The activation function enhances the network's expressive power, and then the second and third dimensions are flattened, and a BN layer is added. The discard method is used to prevent overfitting, and finally a two-dimensional text feature representation is obtained, which is classified and labeled in advance using a classifier to make losses.

[0033] The formula used in S4 is as follows:

[0034]

[0035]

[0036]

[0037]

[0038] represents element-wise multiplication, represents the dot product multiplication operation, Represents the encoding features of the text, Represents the encoding features of the image, Indicates the text encoding vectors, Indicates the image encoding vectors, Indicates the total number of news texts, represents the global features of the text after weighted summation, Represents the projection vector of text information in the visual space, represents adaptive average pooling, represents normalization, represents a linear operation, represents a convolutional network, Represents the weighted sum of the global visual features;

[0039] , )

[0040]

[0041] Represents the fusion features after the image and text features are spliced ​​together, express activation function, Represents a linear activation function , represents normalization, represents the multimodal vector after adaptive fusion, represents a 1D convolutional network, Represents the projection vector of visual information on the text space;

[0042]

[0043]

[0044] represents the embedding dimension, Set the value to 256 to keep the embedding dimensions of text and image consistent; represents the probability distribution of text information, represents the probability distribution of visual information, Indicates the dimension Text feature encoding, Indicates the dimension Image feature encoding, represents the adaptive fused text vector, Represents the adaptively fused visual vector;

[0045]

[0046]

[0047] Indicates text mode, Indicates image modality.

[0048] The formula used in S5 is as follows:

[0049]

[0050]

[0051] represents the cross entropy function, represents the true label, Represents the predicted value, Represents the classification loss after feature fusion, represents the loss of text unimodal classification, Represents the total loss function.

[0052] The present invention also provides a multimodal fake news detection system based on adaptive fusion, comprising the following modules:

[0053] Unimodal feature extraction module, used to construct feature extraction for text and images in news;

[0054] Multimodal feature complementation module, using Transformer multi-head attention blocks to fuse text and image features;

[0055] The single-view prediction module projects the extracted text features into a low-dimensional space for feature representation, and uses a classifier to pre-classify and label them for loss to study the impact of text features on decision-making;

[0056] The multimodal feature interactive fusion module performs dot product operations on the text enhancement features and the initial visual features, normalizes them, and performs adaptive average pooling on the visual fusion features to obtain the final visual fusion vector. The image enhancement features and text features are multiplied and normalized, and an adaptive average operation is performed on the text fusion features to obtain the final text fusion vector. The text fusion vector and the visual fusion vector are concatenated to obtain the fused multimodal features.

[0057] The training module uses the multimodal features after the adaptive fusion module as the final classification features and uses the cross entropy function to calculate the classification loss.

[0058] The present invention also relates to an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, characterized in that when the processor executes the computer program, a multimodal fake news detection method based on adaptive fusion is implemented.

[0059] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a multimodal fake news detection method based on adaptive fusion.

[0060] The above technical solution has the following advantages or beneficial effects:

[0061] 1) To address the problem of text feature extraction, this paper integrates Bert12 hidden layer features and uses language information at different levels to enrich the final representation, enabling the model to obtain more comprehensive text representation information;

[0062] 2) The adaptive fusion module solves the problem of multimodal feature fusion;

[0063] 3) In response to the noise impact caused by multimodal fusion, this paper adopts the method of text unimodal prediction to alleviate the impact of noise on classification results;

[0064] 4) To address the problem of imbalanced data sets, increase the classification loss to increase the weight of minority samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0066] Figure 1 Flowchart of the present invention.

[0067] Figure 2 A diagram of the framework for multimodal fake news detection designed for this invention. DETAILED DESCRIPTION

[0068] To make the objectives, technical solutions, and advantages of the present invention more clear and understandable, the present invention is further described below with reference to the accompanying drawings and specific embodiments. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0069] Example 1

[0070] like Figure 1-2 As shown, a multimodal fake news detection method based on adaptive fusion includes the following steps:

[0071] S1. Use text and image encoders to extract unimodal features respectively, and input text and image into the encoders respectively;

[0072] For text, we used the pre-trained BERT model as the text encoder. The full name of the pre-trained BERT model is Bidirectional Encoder Representation from Transformers, which is a transformer-based bidirectional encoder. For English and Chinese, we used the English pre-trained model BERT-base-uncased and the Chinese pre-trained model BERT-base-chinese, respectively. We extracted BERT's 12 hidden layer features and added them together to obtain the feature representation of the text.

[0073] For images, we use the pre-trained ResNet50 model to extract semantic features. The pre-trained ResNet50 model represents a residual network with 49 convolutional layers and one fully connected layer. We use the fourth block to obtain the feature representation of the image and adjust its features to 3D to keep consistent with the text feature dimension.

[0074] S2. Use cross-attention to achieve inter-modal feature complementarity. This is achieved through the multimodal pre-trained model ViLBERT, effectively fusing the textual and visual features of news and addressing the heterogeneity between multimodal features.

[0075] S3. Multimodal fusion has been shown to be more effective than single-modality in classification tasks. However, the heterogeneity of different modalities can introduce noise into the fused features. This paper explores the impact of text features on decision-making from a single-modality perspective. The extracted text features are projected into a low-dimensional space for single-view prediction. The text representation obtained by the text encoder is then projected into the low-dimensional space and compared with the true label to obtain the text classification loss and the impact of single-view classification decisions.

[0076] S4. Use adaptive fusion module to perform interactive fusion of multimodal features;

[0077] The present invention designs an adaptive fusion module based on channel attention. The extracted text and image feature representations are used as input and compressed in the spatial dimension to obtain a one-dimensional vector. An activation function is used to calculate the weight of each channel. For text, the weight represents the importance of each word to the entire text; for images, the weight represents the importance of each block to the entire image. The global representation of the text and image, namely the text enhancement feature and the image enhancement feature, is obtained through weighted summation. The text enhancement feature and the visual feature are then multiplied together to obtain the visual fusion feature, which represents the projection of the text information in the visual space.

[0078] The visual fusion features are then normalized and adaptively averaged to obtain the final visual fusion vector; the image enhancement features and text features are multiplied together to obtain the text fusion features, which represent the projection of visual information in the text space. The text fusion features are then normalized and adaptively averaged to obtain the final text fusion vector.

[0079] The text fusion vector and the visual fusion vector are spliced ​​to obtain the fused multimodal features. At this time, the multimodal channel becomes twice the size of the unimodal channel. The fused multimodal features are sent to two gating units to regulate the information flow of the vector. Affected by the SE network, the present invention uses channel information to emphasize the multimodal information features and suppress the noise influence after multimodal fusion. Specifically, the multimodal vector adjusted by the gating unit is divided into a visual attention vector and a text attention vector, which are added to the original encoding vector respectively, and the multimodal features are projected to the same dimension and then spliced ​​to obtain the final multimodal feature fusion vector.

[0080] S5. Use the fused multimodal features for classification, and combine the classification loss and single-view loss for network training. Use the cross-entropy function to calculate the classification loss, and alleviate the impact of category imbalance by amplifying the classification loss. Combined with the text single-view classification loss, the classifier maps the feature vector output by the network into a probability distribution to indicate the probability that the news belongs to each category.

[0081] The formula used in S1 is as follows:

[0082] ,

[0083] ,

[0084] in, Indicates the text content corresponding to the news. Represents the semantic representation of each word after concatenating 12 hidden layer features. Indicates the number of words in the text, Indicates the image content corresponding to the news, Represents the image semantic features extracted by the ResNet50 pre-training model, represents the embedding dimension of the image, Represents an image, Represents text.

[0085] S2 is as follows:

[0086] Given an image and text Indicates that, in the standard Transformer model, 、 、 , each modal 、 This is passed as input to another Transformer’s multi-head attention block, which generates attention pooling features for each modality, i.e., image-to-text attention is performed in the visual stream, and text-to-image attention is performed in the text stream. The rest of the Transformer block continues as before, including the residual layer with the initial representation to produce multimodal features.

[0087] The formula used in S2 is as follows:

[0088] ,

[0089] ,

[0090] and Represents two input modes, 、 、 Represents a certain mode Query, key, value, represents the feature representation after feature complementation, 、 、 represents the weight matrix, represents the embedding dimension, represents the normalization function, The query value representing the text, Represents the query value of the image, The key value representing the text, Indicates the key value of the image, Indicates the value of the text. Indicates the value of the image.

[0091] S3 is as follows: use convolution layer for dimensionality reduction, use BN normalization layer for normalizing feature distribution, use The activation function enhances the network's expressive power, and then the second and third dimensions are flattened, and a BN layer is added. The discard method is used to prevent overfitting, and finally a two-dimensional text feature representation is obtained, which is classified and labeled in advance using a classifier to make losses.

[0092] The formula used in S4 is as follows:

[0093]

[0094]

[0095]

[0096]

[0097] represents element-wise multiplication, represents the dot product multiplication operation, Represents the encoding features of the text, Represents the encoding features of the image, Indicates the text encoding vectors, Indicates the image encoding vectors, Indicates the total number of news texts, represents the global features of the text after weighted summation, Represents the projection vector of text information in the visual space, represents adaptive average pooling, represents normalization, represents a linear operation, represents a convolutional network, Represents the weighted sum of the visual global features;

[0098] , )

[0099]

[0100] Represents the fusion features after the image and text features are spliced ​​together, express activation function, Represents a linear activation function , represents normalization, represents the multimodal vector after adaptive fusion, represents a 1D convolutional network, Represents the projection vector of visual information on the text space;

[0101]

[0102]

[0103] represents the embedding dimension, Set the value to 256 to keep the embedding dimensions of text and image consistent; represents the probability distribution of text information, represents the probability distribution of visual information, Indicates the dimension Text feature encoding, Indicates the dimension Image feature encoding, represents the adaptive fused text vector, Represents the adaptively fused visual vector;

[0104]

[0105]

[0106] Indicates text mode, Indicates image modality.

[0107] Finally, we get the adaptive fusion of image and text vectors, and combine the two modalities 、 Adjust to the appropriate dimension for classification and predict the truth or falsity of news.

[0108] The formula used in S5 is as follows:

[0109]

[0110]

[0111] represents the cross entropy function, represents the true label, Represents the predicted value, Represents the classification loss after feature fusion, represents the loss of text unimodal classification, Represents the total loss function.

[0112] The following table shows the detection results of the present invention on Weibo, weibo21, gossip entertainment, and politi political data sets.

[0113]

[0114] Example 2

[0115] A multimodal fake news detection system based on adaptive fusion, including the following modules:

[0116] Unimodal feature extraction module, used to construct feature extraction for text and images in news;

[0117] Multimodal feature complementation module, using Transformer multi-head attention blocks to fuse text and image features;

[0118] The single-view prediction module projects the extracted text features into a low-dimensional space for feature representation, and uses a classifier to pre-classify and label them for loss to study the impact of text features on decision-making;

[0119] The multimodal feature interactive fusion module performs dot product operations on the text enhancement features and the initial visual features, normalizes them, and performs adaptive average pooling on the visual fusion features to obtain the final visual fusion vector. The image enhancement features and text features are multiplied and normalized, and an adaptive average operation is performed on the text fusion features to obtain the final text fusion vector. The text fusion vector and the visual fusion vector are concatenated to obtain the fused multimodal features.

[0120] The training module uses the multimodal features after the adaptive fusion module as the final classification features and uses the cross entropy function to calculate the classification loss.

[0121] Example 3

[0122] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a multimodal fake news detection method based on adaptive fusion when executing the computer program.

[0123] Example 4

[0124] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a multimodal fake news detection method based on adaptive fusion.

[0125] Through the above technical solution, the present invention proposes a multimodal fake news detection method and system based on adaptive fusion, which uses the multi-layer semantics of text to capture rich text information. To address the problem of text feature extraction, the present invention integrates Bert12 hidden layer features and uses language information at different levels to enrich the final representation, so that the model obtains more comprehensive text representation information; uses text perspective prediction to reduce the impact of multimodal feature complexity on classification results, and solves the problem of multimodal feature fusion by referencing an adaptive fusion module; uses a cross-attention module to solve the heterogeneity between multimodal features; to address the noise impact brought by multimodal fusion, the present invention adopts text unimodal prediction to alleviate the impact of noise on classification results; to address the problem of class imbalance in the data set, increases the classification loss to increase the weight of small samples.

[0126] Although the above describes the specific implementation methods of the invention in conjunction with the accompanying drawings, it does not limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A multimodal fake news detection method based on adaptive fusion, characterized by: The following steps are involved: S1. Use text and image encoders to extract unimodal features respectively, and input text and image into the encoders respectively; For text, the pre-trained model BERT is used as the text encoder. The English and Chinese languages ​​are respectively trained using the English pre-trained model BERT-base-uncased and the Chinese pre-trained model BERT-base-chinese. The 12 hidden layer features of BERT are extracted and added together to obtain the feature representation of the text. For images, the pre-trained model ResNet50 is used to extract the semantic features of the image. The fourth block is used to obtain the feature representation of the image, and its features are adjusted to 3 dimensions to be consistent with the text feature dimension. S2. Use cross attention to achieve feature complementarity between modalities, and use the multimodal pre-training model ViLBERT to achieve feature complementarity between modalities, and use the text features of news and visual features Perform effective fusion to resolve the heterogeneity between multimodal features; S3. Project the extracted text features into a low-dimensional space for single-view prediction. Project the text representation obtained by the text encoder into the low-dimensional space and perform a loss calculation with the true label to obtain the text classification loss and the impact of the single-view classification decision. S4. Use adaptive fusion module to perform interactive fusion of multimodal features; The extracted text and image feature representations are used as input and compressed in the spatial dimension to obtain a one-dimensional vector. An activation function is used to calculate the weight of each channel. For text, the weight represents the importance of each word to the entire text. For images, the weight represents the importance of each region to the entire image. The global representation of the text and image is obtained through weighted summation, namely the text enhancement feature and image enhancement feature. Multiply the text enhancement feature and the image enhancement feature to obtain the visual fusion feature, which represents the projection of the text information in the visual space. Then, the visual fusion feature is normalized and adaptively averaged to obtain the final visual fusion vector. Multiply the image enhancement feature and the text enhancement feature to obtain the text fusion feature, which represents the projection of the visual information in the text space. Then, the text fusion feature is normalized and adaptively averaged pooled to obtain the final text fusion vector. The text fusion vector and the visual fusion vector are concatenated to obtain the fused multimodal features. At this time, the multimodal channel becomes twice the size of the unimodal channel. The fused multimodal features are fed into two gating units to regulate the information flow of the vectors. Channel information is used to emphasize the multimodal information features and suppress the noise impact after multimodal fusion. Specifically, the multimodal vector adjusted by the gated unit is divided into a visual attention vector and a text attention vector, which are added to the original encoding vector respectively. The multimodal features are projected to the same dimension and then spliced ​​to obtain the final multimodal feature fusion vector. S5. Use the fused multimodal features for classification, and combine the classification loss and single-view loss for network training. Use the cross-entropy function to calculate the classification loss, and alleviate the impact of category imbalance by amplifying the classification loss. Combined with the text single-view classification loss, the classifier maps the feature vector output by the network into a probability distribution to indicate the probability that the news belongs to each category.

2. The multimodal fake news detection method based on adaptive fusion according to claim 1, characterized in that: The formula used in S1 is as follows: ， ， in, Indicates the text content corresponding to the news. Represents the semantic representation of each word after concatenating 12 hidden layer features. Indicates the number of words in the text, Indicates the image content corresponding to the news, Represents the image semantic features extracted by the ResNet50 pre-training model, represents the embedding dimension of the image, Represents an image, Represents text.

3. The multimodal fake news detection method based on adaptive fusion according to claim 1, characterized in that: S2 is as follows: Given an image and text Indicates that, in the standard Transformer model, 、 、 , each modal 、 This is passed as input to another Transformer’s multi-head attention block, which generates attention pooling features for each modality, i.e., image-to-text attention is performed in the visual stream, and text-to-image attention is performed in the text stream. The rest of the Transformer block continues as before, including the residual layer with the initial representation to produce multimodal features. The formula used in S2 is as follows: ， ， and Represents two input modes, representing text and image respectively, 、 、 Represents a certain mode Query, key, value, represents the feature representation after feature complementation, 、 、 represents the weight matrix, represents the embedding dimension, represents the normalization function, The query value representing the text, Represents the query value of the image, The key value representing the text, Indicates the key value of the image, Indicates the value of the text. Indicates the value of the image.

4. The multimodal fake news detection method based on adaptive fusion according to claim 1, characterized in that: S3 is as follows: use convolution layer for dimensionality reduction, use BN normalization layer for normalizing feature distribution, use The activation function enhances the network's expressive power, and then the second and third dimensions are flattened, and a BN layer is added. The discard method is used to prevent overfitting, and finally a two-dimensional text feature representation is obtained, which is classified and labeled in advance using a classifier to make losses.

5. The multimodal fake news detection method based on adaptive fusion according to claim 1, characterized in that: The formula used in S4 is as follows: ， ， ， ， represents element-wise multiplication, represents the dot product multiplication operation, Represents the encoding features of the text, Represents the encoding features of the image, Indicates the text encoding vectors, Indicates the image encoding vectors, Indicates the total number of news texts, represents the global features of the text after weighted summation, Represents the projection vector of text information in the visual space, represents adaptive average pooling, represents normalization, represents a linear operation, represents a convolutional network, Represents the weighted sum of the global visual features; , )， ， Represents the fusion features after the image and text features are spliced ​​together, express activation function, Represents a linear activation function , represents normalization, represents the multimodal vector after adaptive fusion, represents a 1D convolutional network, Represents the projection vector of visual information on the text space; ， ， represents the embedding dimension, Set the value to 256 to keep the embedding dimensions of text and image consistent; represents the probability distribution of text information, represents the probability distribution of visual information, Indicates the dimension Text feature encoding, Indicates the dimension Image feature encoding, represents the adaptive fused text vector, Represents the adaptively fused visual vector; ， ， Indicates text mode, Indicates image modality.

6. The multimodal fake news detection method based on adaptive fusion according to claim 1, characterized in that: The formula used in S5 is as follows: ， ， represents the cross entropy loss function, represents the true label, Represents the predicted value, Represents the classification loss after feature fusion, represents the loss of text unimodal classification, Represents the total loss function.

7. A multimodal fake news detection system based on adaptive fusion, characterized by: Unimodal feature extraction module, used to construct feature extraction for text and images in news; Multimodal feature complementation module, using Transformer multi-head attention blocks to fuse text and image features; The single-view prediction module projects the extracted text features into a low-dimensional space for feature representation, and uses a classifier to pre-classify and label them for loss to study the impact of text features on decision-making; The multimodal feature interactive fusion module performs a dot product operation on the text enhancement features and the initial visual features, normalizes them, and performs an adaptive average pooling operation on the visual fusion features to obtain the final visual fusion vector. The image enhancement features and text enhancement features are multiplied and normalized, and an adaptive average operation is performed on the text fusion features to obtain the final text fusion vector. The text fusion vector and the visual fusion vector are concatenated to obtain the fused multimodal features. The training module uses the multimodal features after the adaptive fusion module as the final classification features and uses the cross entropy function to calculate the classification loss.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the adaptive fusion-based multimodal fake news detection method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the multimodal fake news detection method based on adaptive fusion as described in any one of claims 1 to 6.

Images