Multi-modal false news data processing method and device, equipment and medium

Abstract

The invention discloses a multi-modal false news data processing method and device, equipment and a medium, and relates to the technical field of data processing. Compared with the prior art, the feature data of the text data and the feature data of the image data are subjected to comparative learning processing to obtain comparative learning processing of comparative learning loss, isomerism between different modals is reduced, accurate alignment of cross-modal information is achieved, and semantic association between the text and the image can be better captured through the aligned features. A mutual learning mechanism for performing mutual learning processing on the first data and fusion variables corresponding to the image data and the text data to obtain mutual learning loss is introduced, and collaborative learning among modals is realized through information sharing while respective modal characteristics of the text and the image are maintained. The mechanism enables the model to automatically balance the importance of different modes.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, equipment and medium for processing multimodal false news data. Background Art

[0002] In the field of multimodal fake news detection, modal gap is a common problem. Due to the differences in the representation of different modalities (such as text and images), there is significant heterogeneity, which affects the integration and coordination of information and increases the difficulty of detection.

[0003] Existing multimodal detection methods often have difficulty in effectively fusing information from different modalities, resulting in inaccurate judgments on news authenticity. For example, key information in an image may not be fully conveyed to a text model through existing feature extraction methods, which can lead to missing information and inconsistent detection results, making information fusion difficult. At the same time, the imbalance in information expression between different modalities can also exacerbate the modal gap, causing the model to over-rely on one modality and ignore the important features of another modality. This situation can easily lead to information bias, making it difficult for the detection system to make comprehensive and accurate judgments.

[0004] Therefore, how to reduce the modal gap and achieve fusion and coordination among the modes is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a method, device, equipment and medium for processing multimodal false news data to solve the problems of modal gap and multimodal imbalance that exist in information fusion difficulties.

[0006] In order to solve the above technical problems, the present invention provides a method for processing multimodal fake news data, comprising:

[0007] Acquire text data and image data corresponding to the fake news data, and perform visual description processing on the image data to obtain description data;

[0008] The text data, the image data and the description data are respectively subjected to embedding layer processing to obtain respective feature data; the feature data of the text data and the image data are respectively subjected to contrastive learning processing to obtain contrastive learning loss;

[0009] The feature data corresponding to the text data, the image data and the description data are subjected to mutual modal fusion processing to obtain corresponding fusion variables; wherein the fusion variables include a first fusion variable, a second fusion variable and a third fusion variable; the first fusion variable is a variable obtained by fusion processing with the text data with the image data as the leading factor; the second fusion variable is a variable obtained by fusion processing with the text data with the description data as the leading factor; the third fusion variable is a variable obtained by fusion processing with the image data with the text data as the leading factor;

[0010] Perform secondary fusion processing on the first fusion variable and the second fusion variable to obtain first data; perform mutual learning processing on the first data and the third fusion variable to obtain mutual learning loss;

[0011] A loss value corresponding to the fake news data is determined based on the contrastive learning loss and the mutual learning loss.

[0012] On the one hand, the text data, the image data and the description data are respectively subjected to embedding layer processing to obtain respective feature data, including:

[0013] Segmenting the text data to obtain first labeled data;

[0014] Inputting the first labeled data into a pre-trained language representation model to obtain text feature data corresponding to the text data;

[0015] Segmenting the description data to obtain second labeled data;

[0016] Inputting the second labeled data into a pre-trained language representation model to obtain description feature data corresponding to the description data;

[0017] Segmenting the image data to obtain block sequence data;

[0018] The block sequence data is input into a value image encoder to obtain image feature data corresponding to the image data.

[0019] On the other hand, performing contrastive learning processing on the feature data of the text data and the image data to obtain contrastive learning loss includes:

[0020] Acquire corresponding multiple text data and multiple image data in the same batch; wherein the multiple text data and multiple image data exist in the form of text-image pairs;

[0021] Using first target image data corresponding to first target text data as first positive sample data, and using other image data other than the first target image data corresponding to the first target text data as first negative sample data;

[0022] Using the second target text data corresponding to the second target image data as the second positive sample data, and using other text data corresponding to the second target image data except the second target text data as the second negative sample data;

[0023] Processing the first positive sample data, the first negative sample data, the second positive sample data, and the second negative sample data to obtain a first similarity matrix;

[0024] The contrastive learning loss is obtained according to the first similarity matrix.

[0025] On the other hand, performing secondary fusion processing on the first fusion variable and the second fusion variable to obtain first data includes:

[0026] Input the first fused variable and the second fused variable into the multi-head attention layer to obtain the first variable corresponding to each head;

[0027] Performing a linear transformation on the first variable corresponding to each head to obtain a first parameter;

[0028] Establishing a residual connection between the feature data corresponding to the description data and the first parameter to obtain a second variable;

[0029] The second variable is connected using a feedforward network and a residual connection to obtain the first data.

[0030] On the other hand, performing mutual learning processing on the first data and the third fusion variable to obtain a mutual learning loss includes:

[0031] Perform loss processing on the first data and the third fused variable respectively to obtain respective loss values;

[0032] The mutual learning loss is obtained by processing the loss values ​​of the first data and the third fusion variable through KL divergence.

[0033] On the other hand, determining the loss value corresponding to the fake news data based on the contrastive learning loss and the mutual learning loss includes:

[0034] Processing the first fused variable and the first variable by a classifier to obtain a first predicted value;

[0035] Obtaining a label corresponding to the fake news data and using it as a first true value;

[0036] Processing the first predicted value and the first true value by a cross entropy function to obtain a first loss;

[0037] The contrastive learning loss, the mutual learning loss and the first loss are weightedly summed to obtain a final loss value.

[0038] On the other hand, performing visual description processing on the image data to obtain description data includes:

[0039] Get a pre-trained model for representing visual descriptions;

[0040] The image data is input into a pre-trained model for characterizing visual descriptions to obtain the description data.

[0041] In order to solve the above technical problems, the present invention further provides a multimodal false news data processing device, comprising:

[0042] An acquisition module, used to acquire text data and image data corresponding to the fake news data, and perform visual description processing on the image data to obtain description data;

[0043] A first processing module is used to perform embedding layer processing on the text data, the image data and the description data to obtain respective feature data; and perform contrastive learning processing on the feature data of the text data and the image data to obtain contrastive learning loss;

[0044] The second processing module is used to perform mutual modal fusion processing on the feature data corresponding to each of the text data, the image data and the description data to obtain corresponding fusion variables; wherein the fusion variables include a first fusion variable, a second fusion variable and a third fusion variable; the first fusion variable is a variable obtained by performing fusion processing with the text data with the image data as the leading factor; the second fusion variable is a variable obtained by performing fusion processing with the text data with the description data as the leading factor; the third fusion variable is a variable obtained by performing fusion processing with the image data with the text data as the leading factor;

[0045] A third processing module is used to perform secondary fusion processing on the first fusion variable and the second fusion variable to obtain first data; and perform mutual learning processing on the first data and the third fusion variable to obtain mutual learning loss;

[0046] A determination module is used to determine the loss value corresponding to the fake news data based on the contrastive learning loss and the mutual learning loss.

[0047] In order to solve the above technical problems, the present invention also provides a multimodal fake news data processing device, comprising:

[0048] Memory for storing computer programs;

[0049] A processor is used to implement the steps of the method for processing multimodal fake news data as described above when executing the computer program.

[0050] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for processing multimodal false news data as described above are implemented.

[0051] The present invention provides a method for processing multimodal false news data, comprising: obtaining text data and image data corresponding to the false news data, and performing visual description processing on the image data to obtain description data; performing embedding layer processing on the text data, image data and description data to obtain respective feature data; performing contrast learning processing on the feature data of the text data and the image data to obtain contrast learning loss; performing mutual modal fusion processing on the feature data corresponding to the text data, the image data and the description data to obtain corresponding fusion variables; wherein the fusion variables include a first fusion variable, a second fusion variable and a third fusion variable; the first fusion variable is a variable obtained by performing fusion processing with the text data with the image data as the leading factor; the second fusion variable is a variable obtained by performing fusion processing with the text data with the description data as the leading factor; the third fusion variable is a variable obtained by performing fusion processing with the image data with the text data as the leading factor; performing secondary fusion processing on the first fusion variable and the second fusion variable to obtain first data; performing mutual learning processing on the first data and the third fusion variable to obtain mutual learning loss; and determining the loss value corresponding to the false news data based on the contrast learning loss and the mutual learning loss. The present invention reduces the heterogeneity between different modalities through contrastive learning processing, realizes the precise alignment of cross-modal information, and can better capture the semantic association between text and image through the aligned features, thereby more comprehensively integrating multimodal information in the process of false news detection and improving the accuracy of detection. A mutual learning mechanism is introduced to achieve collaborative learning between modalities through information sharing while maintaining the respective modal features of text and image. This mechanism enables the model to automatically balance the importance of different modalities, avoid over-reliance on a single modality under different news content, and improve the robustness of detection. In addition, the introduction of mutual learning helps to deal with the problem of imbalance in multimodal data, so that the system can adapt to a wider range of news scenarios. Through progressive feature fusion and alignment strategies, a balance between detection accuracy and efficiency is achieved, making it suitable for high-complexity false news detection applications.

[0052] In addition, the present invention also provides a multimodal false news data processing device, equipment and medium, which have the same beneficial effects as the above-mentioned multimodal false news data processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A flowchart of a method for processing multimodal false news data provided by an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of the structure of a system architecture of a fake news detection system provided by the present invention;

[0056] Figure 3 A data flow diagram of a system architecture of a fake news detection system provided by an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of a computing model of a fake news detection system provided by an embodiment of the present invention;

[0058] Figure 5 A schematic diagram of the internal structure of a modality fusion module provided in an embodiment of the present invention;

[0059] Figure 6 A structural diagram of a multimodal false news data processing device provided by an embodiment of the present invention;

[0060] Figure 7 A structural diagram of a multimodal fake news data processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0062] The core of the present invention is to provide a method, device, equipment and medium for processing multimodal false news data to solve the problems of modal gap and multimodal imbalance in information fusion difficulties.

[0063] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0064] In the modern information dissemination environment, the widespread spread of fake news has had a significant negative impact on society. Especially on platforms such as social media and news websites, fake news, due to its low cost and rapid spread, can easily have an adverse impact on public opinion, social stability, and even national security. Therefore, how to accurately and timely detect and suppress the spread of fake news has become an important research topic.

[0065] In fake news detection, the combination of multimodal information (such as text and images) has shown higher detection accuracy. Compared with single-modal information, multimodal methods can verify the authenticity of news from multiple perspectives. Existing fake news detection methods are mainly divided into the following categories:

[0066] Single-modal methods: Early fake news detection methods mostly rely on text analysis, using natural language processing technology to analyze the vocabulary and syntactic features of articles to identify false information. However, single-modal methods can only be based on text content, ignoring the fact that fake news is often accompanied by misleading images.

[0067] Multimodal methods: With the advancement of multimodal data processing technology, researchers have begun to combine text and images to improve the detection of fake news. Multimodal methods can more accurately judge the authenticity of news by jointly analyzing image and text information. This method usually extracts image and text features and then fuses these features. However, the heterogeneity of images and text often makes it difficult to directly fuse different modalities in the same space, affecting the accuracy of detection.

[0068] Although the current multimodal false news detection technology has made some progress, it still faces many challenges. For example, due to the significant difference (heterogeneity) in the expression of text and image information, the existing methods are not effective in fusing information. In addition, the imbalance of multimodal data in structure and data density further increases the difficulty of false news detection. The method for processing multimodal false news data provided by the present invention can solve the above technical problems.

[0069] Figure 1 A flowchart of a method for processing multimodal false news data provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0070] S11: Obtain text data and image data corresponding to the fake news data, and perform visual description processing on the image data to obtain description data.

[0071] Specifically, Figure 2 A structural diagram of the system architecture of a fake news detection system provided by the present invention is shown in FIG. Figure 2As shown in Figure 1, the description covers the various components of the system and the interactions between them. Data Collection Module 1: The data collection module is mainly responsible for obtaining relevant information from various data sources. These data sources can include social media platforms, news websites, and user-submitted data. In the fake news detection system, the main task of this module is to continuously capture and collect multimodal information such as text and images to provide subsequent analysis and detection. Management Module 2: The management module is mainly used to control and coordinate the operation of each module. It is responsible for managing the data collection process and controlling whether the data needs to enter the next step of processing. At the same time, the management module provides access and control interfaces for the system, enabling administrators to start or stop data collection and detection functions to ensure that the data enters the subsequent modules for processing at the right time. Preprocessing Module 3: The main task of the preprocessing module is to format and clean the collected data, and use large models to identify images to extract relevant information and express it in the form of text, so as to provide high-quality data input for subsequent detection steps. Preprocessing includes text denoising, image standardization, and other feature extraction steps to ensure that the input data is structured and consistent, thereby improving the accuracy of the fake news detection model. Detection Module 4: The detection module is the core module of the system, which mainly realizes the identification and judgment of fake news. This module uses a deep learning model based on contrastives and mutual learning, combined with multimodal text and image features to detect the authenticity of news. It is the part of the entire system responsible for actual judgment and classification, and outputs the final true or false results. Display module 5: The display module is used to visualize the detection results and system status. It will dynamically display the detected false news information, alarm conditions, and other related log data. Users can view the detection results, system operation status, data history records, etc. through the display module, making the operation and monitoring of the system more intuitive. Storage module 6: The storage module is used to save all data in the system, including original acquisition data, preprocessed data, detection results, and related log information. The storage module uses a database to record and manage this data, and provides an access interface to the outside world, allowing administrators to query historical data and generated detection reports. This module ensures the persistence and traceability of system data.

[0072] Figure 3 A data flow diagram of a system architecture of a fake news detection system provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the data collection module 1 needs to capture from the multimodal information website, collect through the data collection server, input into the network algorithm model to return the network multimodal information, the network algorithm model sends data requests in real time, the network algorithm model outputs the judgment results, and issues a warning.

[0073] Figure 4 A schematic diagram of a computational model of a fake news detection system provided by an embodiment of the present invention is shown in FIG. Figure 4As shown, the network model embedded in the detection module includes six main components. The first is the visual description module 7, which is used to describe the picture through a large model, enrich the picture information, and obtain text data through image data. Secondly, the embedding layer 8 extracts features from multimodal data to obtain corresponding feature data. Next, the text data, image data, and text generated by image data are encoded in the modal alignment module 9, and then the text data and image data are aligned through contrast learning. After obtaining the contrast loss, the features of each modality are fused through the modal fusion module 10. Then, the visual information is finally fused through the visual modality fusion module 11. Then, the modal gap is further reduced through joint learning in the modal collaboration module 12. Finally, the classifier module for outputting the diagnosis result, that is, the detection result classifier 13, is constructed using a multi-layer perceptron. The classifier passes the extracted feature map as input to the hidden layer, and then obtains the final result through the activation function and the softmax function on the output layer.

[0074] Regarding the multimodal data of text data and image data corresponding to the fake news data, it can be crawled by crawling, and the acquisition can be based on certain databases, etc. There is no limitation here, and it can be set according to the actual situation.

[0075] In some embodiments, performing visual description processing on the image data to obtain description data includes:

[0076] Get a pre-trained model for representing visual descriptions;

[0077] The image data is input into a pre-trained model for representing visual descriptions to obtain description data.

[0078] For image data, corresponding to the generation of description data, the Large Language and Vision Assistant (LLaVA) model can be used to generate descriptions, and other visual modules can also be used to generate descriptions. There is no limitation here, and it can be set according to the actual situation. Visual data usually contains rich contextual information, which is crucial for accurately detecting fake news.

[0079] LLaVA is an advanced model that integrates advanced language and visual processing techniques. It can generate detailed text descriptions that include the basic visual elements and context of the image. Formally, given the i-th sample image Ii, the LLaVA model will generate the following description:

[0080] ;

[0081] This embodiment provides a method of obtaining corresponding description data using a pre-trained model for image data, thereby enhancing feature information of the original image.

[0082] S12: Perform embedding layer processing on the text data, image data and description data to obtain their respective feature data; perform contrastive learning processing on the feature data of the text data and image data to obtain contrastive learning loss.

[0083] Specifically, the data corresponding to the three modalities are processed in the embedding layer to obtain their respective feature data. Here, feature extraction can be used for processing, and other methods can also be used for processing.

[0084] In some embodiments, the text data, the image data, and the description data are respectively subjected to embedding layer processing to obtain respective feature data, including:

[0085] Segmenting the text data to obtain first labeled data;

[0086] Inputting the first labeled data into a pre-trained language representation model to obtain text feature data corresponding to the text data;

[0087] Segmenting the description data to obtain second labeled data;

[0088] Inputting the second labeled data into the pre-trained language representation model to obtain description feature data corresponding to the description data;

[0089] Segment the image data to obtain block sequence data;

[0090] The block sequence data is input into the value image encoder to obtain image feature data corresponding to the image data.

[0091] It can be understood that the text is first divided into a series of tags, namely the first tag data: Text = {[CLS], w1, ..., w n-1}, where [CLS] represents the global tag and n is the length of all tags. Then, the text is input into a pre-trained language representation model, such as Bidirectional Encoder Representations from Transformers (BERT), to obtain the text feature T∈R^(n×dt), which is represented as follows:

[0092] ;

[0093] Among them, t h ∈R^dt is the hth tag w h In the text embedding, dt is the dimension of the text embedding.

[0094] The description is divided into a tag sequence, namely the second tag data Description = {[CLS], u1, ..., um-1}, where m is the length of all tags. The description D∈R^(n×dt) is expressed as:

[0095] ;

[0096] Among them, e a ∈R^de is the ath label u a In the description embedding, de is the dimension of the description embedding.

[0097] Use the pre-trained Transformer-based image classification model (Vision Transformer, ViT) as the image encoder. First, split the image ∈ R^(H×W×C) into a sequence of q flattened two-dimensional blocks, where H, W, and C represent the height, width, and number of channels of the image, respectively. Then, input the image into ViT to obtain a block of image features I∈R^(q×dv), which is represented as follows:

[0098] ;

[0099] Among them, v j ∈R^dv is the image embedding of the jth patch in the image and dv is the dimension of the image embedding.

[0100] This embodiment provides that text data, image data and description data are respectively subjected to embedding layer processing to obtain respective feature data, thereby improving the accuracy of feature extraction.

[0101] Contrastive learning is a self-supervised learning method that enhances feature expression by distinguishing positive and negative sample pairs. It maximizes the similarity between similar samples while minimizing the similarity between dissimilar samples. In multimodal fake news detection, contrastive learning is used to align data features of different modalities, making text and image features closer in the same feature space and reducing the modality gap.

[0102] In some embodiments, the feature data of the text data and the image data are subjected to contrastive learning processing to obtain contrastive learning loss, including:

[0103] Acquire corresponding multiple text data and multiple image data in the same batch; wherein the multiple text data and multiple image data exist in the form of text-image pairs;

[0104] Using the first target image data corresponding to the first target text data as the first positive sample data, and using other image data other than the first target image data corresponding to the first target text data as the first negative sample data;

[0105] Using the second target text data corresponding to the second target image data as the second positive sample data, and using other text data except the second target text data corresponding to the second target image data as the second negative sample data;

[0106] Processing the first positive sample data, the first negative sample data, the second positive sample data, and the second negative sample data to obtain a first similarity matrix;

[0107] The contrastive learning loss is obtained according to the first similarity matrix.

[0108] In order to align different modalities, contrastive learning techniques are used to align text and image representations in the feature space by maximizing the similarity of positive samples and minimizing the similarity of negative samples. The goal of this module is to reduce the heterogeneity between modalities so that text and image features are closely aligned in the same feature space. For a new batch, a pair of corresponding text-image pairs (T f ∈R^(b×dt),I f ∈R^(b×dv)), they are positive samples of each other. f The text in the same batch as other images I g ∈R^(b×dt) is a negative sample. The same applies to image I f and other texts T g ∈R^(b×dt). The first target image data corresponding to the first target text data is used as the first positive sample data, and the image data other than the first target image data corresponding to the first target text data is used as the first negative sample data; the second target text data corresponding to the second target image data is used as the second positive sample data, and the text data other than the second target text data corresponding to the second target image data is used as the second negative sample data.

[0109] Next, calculate the cosine similarity between each sample. Then, use the matrix S∈R^(b×(1+2×(b-1))) containing the similarities of these samples:

[0110] ;

[0111] The first column of S is the similarity between positive samples, and the other columns are the similarity between negative samples. Then, we define a loss function as a contrast loss to maximize the value of the first column of S and minimize the values ​​of other columns:

[0112] ;

[0113] Where B is the number of samples in a batch, r represents the rth sample in the batch, and 1≤r≤B.

[0114] This embodiment provides a comparative learning method to reduce modal gaps, reduce heterogeneity between different modalities, and achieve accurate alignment of cross-modal information. The aligned features can better capture the semantic association between text and images, thereby more comprehensively integrating multimodal information in the process of false news detection and improving the accuracy of detection.

[0115] S13: Perform mutual modal fusion processing on the feature data corresponding to the text data, image data and description data to obtain corresponding fusion variables. The fusion variables include a first fusion variable, a second fusion variable and a third fusion variable; the first fusion variable is a variable obtained by fusion processing with the image data as the main factor and the text data; the second fusion variable is a variable obtained by fusion processing with the description data as the main factor and the text data; the third fusion variable is a variable obtained by fusion processing with the text data as the main factor and the image data.

[0116] Figure 5 A schematic diagram of the internal structure of a modality fusion module provided in an embodiment of the present invention is shown in FIG. Figure 5 As shown, the first fusion module 14 is responsible for fusing text and image modalities with text as the main modality, the second fusion module 15 is responsible for fusing text and image modalities with image as the main modality, and the third fusion module 16 is responsible for fusing text and description modalities with description as the main modality.

[0117] The three fusion modules have the same calculation method, so take the first fusion module 14 as an example:

[0118] When a new batch appears, the affinity matrix C is used to fuse different modes. Taking the fusion block that fuses text features T and image features I as an example, the calculation formula of C is:

[0119] ;

[0120] Among them, Wb∈Rb×b is a learnable parameter containing weights, and b represents the batch size. After calculating the affinity matrix C∈Rdv×dt, the weight vector a∈Rdt is calculated by performing a column-wise maximum pooling operation on the matrix C. Finally, the calculation formula for the third fusion variable Ht is:

[0121] ;

[0122] Among them, H t ∈Rb×dt is the image-text representation.

[0123] Similarly, the first fused variable Hv∈Rb×dv is the text-image representation. The second fused variable Hd∈Rb×dt is the text-description representation.

[0124] S14: Perform secondary fusion processing on the first fusion variable and the second fusion variable to obtain first data; perform mutual learning processing on the first data and the third fusion variable to obtain mutual learning loss.

[0125] Among them, Hv and Hd are both features obtained from the fusion of visual information as the main body, so they need to be fused again. The specific fusion method can be the same as the fusion method of the above embodiment, or it can be different.

[0126] In some embodiments, performing secondary fusion processing on the first fusion variable and the second fusion variable to obtain the first data includes:

[0127] Input the first fused variable and the second fused variable into the multi-head attention layer to obtain the first variable corresponding to each head;

[0128] Performing a linear transformation on the first variable corresponding to each head to obtain a first parameter;

[0129] Establish a residual connection between the feature data corresponding to the description data and the first parameter to obtain a second variable;

[0130] The second variable is connected using a feedforward network and a residual connection to obtain the first data.

[0131] Specifically, Hv is the first fusion variable; Hd is the second fusion variable. The visual fusion module uses a multi-head attention structure, where Hd is the query and Hv is the key and value. Specifically, for the lth head of the visual fusion module, i.e., the first variable, its form is as follows:

[0132] ;

[0133] Where, dk = dt / h, ATT l ∈R^(b×dk).

[0134] ∈R^(dk×dt) , ∈R^(dk×dv) , ∈R^(dk×dv) is a learnable parameter.

[0135] Connect the outputs of the h heads and then perform a linear transformation, i.e. the first parameter, as shown below:

[0136] ;

[0137] Among them, W 0 ∈R^(dt×dt) is a learnable parameter.

[0138] A residual connection is established on the output of the description feature Hd and the self-attention layer (Multi Attention Heads, MATT) to obtain the second variable, namely:

[0139] ;

[0140] Among them, LN is the layer normalization operation. Then, a feedforward network (also known as a multi-layer perceptron (MLP)) and another residual connection are used on Z to obtain the output of the first transformer encoder, that is, the first data:

[0141] ;

[0142] Among them, TIM=Hi; TIM∈R^(b×dt) is the output of the visual modality fusion module.

[0143] The first data obtained by the secondary fusion processing provided in the embodiment of the present invention, since the two fusion variables are obtained by fusing the image data and the description data respectively, the secondary fusion processing realizes the restoration processing to obtain the first data, thereby improving the accuracy of subsequent detection.

[0144] In some embodiments, performing mutual learning processing on the first data and the third fusion variable to obtain a mutual learning loss includes:

[0145] Perform loss processing on the first data and the third fused variable respectively to obtain their respective loss values;

[0146] The mutual learning loss is obtained by processing the loss values ​​of the first data and the third fusion variable through KL divergence.

[0147] Specifically, mutual learning is used to further reduce the gap between modalities. For Ht and Hi (the third fusion variable, i.e., the fusion variable corresponding to image data and text data) after passing through the fusion module, the logistic distribution loss function is used to collaboratively process Ht and Hi:

[0148] ;

[0149] ;

[0150] Among them, softmax is the activation function and LN is the linear layer.

[0151] The Kullback Leibler (KL) divergence is used to measure the consistency of the prediction results of the two classifiers and obtain their respective loss values:

[0152] ;

[0153] ;

[0154] Where KL(·) represents the KL divergence between two probability distributions, 1≤c≤N, and N is a positive integer.

[0155] Specifically, .

[0156] The embodiment of the present invention provides a method for improving overall performance by using mutual learning and exchanging information between multiple models or modules. In multimodal fake news detection, mutual learning promotes the synergy of text and image features, enabling them to achieve effective alignment while maintaining their respective characteristics, reducing modal heterogeneity and improving detection accuracy.

[0157] S15: Determine the loss value corresponding to the fake news data based on the contrastive learning loss and the mutual learning loss.

[0158] The classifier module that outputs the diagnosis result is constructed by MLP (multi-layer perceptron). The classifier inputs the extracted feature map into the hidden layer and obtains the final diagnosis result through the activation function and the SoftMax function of the output layer. The dimension of the first layer MLP and the hidden layer is equal to the output frequency scale dimension of the attention block in the last stage. The dimension of the time scale is reduced to one dimension through the global average pooling layer. The last layer MLP is the fault result category of the equipment in this project.

[0159] In some embodiments, determining a loss value corresponding to the fake news data based on contrastive learning loss and mutual learning loss includes:

[0160] Processing the first fused variable and the first variable by a classifier to obtain a first predicted value;

[0161] Obtain the label corresponding to the fake news data and use it as the first true value;

[0162] The first predicted value and the first true value are processed by a cross entropy function to obtain a first loss;

[0163] The contrastive learning loss, mutual learning loss and the first loss are added together to get the final loss value.

[0164] Specifically, during the training process, the first predicted value obtained by the classifier classifying the first fusion variable and the first variable is the first true value of the label corresponding to the false news data, and the classification loss loss is calculated using the cross entropy formula as the loss function. f , that is, the first loss, and then add the contrastive learning loss loss1 and the mutual learning loss loss m :

[0165] ;

[0166] in, , is the weight coefficient, , and The numbers and letters in the lower right corner only distinguish different loss values ​​and have no other meanings.

[0167] A method for processing multimodal false news data provided by an embodiment of the present invention includes: obtaining text data and image data corresponding to false news data, and performing visual description processing on the image data to obtain description data; performing embedding layer processing on the text data, image data and description data to obtain respective feature data; performing contrast learning processing on the feature data of the text data and the image data to obtain contrast learning loss; performing mutual modal fusion processing on the feature data corresponding to the text data, the image data and the description data to obtain corresponding fusion variables; wherein the fusion variables include a first fusion variable, a second fusion variable and a third fusion variable; the first fusion variable is a variable obtained by performing fusion processing with the text data with the image data as the leading factor; the second fusion variable is a variable obtained by performing fusion processing with the text data with the description data as the leading factor; the third fusion variable is a variable obtained by performing fusion processing with the image data with the text data as the leading factor; performing secondary fusion processing on the first fusion variable and the second fusion variable to obtain first data; performing mutual learning processing on the first data and the third fusion variable to obtain mutual learning loss; and determining the loss value corresponding to the false news data based on the contrast learning loss and the mutual learning loss. The present invention reduces the heterogeneity between different modalities through contrastive learning processing, realizes the precise alignment of cross-modal information, and can better capture the semantic association between text and image through the aligned features, thereby more comprehensively integrating multimodal information in the process of false news detection and improving the accuracy of detection. A mutual learning mechanism is introduced to achieve collaborative learning between modalities through information sharing while maintaining the respective modal features of text and image. This mechanism enables the model to automatically balance the importance of different modalities, avoid over-reliance on a single modality under different news content, and improve the robustness of detection. In addition, the introduction of mutual learning helps to deal with the problem of imbalance in multimodal data, so that the system can adapt to a wider range of news scenarios. Through progressive feature fusion and alignment strategies, a balance between detection accuracy and efficiency is achieved, making it suitable for high-complexity false news detection applications.

[0168] Large model-assisted multimodal feature extraction and alignment: This invention enriches image modal information through large models and converts it into text information, thereby reducing the modal gap. Alignment of text and image features is achieved through a contrastive learning mechanism, further reducing the heterogeneity between modalities. This innovation ensures efficient alignment and integration of multimodal information and improves the accuracy of fake news detection.

[0169] Progressive strategy of contrastive learning and mutual learning: This paper adopts contrastive learning for inter-modal alignment, and further introduces a mutual learning mechanism to achieve information synergy between text and image. Contrastive learning maximizes the similarity between positive samples and minimizes the similarity between negative samples; mutual learning measures the consistency of text and image classifiers through KL divergence, ensuring the collaborative optimization of multimodal information. This innovation enhances the adaptability of multimodal data in different scenarios, and is particularly suitable for the complex detection task of fake news.

[0170] In this dataset, real news is verified by Xinhua News Agency, an authoritative news agency in China. Fake news is verified by Weibo's official rumor-refuting system. Each tweet in the dataset contains text, pictures / videos, and social background information. In this work, we mainly focus on text and picture information. Tweets with videos and tweets without text or pictures are deleted. Table 1 lists the detailed statistical information of the dataset, as shown in Table 1.

[0171] Table 1

[0172] The various benchmark models are compared with some models from previous years. The experimental data are shown in the data comparison table in Table 2:

[0173] Table 2

[0174] The experimental results summarized in Table 2 show that our method achieves superior performance on various metrics compared to the baseline models. Specifically, our method achieves an accuracy of 0.917, surpassing models such as Event Adversarial Neural Networks (EANN) (0.827), Multimodal Variational Autoencoder (MVAE) (0.824), and even the more advanced Multimodal Fake News Detection network based on Contrastive Language-Image Pretraining (FND-CLIP) (0.907). This performance improvement can be attributed to our enhanced feature fusion method, which effectively combines text and visual data while leveraging the visual enhancement capabilities of the LLaVA model. The addition of contrastive learning and mutual learning further strengthens the model's ability to identify fake news, ensuring improved precision without sacrificing recall.

[0175] Use the self-attention mechanism to achieve modal alignment of text and images. Through the self-attention mechanism, the model can capture the relationship between different modal features, thereby achieving a certain degree of modal fusion. This method can reduce the reliance on large models, thereby reducing the demand for computing resources.

[0176] Problems: The self-attention mechanism has high computational complexity when processing long sequences or high-dimensional data, especially when processing image data containing a lot of contextual information. In addition, the self-attention mechanism itself is less sensitive to position information and it is difficult to fully capture the spatial and semantic relationship of multimodal data. For fake news detection tasks that require deep semantic alignment, the self-attention mechanism has limited effect in information integration and heterogeneity elimination, which may lead to low detection accuracy.

[0177] The multimodal data is converted into a time-frequency graph through wavelet transform or Fourier transform, and then the Convolutional Neural Network (CNN) is used to extract features and fuse them. The image and text data are preprocessed to generate a time-frequency graph. The CNN extracts multi-scale features from the graph through convolution and pooling layers, and then passes through the fully connected layer to detect and judge fake news. This non-end-to-end processing method can achieve multimodal feature fusion to a certain extent.

[0178] Problems: Wavelet transform and Fourier transform are preprocessing steps. Such non-end-to-end network models may cause feature loss during the processing process and have high requirements for hardware resources. They usually require a graphics processing unit (GPU) for training. At the same time, CNN may cause position information loss due to convolution and pooling operations during feature extraction, affecting the accuracy of detection. In addition, since this method fails to effectively solve the heterogeneity between modalities, it may be limited in fusing deep semantic information, resulting in poor performance in complex fake news detection scenarios.

[0179] The above describes in detail various embodiments corresponding to the method for processing multimodal false news data. On this basis, the present invention also discloses a device for processing multimodal false news data corresponding to the above method. Figure 6 A structural diagram of a multimodal false news data processing device provided by an embodiment of the present invention. Figure 6 As shown, the processing device for multimodal false news data includes:

[0180] An acquisition module 18 is used to acquire text data and image data corresponding to the fake news data, and perform visual description processing on the image data to obtain description data;

[0181] The first processing module 19 is used to perform embedding layer processing on the text data, the image data and the description data to obtain their respective feature data; perform contrastive learning processing on the feature data of the text data and the image data to obtain contrastive learning loss;

[0182] The second processing module 20 is used to perform mutual modal fusion processing on the feature data corresponding to the text data, the image data and the description data to obtain corresponding fusion variables; wherein the fusion variables include a first fusion variable, a second fusion variable and a third fusion variable; the first fusion variable is a variable obtained by fusion processing with the image data with the text data as the main body; the second fusion variable is a variable obtained by fusion processing with the description data with the text data as the main body; the third fusion variable is a variable obtained by fusion processing with the text data with the image data as the main body;

[0183] The third processing module 21 is used to perform secondary fusion processing on the first fusion variable and the second fusion variable to obtain first data; perform mutual learning processing on the first data and the third fusion variable to obtain mutual learning loss;

[0184] The determination module 22 is used to determine the loss value corresponding to the fake news data based on the contrastive learning loss and the mutual learning loss.

[0185] Since the embodiments of the device part correspond to the above embodiments, please refer to the description of the embodiments of the method part for the embodiments of the device part, and will not be repeated here.

[0186] For an introduction to a multimodal false news data processing device provided by the present invention, please refer to the above method embodiment, and the present invention will not be repeated here. It has the same beneficial effects as the above multimodal false news data processing method.

[0187] Figure 7 A structural diagram of a multimodal false news data processing device provided by an embodiment of the present invention, such as Figure 7 As shown, the device includes:

[0188] A memory 23, used for storing computer programs;

[0189] The processor 24 is used to implement the steps of the method for processing multimodal fake news data when executing a computer program.

[0190] The multimodal fake news data processing device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0191] Among them, the processor 24 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 24 can be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 24 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 24 may be integrated with a GPU, which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 24 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0192] The memory 23 may include one or more computer-readable storage media, which may be non-transitory. The memory 23 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 23 is at least used to store the following computer program 231, wherein, after the computer program is loaded and executed by the processor 24, it can implement the relevant steps of the method for processing multimodal false news data disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 23 may also include an operating system 232 and data 233, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 232 may include Windows, Unix, Linux, etc. The data 233 may include, but is not limited to, data involved in the method for processing multimodal false news data, etc.

[0193] In some embodiments, the multimodal fake news data processing device may also include a display screen 25 , an input and output interface 26 , a communication interface 27 , a power supply 28 , and a communication bus 29 .

[0194] Those skilled in the art can understand that Figure 7 The structure shown in the figure does not constitute a limitation on the device for processing multimodal fake news data, and may include more or fewer components than those shown in the figure.

[0195] The processor 24 implements the method for processing multimodal fake news data provided by any of the above embodiments by calling the instructions stored in the memory 23.

[0196] For an introduction to a multimodal false news data processing device provided by the present invention, please refer to the above method embodiment, and the present invention will not be repeated here. It has the same beneficial effects as the above multimodal false news data processing method.

[0197] Furthermore, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by the processor 24, the steps of the method for processing multimodal false news data as described above are implemented.

[0198] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0199] For an introduction to a computer-readable storage medium provided by the present invention, please refer to the above method embodiment, and the present invention will not be repeated here. It has the same beneficial effects as the above-mentioned method for processing multimodal false news data.

[0200] The above is a detailed introduction to a method, device, equipment and medium for processing multimodal fake news data provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

[0201] It should also be noted that, in this specification, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A method for processing multimodal fake news data, characterized in that: include: Acquire text data and image data corresponding to the fake news data, and perform visual description processing on the image data to obtain description data; The text data, the image data and the description data are respectively subjected to embedding layer processing to obtain respective feature data; the feature data of the text data and the image data are respectively subjected to contrastive learning processing to obtain contrastive learning loss; The feature data corresponding to the text data, the image data and the description data are subjected to mutual modal fusion processing to obtain corresponding fusion variables; wherein the fusion variables include a first fusion variable, a second fusion variable and a third fusion variable; the first fusion variable is a variable obtained by fusion processing with the text data with the image data as the leading factor; the second fusion variable is a variable obtained by fusion processing with the text data with the description data as the leading factor; the third fusion variable is a variable obtained by fusion processing with the image data with the text data as the leading factor; Perform secondary fusion processing on the first fusion variable and the second fusion variable to obtain first data; perform mutual learning processing on the first data and the third fusion variable to obtain mutual learning loss; A loss value corresponding to the fake news data is determined based on the contrastive learning loss and the mutual learning loss.

2. The method for processing multimodal false news data according to claim 1, characterized in that: The text data, the image data and the description data are respectively subjected to embedding layer processing to obtain respective feature data, including: Segmenting the text data to obtain first labeled data; Inputting the first labeled data into a pre-trained language representation model to obtain text feature data corresponding to the text data; Segmenting the description data to obtain second labeled data; Inputting the second labeled data into a pre-trained language representation model to obtain description feature data corresponding to the description data; Segmenting the image data to obtain block sequence data; The block sequence data is input into a value image encoder to obtain image feature data corresponding to the image data.

3. The method for processing multimodal false news data according to claim 1, characterized in that: The step of performing contrastive learning processing on the feature data of the text data and the image data to obtain contrastive learning loss includes: Acquire corresponding multiple text data and multiple image data in the same batch; wherein the multiple text data and multiple image data exist in the form of text-image pairs; Using first target image data corresponding to first target text data as first positive sample data, and using other image data other than the first target image data corresponding to the first target text data as first negative sample data; Using the second target text data corresponding to the second target image data as the second positive sample data, and using other text data corresponding to the second target image data except the second target text data as the second negative sample data; Processing the first positive sample data, the first negative sample data, the second positive sample data, and the second negative sample data to obtain a first similarity matrix; The contrastive learning loss is obtained according to the first similarity matrix.

4. The method for processing multimodal false news data according to claim 1, characterized in that: Performing secondary fusion processing on the first fusion variable and the second fusion variable to obtain first data includes: Input the first fused variable and the second fused variable into the multi-head attention layer to obtain the first variable corresponding to each head; Performing a linear transformation on the first variable corresponding to each head to obtain a first parameter; Establishing a residual connection between the feature data corresponding to the description data and the first parameter to obtain a second variable; The second variable is connected using a feedforward network and a residual connection to obtain the first data.

5. The method for processing multimodal false news data according to claim 4, characterized in that: The first data and the third fusion variable are subjected to mutual learning processing to obtain a mutual learning loss, including: Perform loss processing on the first data and the third fused variable respectively to obtain respective loss values; The mutual learning loss is obtained by processing the loss values ​​of the first data and the third fusion variable through KL divergence.

6. The method for processing multimodal false news data according to claim 4, characterized in that: Determining a loss value corresponding to the fake news data based on the contrastive learning loss and the mutual learning loss includes: Processing the first fused variable and the first variable by a classifier to obtain a first predicted value; Obtaining a label corresponding to the fake news data and using it as a first true value; Processing the first predicted value and the first true value by a cross entropy function to obtain a first loss; The contrastive learning loss, the mutual learning loss and the first loss are weightedly summed to obtain a final loss value.

7. The method for processing multimodal false news data according to claim 1, characterized in that: The image data is subjected to visual description processing to obtain description data, including: Get a pre-trained model for representing visual descriptions; The image data is input into a pre-trained model for characterizing visual descriptions to obtain the description data.

8. A device for processing multimodal false news data, characterized in that: include: An acquisition module, used to acquire text data and image data corresponding to the fake news data, and perform visual description processing on the image data to obtain description data; A first processing module is used to perform embedding layer processing on the text data, the image data and the description data to obtain respective feature data; and perform contrastive learning processing on the feature data of the text data and the image data to obtain contrastive learning loss; The second processing module is used to perform mutual modal fusion processing on the feature data corresponding to each of the text data, the image data and the description data to obtain corresponding fusion variables; wherein the fusion variables include a first fusion variable, a second fusion variable and a third fusion variable; the first fusion variable is a variable obtained by performing fusion processing with the text data with the image data as the leading factor; the second fusion variable is a variable obtained by performing fusion processing with the text data with the description data as the leading factor; the third fusion variable is a variable obtained by performing fusion processing with the image data with the text data as the leading factor; A third processing module is used to perform secondary fusion processing on the first fusion variable and the second fusion variable to obtain first data; and perform mutual learning processing on the first data and the third fusion variable to obtain mutual learning loss; A determination module is used to determine the loss value corresponding to the fake news data based on the contrastive learning loss and the mutual learning loss.

9. A device for processing multimodal fake news data, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method for processing multimodal false news data as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for processing multimodal false news data as described in any one of claims 1 to 7.

Images