Multi-modal false news detection method, device and equipment

Abstract

The invention provides a multi-mode false news detection method, device and equipment, and the method comprises the steps: carrying out the feature extraction and semantic alignment of first multi-mode news data, and obtaining a first single-mode feature vector and a second single-mode feature vector; processing the single-mode feature vector based on a long-acting short-term memory neural network to obtain a hidden state context vector; processing the hidden state context vector based on a low-rank tensor algorithm to obtain a cross-modal fusion feature vector; respectively capturing feature distribution of the single-mode feature vectors based on a variational encoder; calculating and determining KL divergence based on the obtained feature distribution corresponding to different single-mode features; the weights of the first single-mode feature vector, the second single-mode feature vector and the cross-mode fusion feature vector are adaptively adjusted based on the adaptation degree, and target input data are generated and formed; and inputting the target input data into the detection model to obtain a detection result. According to the method disclosed by the invention, the multi-modal false news can be quickly and accurately detected.

Description

Technical Field

[0001] The present invention belongs to the technical field of fake news detection, and specifically relates to a multimodal fake news detection method, device and equipment. Background Art

[0002] To date, most studies on fake news detection have focused on single-modality pure text, ranging from manual detection methods, machine learning methods to deep learning methods, which has led to an excessive focus on extracting semantic features from text to detect fake news. Although traditional text-based single-modality methods perform well in extracting semantic features, this approach has shown limitations with the development of multimedia technology.

[0003] The concept of cross-modal fusion is a promising approach to address these problems by integrating information features from multiple modalities to improve detection performance. For example, juxtaposing text content with the attached pictures can reveal different signals of potential fake news, which emphasizes the importance of modal collaboration in improving detection accuracy. However, indiscriminate cross-modal fusion may introduce noise and affect detection accuracy. Therefore, it is necessary to take a meticulous approach to cross-modal fusion. Existing methods often ignore the intrinsic relationship between unimodal features and multimodal features, resulting in increased algorithm complexity and unsatisfactory detection results. For example, fake news can be indicated based on text or pictures alone because they violate common sense. However, this point is often ignored in multimodal detection, resulting in cumbersome detection and inaccurate results. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method, device and equipment that can perform multimodal fake news detection more efficiently and accurately.

[0005] The present invention provides a multimodal fake news detection method, comprising:

[0006] Extracting features from multimodal news data and semantically aligning the extracted features to obtain a unimodal feature vector, wherein the multimodal news data includes text data and image data;

[0007] Processing the unimodal feature vector based on a long-term short-term memory neural network to obtain a hidden state context vector;

[0008] Processing the hidden state context vector based on a low-rank tensor algorithm to obtain a cross-modal fusion feature vector;

[0009] Capturing the feature distribution of the unimodal feature vector based on a variational encoder;

[0010] Determining a KL divergence based on the feature distribution calculation, wherein the KL divergence is used to quantify the degree of fitness between different unimodal feature vectors, wherein the degree of fitness between the different unimodal feature vectors is the degree of fitness between a first unimodal feature vector corresponding to the text data and a second unimodal feature vector corresponding to the image data;

[0011] Adaptively adjusting the weights of the single-modal feature vector and the cross-modal fusion feature vector based on the degree of fitness, and forming target input data based on the adjusted weights and the single-modal feature vector and the cross-modal fusion feature vector;

[0012] The target input data is input into the detection model to obtain a detection result corresponding to whether the multimodal news data is true news or false news.

[0013] In some embodiments, extracting features from multimodal news data and semantically aligning the extracted features includes:

[0014] Extracting text data and image data from the multimodal news data;

[0015] Performing feature extraction on the text data based on a text encoder to obtain a text feature vector;

[0016] Extracting features from the image data based on an image encoder to obtain an image feature vector;

[0017] The text feature vector and the image feature vector are mapped into a shared semantic space to obtain a plurality of different unimodal feature vectors.

[0018] In some embodiments, the method further comprises:

[0019] Obtain multiple text features and multiple image features of multimodal news data for training;

[0020] Predict the positive and negative correlations between different text features and image features;

[0021] Construct a training set based on positively correlated text features and image features, and negatively correlated text features and image features;

[0022] Construct a contrastive learning framework based on a multi-task learning model, a modality-specific multi-layer task perceptron, and a modality sharing layer;

[0023] Using the training set to train the contrastive learning framework, so that the trained contrastive learning framework can perform feature extraction and semantic alignment of multimodal news data to obtain a unimodal feature vector;

[0024] The feature loss amount of the contrastive learning framework when performing semantic alignment is calculated based on the unimodal feature vector, cosine loss, and margin value;

[0025] The feature loss amount is positively fed back to the contrastive learning framework to optimize the contrastive learning framework.

[0026] In some embodiments, the step of processing the hidden state context vector based on a low-rank tensor algorithm to obtain a cross-modal fusion feature vector includes:

[0027] Increasing each of the hidden state context vectors by a one-dimensional value;

[0028] Perform outer product calculation on the hidden state context vector with one-dimensional value added to obtain the cross-modal fusion hidden context vector;

[0029] The cross-modal fusion hidden context vector is calculated based on batch matrix multiplication of low-rank specific modality factors and additional modality representation to obtain the cross-modal fusion feature vector.

[0030] In some embodiments, the method of capturing the feature distribution of the unimodal feature vector based on a variational encoder includes:

[0031] Sampling a unimodal feature vector using an isotropic Gaussian prior algorithm in a specified space to obtain a variational posterior probability distribution corresponding to the unimodal feature vector;

[0032] The feature distribution of the unimodal feature vector is calculated based on the variational posterior probability distribution.

[0033] In some embodiments, different unimodal feature vectors correspond to different feature distributions, and the calculating and determining the KL divergence based on the feature distributions includes:

[0034] A first KL divergence is calculated and determined based on the feature distribution of the first unimodal feature vector; a second KL divergence is calculated and determined based on the feature distribution of the second unimodal feature vector, wherein the first KL divergence and the second KL divergence have different operation positions of parameters involved in the calculation formula.

[0035] In some embodiments, the method further comprises:

[0036] The divergence average is calculated based on the first KL divergence and the second KL divergence, and the divergence average is mapped into the range of (0, 1) by using an activation function to obtain the degree of adaptation.

[0037] In some embodiments, the forming of target input data based on the adjusted weights and the single-modal feature vector and the cross-modal fusion feature vector includes:

[0038] Multiplying the unimodal feature vector and the cross-modal fusion feature vector by corresponding weights respectively to obtain a unimodal representation and a multimodal representation;

[0039] The single-modal representation and the multi-modal representation are cascaded to obtain the target input data.

[0040] The present invention also discloses a multimodal fake news detection device, comprising:

[0041] A feature extraction module, used to extract features from multimodal news data and semantically align the extracted features to obtain a unimodal feature vector, wherein the multimodal news data includes text data and image data;

[0042] A first processing module, configured to process the unimodal feature vector according to a long-term short-term memory neural network to obtain a hidden state context vector;

[0043] A second processing module is used to process the hidden state context vector according to a low-rank tensor algorithm to obtain a cross-modal fusion feature vector;

[0044] A capture module, used for capturing the feature distribution of the unimodal feature vector using a variational encoder;

[0045] A first calculation module, used for calculating and determining a KL divergence according to the feature distribution, wherein the KL divergence is used to quantify the degree of fitness between different unimodal feature vectors, wherein the degree of fitness between the different unimodal feature vectors is the degree of fitness between a first unimodal feature vector corresponding to the text data and a second unimodal feature vector corresponding to the image data;

[0046] An adjustment module, configured to adaptively adjust the weights of the unimodal feature vector and the cross-modal fusion feature vector according to the degree of fitness, and form target input data based on the adjusted weights and the unimodal feature vector and the cross-modal fusion feature vector;

[0047] The detection module is used to input the target input data into the detection model to obtain a detection result corresponding to whether the multimodal news data is true news or false news.

[0048] Another embodiment of the present invention further provides an electronic device, comprising:

[0049] at least one processor; and,

[0050] a memory communicatively connected to the at least one processor; wherein,

[0051] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to implement the multimodal fake news detection method as described in any one of the embodiments above.

[0052] The beneficial effect of the present invention is that by introducing modal fitness and utilizing the consistency relationship between the distribution of unimodal features, namely, KL divergence, the modal fitness is quantified, and based on this, the weights of unimodal features and multimodal features are adaptively adjusted, so that when the adjusted weights are combined with the corresponding modal features and then analyzed and detected by the target detector, the classification detection results of multimodal fake news can be effectively improved, the number of parameters involved in the detection is reduced, the computational load is reduced, and cross-modal feature fusion can be effectively realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Flowchart of a multimodal fake news detection method according to an embodiment of the present invention.

[0054] Figure 2 This is a flowchart of a multimodal fake news detection method according to another embodiment of the present invention.

[0055] Figure 3 4 is a structural block diagram of a multimodal fake news detection device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings, but are not intended to limit the present invention.

[0057] It should be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the following description should not be considered as limiting, but merely as an example of an embodiment. Other modifications within the scope and spirit of the present disclosure will occur to those skilled in the art.

[0058] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the general description of the present disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the present disclosure.

[0059] These and other characteristics of the invention will become apparent from the following description of a preferred form of embodiment given as a non-limiting example, with reference to the accompanying drawings.

[0060] It should also be understood that, although the invention has been described with reference to some specific examples, those skilled in the art will be able to realize many other equivalent forms of the invention that have the characteristics recited in the claims and are therefore within the scope of protection defined thereby.

[0061] The above and other aspects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0062] Specific embodiments of the present disclosure are described hereinafter with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid obscuring the present disclosure with unnecessary or redundant details. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present disclosure in a variety of ways with substantially any suitable detailed structure.

[0063] This specification may use the phrases "in one embodiment," "in another embodiment," "in a further embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present disclosure.

[0064] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0065] like Figure 1 As shown, the present invention includes a multimodal fake news detection method, comprising:

[0066] S1: extracting features from multimodal news data and semantically aligning the extracted features to obtain a unimodal feature vector, wherein the multimodal news data includes text data and image data;

[0067] S2: Processing the unimodal feature vector based on a long-term short-term memory neural network to obtain a hidden state context vector;

[0068] S3: Processing the hidden state context vector based on a low-rank tensor algorithm to obtain a cross-modal fusion feature vector;

[0069] S4: Capturing the feature distribution of the unimodal feature vector based on a variational encoder;

[0070] S5: determining a KL divergence based on the feature distribution calculation, wherein the KL divergence is used to quantify the degree of fitness between different unimodal feature vectors, wherein the degree of fitness between different unimodal feature vectors is the degree of fitness between a first unimodal feature vector corresponding to the text data and a second unimodal feature vector corresponding to the image data;

[0071] S6: adaptively adjusting the weights of the single-modal feature vector and the cross-modal fusion feature vector based on the degree of fitness, and forming target input data based on the adjusted weights and the single-modal feature vector and the cross-modal fusion feature vector;

[0072] S7: Input the target input data into the detection model to obtain a detection result corresponding to whether the multimodal news data is true news or false news.

[0073] For example, multimodal news data is a set D, which consists of multimodal information, such as text data and image data, and can be represented as D = {X, Y}, where X is the set of modalities and Y is the set of corresponding labels. For example, a multimodal news data sample A can be represented as {x, y} = {{x i} n ,y}∈D, where x i is the i-th modality of A. In multimodal problems, the modality can be text, image, etc., and y is the label of A. Furthermore, for a given news sample {x, y}∈D, the modality fit between the i-th modality and the j-th modality is ε ij (i.e., the degree of fitness described in the above embodiment) is defined as the distance between the two modalities, i.e., dis(i, j). This embodiment mainly involves the modal fitness between the text modality and the image modality, and converts the distance between the two modalities into a unimodal feature with KL divergence, which is quantitatively expressed as dis(text, image). The modal fitness described in this embodiment (i.e., the degree of fitness described in the above embodiment) is a key factor in the proposed method that can adaptively adjust the relationship between the unimodal feature and the cross-modal fusion feature. If the degree of fitness between the text modality and the image modality is low, it means that the information difference between them is large, and it is necessary to make a comprehensive judgment through modal information fusion, that is, it is necessary to focus on the cross-modal fusion feature vector. On the contrary, if the modal fitness between the text modality and the image modality is very high, it is possible to achieve accurate classification only through the information of a single modality. Therefore, the proposal and quantification of modal fitness can better measure the information difference between the two modalities and make up for the deficiency of unimodal information.

[0074] In this embodiment, the proposed method includes three key parts: 1) information extraction and semantic alignment, including a unimodal feature information extractor and semantic alignment of cross-modal information. 2) Cross-modal fusion, which realizes cross-modal feature fusion by compressing and converting the semantically aligned feature vector into a low-rank tensor. 3) Modality adaptive adjustment, which re-implements the weight distribution of unimodal features and cross-modal fusion features according to the modality fit between the text modality and the image modality, thereby improving the accuracy of subsequent classification.

[0075] Based on the above content, it can be known that the beneficial effects of this embodiment include introducing modal fitness and utilizing the consistency relationship between the distribution of unimodal features, namely KL divergence, to quantify the modal fitness, and based on this, adaptively adjust the weights of unimodal features and multimodal features, which is equivalent to decomposing the weights into low-level factors. This not only allows the adjusted weights to be combined with the corresponding modal features and then analyzed and detected by the target detector, which can effectively improve the classification detection results of multimodal fake news, but also reduces the number of parameters involved in the detection, reduces the computational load, and can effectively realize cross-modal feature fusion. The fake news described in this embodiment, that is, false news, refers to a kind of public opinion that uses false information to deceive the parties or the masses in order to achieve a certain purpose, and fails to truly reflect the true appearance of objective things, and contains false reports. Its manifestations include but are not limited to untrue content, mismatched pictures and texts, etc.

[0076] In some embodiments, extracting features from the first multimodal news data and semantically aligning the extracted features includes:

[0077] S8: extracting text data and image data from the first multimodal news data;

[0078] S9: extracting features from the text data based on a text encoder to obtain a text feature vector;

[0079] S10: extracting features from the image data based on an image encoder to obtain an image feature vector;

[0080] S11: Mapping the text feature vector and the image feature vector into a shared semantic space to obtain a plurality of different unimodal feature vectors.

[0081] In this embodiment, the feature data is converted into vector representation, and the text and image information in each news item is converted into corresponding feature vectors through a specific modal encoder to improve the machine readability and operability of the data. Specifically, after obtaining the text data and image data, the text data (denoted as x T ) is input to the text encoder, for example, a pre-trained BERT model, and then the corresponding text embedding vector e is obtained through a fully connected layer. T ∈R 256 At the same time, the image data (represented as x I ) is input to the image encoder, for example, ResNet-34 can be applied to extract the meaningful parts of the image and describe its features. The result of the featurization will pass through a fully connected layer to obtain the image embedding vector e I ∈R 512 Since the feature embedding vector e of the obtained text and imageT and e I There may be a large semantic gap. Therefore, it is necessary to map the respective unimodal embedding vectors into a shared semantic space to facilitate subsequent cross-modal feature fusion. However, due to the different dimensions of the feature vectors before and after the mapping, there will be a certain feature loss during the mapping process. To solve this problem and optimize the mapping model, this embodiment introduces an auxiliary correlation learning task to better achieve cross-modal alignment. Specifically, a simple encoder is used to construct different visual and language semantic levels to form a cross-modal contrast learning framework, and the methods involved include:

[0082] S12: obtaining a plurality of text features and a plurality of image features of multimodal news data for training;

[0083] S13: Predict the positive and negative correlations between different text features and image features;

[0084] S14: Construct training sets based on positively correlated text features and image features, and negatively correlated text features and image features;

[0085] S15: Construct a contrastive learning framework based on a multi-task learning model, a modality-specific multi-layer task perceptron, and a modality sharing layer;

[0086] S16: using the training set to train the contrastive learning framework, so that the trained contrastive learning framework can perform feature extraction and semantic alignment of the first multimodal news data, and obtain a first unimodal feature vector and a second unimodal feature vector respectively;

[0087] S17: Calculate the feature loss amount of the contrastive learning framework when performing semantic alignment based on the first unimodal feature vector, the second unimodal feature vector, the cosine loss, and the margin value;

[0088] S18: Feedback the feature loss amount to the litigation comparative learning framework to optimize the comparative learning framework.

[0089] For example, the auxiliary correlation learning task is a binary classification task, that is, given the text and image features of a specific news report, predict whether the text and image pair are positively correlated (True) or negatively correlated (False). First, we can Build a new dataset based on The original data set may be the aforementioned multimodal news data, that is, the training multimodal news data may be the same as or different from the aforementioned multimodal news data. represents a positive set, represents the negative set. If the text and image embeddings are from the same real news, then they belong to and are marked as y1=1; otherwise, they are classified as And marked as y1=0. Further, the contrastive learning framework in this embodiment. The framework consists of MMoE, a modality-specific multi-layer task perceptron (MLP) and a modality sharing layer. First, MMoE can be used to extract meaningful text and image feature parts. The obtained meaningful features are then input into two parallel multi-layer perceptrons (MLPs) to obtain different modal features. Next, the different modal features are semantically aligned, and the resulting shared semantic embedding is input into the average pooling layer and processed by the binary classifier constructed by the fully connected layer. When calculating the feature loss amount, this embodiment trains the positive and negative pairing sets by introducing cosine loss and margin value, and then obtains the loss value, the mathematical representation of which is as follows:

[0090]

[0091] Among them, cos() is the normalized cosine similarity, m is the margin value, and based on the experience of previous studies, m is set to 0.2. The purpose of the above scheme is to make the positive text image pairs close to each other and the negative text image pairs far away from each other. The semantic regularization involved can automatically embed and map the different modal data into a shared semantic space by back-propagating the gradient. Finally, the semantically aligned unimodal representation of text and image can be used as the input of the cross-modal fusion module to achieve cross-modal feature fusion, and also as the input of the modal adaptation module to calculate the modal fitness (i.e., fitness).

[0092] In order to promote more effective fusion of specific modalities, this embodiment uses LSTM (Long Short-Term Memory Neural Network) to achieve feature fusion. Inspired by sentiment analysis technology, this embodiment uses low-rank tensors to achieve multimodal feature fusion, while abandoning the previous solution of triple Cartesian products embedded from specific modalities. Specifically, in order to avoid long dependency problems, this embodiment first uses LSTM to compress the time series information of each modality, and extracts the hidden state context vector for the fusion of specific modalities. When processing single-modal feature data based on long-term short-term memory neural networks, the sigmoid layer (activation function layer) of the forget gate first decides which information to discard from the input features. It is based on h t-1 (upper link output) and e t (Current Input), C t-1 Output a number between 0 and 1 for the cell state (previous link output), 1 means completely retaining the information, 0 means completely deleting the information. Then, use the sigmoid function and tanh function to calculate i respectively. t and f t 、i t , They represent the symbolic representation of the input calculation of the forget gate, the symbolic representation of the input calculation of the input gate, and the current input unit state. Next, multiply the previous state value by f t to indicate the portion that is expected to be forgotten, and then add Get C t Finally, the sigmoid layer is used to determine the final input vector and normalize it between -1 and 1 to obtain the output h t , which is equivalent to the hidden context vector corresponding to the single modality obtained at the end The formulas involved are:

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099] Among them, W f , W i , W C and W o is the learnable weight matrix, b f 、b i 、b C and b o is the bias vector. Based on the same algorithm as above, we can also calculate Related h t ,as well as After obtaining the hidden layer context vectors of different modalities, before performing the outer product between the two, the unimodal tensor sequence (i.e., the hidden context vector) is first added by 1, which is equivalent to the tensor representation, and can capture the interaction between unimodality and multimodality. Specifically, the low-rank tensor algorithm is used to process the hidden state context vector to obtain the cross-modal fusion feature vector corresponding to the first unimodal feature vector and the second unimodal feature vector, including:

[0100] S19: increasing each of the hidden state context vectors by a one-dimensional value;

[0101] S20: performing outer product calculation on the hidden state context vector with the one-dimensional value added to obtain a cross-modal fusion hidden context vector;

[0102] S21: The cross-modal fusion hidden context vector is calculated based on batch matrix multiplication of low-rank specific modality factors and additional modality representations to obtain the cross-modal fusion feature vector.

[0103] For example, after adding 1 to the hidden context vector and performing the outer product, the expression is:

[0104]

[0105] Next, the compressed representation (cross-modal fusion feature vector) h is calculated using batch matrix multiplication of low-rank modality-specific factors and additive modality representations: in, is the low-order factor of mode m, Represents the element-wise product of a series of tensors, where the value of M is 2.

[0106] As mentioned above, according to the definition of modal fitness, this embodiment provides a modal fitness learning method by evaluating the KL divergence between the unimodal distributions in the two specific modal variational autoencoders. Then, the obtained fitness score is used to adaptively balance the weight scores of the unimodal features and the cross-modal fusion features. When the fitness is large, it indicates that the two unimodal distributions are close to each other, which is conducive to the use of unimodal distributions in fake news classification. On the contrary, cross-modal fusion features are more commonly used to complete classification tasks.

[0107] Since the features obtained from different unimodal samples are different, a variational encoder is used in this embodiment to capture the distribution of unimodal features. Specifically, Figure 2 As shown, the variational encoder is based on capturing the feature distribution of the first unimodal feature vector and the second unimodal feature vector respectively, including:

[0108] S22: sampling the unimodal feature vector in the specified space using an isotropic Gaussian prior algorithm to obtain a variational posterior probability distribution corresponding to the unimodal feature vector;

[0109] S23: Obtaining a feature distribution of the unimodal feature vector based on the variational posterior probability distribution.

[0110] For example, the unimodal feature vector includes a first unimodal feature vector and a second unimodal feature vector, which correspond to text features and image features respectively. The isotropic Gaussian prior is used to predict the single-mode features (including e T and e I ) is sampled. The variational posterior probability of a single-modal observation can be expressed as follows:

[0111] where the mean μ and variance σ can be obtained from the modality-specific encoder. More generally, given each data sample x s The text feature vector e with alignment T and the image feature vector e I , the variational posterior for the two modes is defined as follows:

[0112] and They represent the latent variables in the variational distribution of text and image respectively, which is a form of expression of variational distribution and can also be simply understood as a distribution.

[0113] Based on all news data samples, the final distribution is as follows:

[0114]

[0115]

[0116] Next, the feature distribution can be used to calculate the KL divergence. Specifically, the KL divergence is calculated and determined based on the obtained feature distribution corresponding to different unimodal features, including:

[0117] S24: Based on the feature distribution of the first unimodal feature vector, a first KL divergence is calculated and determined; based on the feature distribution of the second unimodal feature vector, a second KL divergence is calculated and determined, and the first KL divergence and the second KL divergence have different operation positions of parameters involved in the calculation formula.

[0118] The consistency of the two mode distributions in this embodiment can be quantified by the following two KL divergences:

[0119]

[0120]

[0121] Where D KL represents the KL divergence.

[0122] The method further comprises:

[0123] S25: Calculate the divergence average based on the first KL divergence and the second KL divergence, and use an activation function to map the fitness degree within the range of (0, 1) to obtain the fitness degree.

[0124] The fitness score (i.e. fitness) ε s Based on the normalized value and The average of the symmetric KL divergence, that is, the average of the first KL divergence and the second KL divergence after normalization. Sigmoid is an activation function that aims to map the fitness score between 0 and 1. The formula involved is:

[0125]

[0126] The forming target input data based on the adjusted weights and the first unimodal feature vector, the second unimodal feature vector and the cross-modal fusion feature vector includes:

[0127] S26: multiplying the unimodal feature vector and the cross-modal fusion feature vector by corresponding weights respectively to obtain a unimodal representation and a multimodal representation;

[0128] S27: Formal cascading operation is performed on the unimodal representation and the multimodal table to obtain the target input data.

[0129] For example, still taking the example that the unimodal feature vector includes the first unimodal feature vector and the second unimodal feature vector corresponding to the text feature and the image feature respectively, since the sum of each unimodal weight and the multimodal weight is 1, that is, ε t +ε h +ε i =1, so after knowing the single modal weight, the multimodal weight can be directly determined. After knowing the weights of each modal feature, the target input data x can be constructed based on the following formula:

[0130]

[0131] in, Represents a cascade operation. After obtaining the target input, it can be put into a fully connected network for label prediction, that is, fake news detection:

[0132] Considering that fake news detection is a binary classification task, the cross entropy function can also be introduced as the loss function for classification:

[0133]

[0134] Where y represents the true label. By setting the loss function, the influence of semantic alignment on the classification task can be controlled. In this embodiment, the final loss function can be expressed as: Where γ is a weight, its purpose is to limit the impact of semantic alignment on the classification task, and its value range is (0, 1).

[0135] Based on the above embodiments, it can be seen that the method of this embodiment is different from the previous methods. This embodiment performs information enhancement on text and images before semantic alignment to extract hidden information and complete denoising. In addition, the best performance is achieved through the improved LMF while minimizing the number of parameters. Finally, an adaptive mechanism is designed to optimize the model by dynamically adjusting the weights of multimodal and unimodal features according to the current modality fitness. In addition, after the method in this embodiment is verified on two widely used data sets (Twitter and Weibo), the data obtained is superior to the existing methods in terms of accuracy and parameter size.

[0136] like Figure 3 As shown, another embodiment of the present invention also provides a multimodal fake news detection device, including:

[0137] A feature extraction module, used for extracting features from multimodal news data and semantically aligning the extracted features to obtain a unimodal feature vector, wherein the multimodal news data includes text data and image data;

[0138] A first processing module, configured to obtain a hidden state context vector by processing the unimodal feature vector based on a long-term short-term memory neural network;

[0139] A second processing module processes the hidden state context vector based on a low-rank tensor algorithm to obtain a cross-modal fusion feature vector;

[0140] A capture module, configured to capture the feature distribution of the unimodal feature vector based on a variational encoder;

[0141] A first calculation module, used for determining a KL divergence based on the feature distribution calculation, wherein the KL divergence is used to quantify the degree of fitness between different unimodal feature vectors, where the degree of fitness between the different unimodal feature vectors is the degree of fitness between a first unimodal feature vector corresponding to the text data and a second unimodal feature vector corresponding to the image data;

[0142] An adjustment module, configured to adaptively adjust the weights of the unimodal feature vector and the cross-modal fusion feature vector based on the degree of fitness, and form target input data based on the adjusted weights and the unimodal feature vector and the cross-modal fusion feature vector;

[0143] The detection module is used to input the target input data into the detection model to obtain a detection result corresponding to whether the multimodal news data is true news or false news.

[0144] In some embodiments, extracting features from the first multimodal news data and semantically aligning the extracted features includes:

[0145] Extracting text data and image data from the multimodal news data;

[0146] Performing feature extraction on the text data based on a text encoder to obtain a text feature vector;

[0147] Extracting features from the image data based on an image encoder to obtain an image feature vector;

[0148] The text feature vector and the image feature vector are mapped into a shared semantic space to obtain a plurality of different unimodal feature vectors.

[0149] In some embodiments, the apparatus further comprises:

[0150] Obtain multiple text features and multiple image features of multimodal news data for training;

[0151] Predict the positive and negative correlations between different text features and image features;

[0152] Construct a training set based on positively correlated text features and image features, and negatively correlated text features and image features;

[0153] Construct a contrastive learning framework based on a multi-task learning model, a modality-specific multi-layer task perceptron, and a modality sharing layer;

[0154] Using the training set to train the contrastive learning framework, so that the trained contrastive learning framework can perform feature extraction and semantic alignment of multimodal news data to obtain a unimodal feature vector;

[0155] The feature loss amount of the contrastive learning framework when performing semantic alignment is calculated based on the unimodal feature vector, cosine loss, and margin value;

[0156] The feature loss amount is positively fed back to the contrastive learning framework to optimize the contrastive learning framework.

[0157] In some embodiments, the processing of the hidden state context vector based on a low-rank tensor algorithm to obtain a cross-modal fusion feature vector corresponding to the first unimodal feature vector and the second unimodal feature vector includes:

[0158] Increasing each of the hidden state context vectors by a one-dimensional value;

[0159] Perform outer product calculation on the hidden state context vector with one-dimensional value added to obtain the cross-modal fusion hidden context vector;

[0160] The cross-modal fusion hidden context vector is calculated based on batch matrix multiplication of low-rank specific modality factors and additional modality representation to obtain the cross-modal fusion feature vector.

[0161] In some embodiments, the capturing of the feature distribution of the first unimodal feature vector and the second unimodal feature vector based on a variational encoder respectively includes:

[0162] Sampling a unimodal feature vector using an isotropic Gaussian prior algorithm in a specified space to obtain a variational posterior probability distribution corresponding to the unimodal feature vector;

[0163] The feature distribution of the unimodal feature vector is calculated based on the variational posterior probability distribution.

[0164] In some embodiments, the step of calculating and determining the KL divergence based on the obtained feature distributions corresponding to different unimodal features includes:

[0165] A first KL divergence is calculated and determined based on the feature distribution of the first unimodal feature vector; a second KL divergence is calculated and determined based on the feature distribution of the second unimodal feature vector, wherein the first KL divergence and the second KL divergence have different operation positions of parameters involved in the calculation formula.

[0166] In some embodiments, the apparatus further comprises:

[0167] The second calculation module is used to calculate the divergence average according to the first KL divergence and the second KL divergence, and use the activation function to map the divergence average to the range of (0, 1) to obtain the fitness degree.

[0168] In some embodiments, the forming of target input data based on the adjusted weights and the first unimodal feature vector, the second unimodal feature vector, and the cross-modal fusion feature vector includes:

[0169] Multiplying the unimodal feature vector and the cross-modal fusion feature vector by corresponding weights respectively to obtain a unimodal representation and a multimodal representation;

[0170] The single-modal representation and the multi-modal representation are cascaded to obtain the target input data.

[0171] Another embodiment of the present invention further provides an electronic device, including:

[0172] at least one processor; and,

[0173] a memory communicatively connected to the at least one processor; wherein,

[0174] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to implement the multimodal fake news detection method as described in any one of the embodiments above.

[0175] Another embodiment of the present invention further provides a storage medium, which includes a stored program, wherein when the program is running, a device including the storage medium is controlled to execute the large-scale task planning method as described in any of the embodiments above.

[0176] The embodiment of the present invention also provides a computer program product, which is tangibly stored on a computer-readable medium and includes computer-readable instructions, and when the computer-executable instructions are executed, at least one processor executes a large-scale task planning method such as in the embodiment described above. It should be understood that each solution in this embodiment has the corresponding technical effect in the above method embodiment, which will not be repeated here.

[0177] It should be noted that the computer storage medium of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access storage medium (RAM), a read-only storage medium (ROM), an erasable programmable read-only storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only storage medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program configured to be used by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, antenna, optical cable, RF, etc., or any suitable combination of the above.

[0178] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of protection of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0179] One or more embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A multimodal fake news detection method, It is characterized in that include: Extracting features from multimodal news data and semantically aligning the extracted features to obtain a unimodal feature vector, wherein the multimodal news data includes text data and image data; Processing the unimodal feature vector based on a long-term short-term memory neural network to obtain a hidden state context vector; Processing the hidden state context vector based on a low-rank tensor algorithm to obtain a cross-modal fusion feature vector; Capturing the feature distribution of the unimodal feature vector based on a variational encoder; Determining a KL divergence based on the feature distribution calculation, wherein the KL divergence is used to quantify the degree of fitness between different unimodal feature vectors, wherein the degree of fitness between the different unimodal feature vectors is the degree of fitness between a first unimodal feature vector corresponding to the text data and a second unimodal feature vector corresponding to the image data; Adaptively adjusting the weights of the single-modal feature vector and the cross-modal fusion feature vector based on the degree of fitness, and forming target input data based on the adjusted weights and the single-modal feature vector and the cross-modal fusion feature vector; The target input data is input into the detection model to obtain a detection result corresponding to whether the multimodal news data is true news or false news.

2. The multimodal fake news detection method according to claim 1, It is characterized in that The extracting features of the multimodal news data and performing semantic alignment on the extracted features include: Extracting text data and image data from the multimodal news data; Performing feature extraction on the text data based on a text encoder to obtain a text feature vector; Extracting features from the image data based on an image encoder to obtain an image feature vector; The text feature vector and the image feature vector are mapped into a shared semantic space to obtain a plurality of different unimodal feature vectors.

3. The multimodal fake news detection method according to claim 1, It is characterized in that The method further comprises: Obtain multiple text features and multiple image features of multimodal news data for training; Predict the positive and negative correlations between different text features and image features; Construct a training set based on positively correlated text features and image features, and negatively correlated text features and image features; Construct a contrastive learning framework based on a multi-task learning model, a modality-specific multi-layer task perceptron, and a modality sharing layer; Using the training set to train the contrastive learning framework, so that the trained contrastive learning framework can perform feature extraction and semantic alignment of multimodal news data to obtain a unimodal feature vector; The feature loss amount of the contrastive learning framework when performing semantic alignment is calculated based on the unimodal feature vector, cosine loss, and margin value; The feature loss amount is positively fed back to the contrastive learning framework to optimize the contrastive learning framework.

4. The multimodal fake news detection method according to claim 1, It is characterized in that The step of processing the hidden state context vector based on a low-rank tensor algorithm to obtain a cross-modal fusion feature vector includes: Increasing each of the hidden state context vectors by a one-dimensional value; Perform outer product calculation on the hidden state context vector with one-dimensional value added to obtain the cross-modal fusion hidden context vector; The cross-modal fusion hidden context vector is calculated based on batch matrix multiplication of low-rank specific modality factors and additional modality representation to obtain the cross-modal fusion feature vector.

5. The multimodal fake news detection method according to claim 1, It is characterized in that The method of capturing the feature distribution of the unimodal feature vector based on a variational encoder includes: Sampling a unimodal feature vector using an isotropic Gaussian prior algorithm in a specified space to obtain a variational posterior probability distribution corresponding to the unimodal feature vector; The feature distribution of the unimodal feature vector is calculated based on the variational posterior probability distribution.

6. The multimodal fake news detection method according to claim 1, It is characterized in that Different unimodal feature vectors correspond to different feature distributions, and the KL divergence is determined based on the feature distribution, including: A first KL divergence is calculated and determined based on the feature distribution of the first unimodal feature vector; a second KL divergence is calculated and determined based on the feature distribution of the second unimodal feature vector, wherein the first KL divergence and the second KL divergence have different operation positions of parameters involved in the calculation formula.

7. The multimodal fake news detection method according to claim 6, It is characterized in that The method further comprises: The divergence average is calculated based on the first KL divergence and the second KL divergence, and the divergence average is mapped into the range of (0, 1) by using an activation function to obtain the degree of adaptation.

8. The multimodal fake news detection method according to claim 1, It is characterized in that The forming of target input data based on the adjusted weights and the single-modal feature vector and the cross-modal fusion feature vector includes: Multiplying the unimodal feature vector and the cross-modal fusion feature vector by corresponding weights respectively to obtain a unimodal representation and a multimodal representation; The single-modal representation and the multi-modal representation are cascaded to obtain the target input data.

9. A multimodal fake news detection device, It is characterized in that include: A feature extraction module, used to extract features from multimodal news data and semantically align the extracted features to obtain a unimodal feature vector, wherein the multimodal news data includes text data and image data; A first processing module, configured to process the unimodal feature vector according to a long-term short-term memory neural network to obtain a hidden state context vector; A second processing module is used to process the hidden state context vector according to a low-rank tensor algorithm to obtain a cross-modal fusion feature vector; A capture module, used for capturing the feature distribution of the unimodal feature vector using a variational encoder; A first calculation module, used for calculating and determining a KL divergence according to the feature distribution, wherein the KL divergence is used to quantify the degree of fitness between different unimodal feature vectors, wherein the degree of fitness between the different unimodal feature vectors is the degree of fitness between a first unimodal feature vector corresponding to the text data and a second unimodal feature vector corresponding to the image data; An adjustment module, configured to adaptively adjust the weights of the unimodal feature vector and the cross-modal fusion feature vector according to the degree of fitness, and form target input data based on the adjusted weights and the unimodal feature vector and the cross-modal fusion feature vector; The detection module is used to input the target input data into the detection model to obtain a detection result corresponding to whether the multimodal news data is true news or false news.

10. An electronic device, It is characterized in that include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to implement the multimodal fake news detection method according to any one of claims 1-8.

Images