Social news fake detection method based on multi-modal reasoning and domain expert learning

Abstract

This invention discloses a method for detecting fake news in social media based on multimodal reasoning and domain expert learning. The method includes: acquiring image and text data; a multimodal feature extraction and knowledge enhancement module extracting image semantic features, text features, text reasoning enhancement features, and image description features from the image and text data, respectively, and inputting them into a multi-domain expert learning module that uses a Gumbel-Softmax gating mechanism to obtain expert selection weights and performs weighted fusion processing; a cross-modal attention fusion module employing a multi-head attention mechanism to perform scaled dot product attention calculation, concatenation and fusion processing, and mapping through an FFN feedforward neural network to obtain multimodal fusion features; and a prediction optimization module using a classifier to predict the probability of fake news based on the input multimodal fusion features. This invention can fully utilize the core semantic information of each modality, significantly improving the overall accuracy and stability of multimodal fake news detection and enhancing cross-domain generalization ability.

Description

Technical Field

[0001] This invention relates to the field of fake news detection, and more particularly to a method for detecting fake social news based on multimodal reasoning and domain expert learning. Background Technology

[0002] With the rapid development of mobile internet and social media platforms, the dissemination of news information has gradually evolved from the traditional single text format to multimodal forms such as text-image fusion and video embedding. Multimodal news, due to its intuitive, vivid, and information-dense characteristics, greatly improves information dissemination efficiency and user reading experience. However, the high editability of multimodal content also makes it easier for false information to be packaged and spread.

[0003] In recent years, the spread of fake news on social media has accelerated. Some false content misleads public perception through methods such as splicing images, taking things out of context, or fabricating event scenes, seriously impacting social stability, public safety, and the public opinion environment. Especially in political, social, and public health fields, fake news can trigger mass panic or erroneous decision-making. Compared to traditional text news, multimodal news not only includes textual descriptions but also incorporates visual information such as images or videos. Mismatches between image content and text are common, enhancing the deceptive nature of the false content. Furthermore, fake news texts often contain logical flaws, factual inconsistencies, or contradictions in common sense. These problems are usually subtle and require background knowledge for judgment, increasing the difficulty of automatic identification.

[0004] Existing research on multimodal fake news detection mainly focuses on three directions: cross-modal semantic consistency detection methods, intramodal semantic consistency detection methods, and methods for cross-domain scenarios. Cross-modal semantic consistency detection methods primarily determine the authenticity of news by the semantic correspondence between text and images. Existing methods often directly use the inference results generated by LLM for discrimination or simple fusion, making it difficult to model the reliability of the generated content, easily introducing redundant or erroneous information, and affecting model stability and computational efficiency. Previously, pre-trained visual-language models were typically used to map image features and text features to a unified semantic space, achieving image-text alignment through similarity calculation or attention mechanisms. Existing cross-modal alignment methods mostly rely on global feature matching or simple attention mechanisms, making it difficult to capture fine-grained image-text semantic correspondences, resulting in insufficient ability to identify implicit semantic conflicts. Most existing methods introduce large visual models to convert images into text descriptions before semantic matching to enhance cross-modal understanding capabilities. Intramodal semantic consistency detection methods determine the semantic correlation within a single modality. Existing methods typically use prompts to guide the generation of text-level inference information or common sense analysis. The third approach addresses cross-domain scenarios. Considering the differences in expression, semantic distribution, and discrimination criteria across different news domains, some studies introduce domain modeling mechanisms to enhance generalization capabilities in multi-domain scenarios. Furthermore, different news domains exhibit significant differences in expression, semantic focus, and event presentation; the same text and images may have different meanings in different domains. Existing technologies have poor few-sample domain detection capabilities in cross-domain news, and similar text and images may have different meanings in different news domains. Existing methods struggle to dynamically adjust domain modeling methods, and performance is prone to degradation in few-sample domain information in cross-domain scenarios. This cross-domain semantic difference further increases the difficulty of model generalization, making traditional single-scenario training detection methods ill-suited for complex and ever-changing real-world application environments.

[0005] Therefore, how to enhance the semantic alignment capability of text and images while ensuring the reliability of inference, and effectively alleviate the problem of cross-domain semantic offset, has become a key technical problem that needs to be solved in the field of multimodal fake news detection. How to achieve a stable, accurate multimodal fake news detection technology with cross-domain generalization capability has become an urgent technical problem to be solved. Summary of the Invention

[0006] The purpose of this invention is to provide a social news fake news detection method based on multimodal reasoning and domain expert learning. The news fake news detection combined model integrates a multimodal feature extraction and knowledge enhancement module, a multi-domain expert learning module, a cross-modal attention fusion module, and a prediction optimization module. In the multimodal feature extraction and knowledge enhancement module, the text encoder extracts text features from text data in image and text data, and the image and text visual encoder extracts image semantic features from image data in image and text data. A large language model is used to perform text logic reasoning and common sense reasoning on image and text data, and semantic complementary fusion enhancement is performed through a bidirectional cross-attention mechanism. The multi-domain expert learning module obtains expert selection weights and performs weighted fusion processing based on the Gumbel-Softmax gating mechanism. The cross-modal attention fusion module uses a multi-head attention mechanism to perform scaling dot product attention calculation, splicing and fusion processing on image semantic enhancement features, text enhancement features, reasoning enhancement features, and image description enhancement features, and obtains multimodal fusion features through FFN feedforward neural network mapping. The prediction optimization module uses a classifier to identify and predict the probability of fake news, which significantly improves the multimodal fake news detection capability.

[0007] The objective of this invention is achieved through the following technical solution: A method for detecting fake social news based on multimodal reasoning and domain expert learning, the method comprising: S1. Obtain image and text data, and construct a combined model for detecting fake news, including a multimodal feature extraction and knowledge enhancement module, a multi-domain expert learning module, a cross-modal attention fusion module, and a prediction optimization module. S2, the multimodal feature extraction and knowledge enhancement module extracts image semantic features, text features, text reasoning enhancement features, and image description features from the image and text data, respectively, and inputs them into the multi-domain expert learning module. Based on the Gumbel-Softmax gating mechanism, expert selection weights are obtained, and weighted fusion processing is performed to obtain image semantic enhancement features. Text enhancement features Reasoning Enhancement Features and image description enhancement features ; S3, the cross-modal attention fusion module employs a multi-head attention mechanism to enhance image semantic features. Text enhancement features Reasoning Enhancement Features and image description enhancement features Scaling dot product attention calculations, concatenation and fusion processing, and mapping via FFN feedforward neural network to obtain multimodal fusion features. ; S4. The prediction optimization module uses a classifier containing a multilayer perceptron (MLP) and a sigmoid activation function to process the input multimodal fusion features. The probability prediction for fake news is expressed as follows: , This indicates the probability that the news is fake news.

[0008] To better realize the present invention, the present invention constructs an image and text label sample dataset containing image and text sample data and corresponding false labels. The image and text sample data includes image sample data and / or text data, and the false labels are label information indicating whether the image and text sample data is false and domain labels. The news false detection combined model uses the image and text label sample dataset for feature extraction, false identification and probability prediction model training.

[0009] Preferably, the multimodal feature extraction and knowledge enhancement module includes a text encoder, a visual encoder, and a large language model. The text encoder extracts text features from the text data in the visual data. Text features contain semantic information, and the image-text visual encoder extracts the image semantic features from the image data in the image-text data. Image semantic features include semantic information and visual feature information; the large language model performs textual logical reasoning and common sense reasoning on the image and text data, and obtains the textual logical reasoning TD view and common sense reasoning CS view respectively. The large language model identifies and extracts image description information ImaDesc from the image and text data. The textual logical reasoning TD view and common sense reasoning CS view are encoded by a text encoder to obtain features respectively. and ; Features and Text reasoning enhancement features are obtained through semantic complementarity fusion enhancement using a bidirectional cross-attention mechanism. The image description information ImaDesc is encoded using a text encoder to obtain image description features. .

[0010] Preferably, the multimodal feature extraction and knowledge enhancement module further includes an image-text consistency loss module, which incorporates image description features. Image semantic features Consistency judgment is performed to calculate the I / T Loss and it is used for dynamic loss constraints.

[0011] Preferably, the bidirectional cross-attention mechanism is characterized by... For query vectors, based on features Calculate the features from the key vector and value vector. The corresponding augmented representation Bidirectional cross-attention mechanism with features For query vectors, based on features Calculate the features from the key vector and value vector. The corresponding augmented representation This will enhance the representation , After pooling and concatenation according to feature dimensions, complementary features are enhanced using an FFN feedforward neural network. Text inference enhancement features are obtained through residual connections and LayerNorm normalization. The expression is as follows: .

[0012] Preferably, the multi-domain expert learning module has N domain categories for domain attributes and N corresponding domain experts. The module extracts the domain attribute information to which the image and text data belong. It constructs a gating network of dedicated experts and shared experts corresponding to the domain attribute information of the image and text data. Based on the Gumbel-Softmax gating mechanism, it obtains the expert selection weights of all experts in the gating network. These expert selection weights are then used to perform weighted fusion processing on image semantic features, text features, text reasoning enhancement features, and image description features to obtain the weighted fusion-processed image semantic enhancement features. Text enhancement features Reasoning Enhancement Features and image description enhancement features .

[0013] Preferably, the expression for expert selection of weights using the Gumbel-Softmax gating mechanism in conjunction with the gating network is as follows: , The selection weights for expert k, , These represent the weights of the original information and the noise in the current expert selection process, respectively. It is an exponential function; This is a parameter used to control the smoothness of the weight distribution.

[0014] Preferably, the multi-head attention mechanism in the cross-modal attention fusion module uses text enhancement features. query vector Image semantic enhancement features Reasoning Enhancement Features and image description enhancement features as a key vector AND value vector A linear mapping is performed to obtain the corresponding mapping vector, as shown in the following formula: , , , Image semantic enhancement features Reasoning Enhancement Features and image description enhancement features The three types of modal enhancement features, , , These are the weight matrices; the correlation between the text and each modality is calculated using scaled dot product attention, as shown in the following expression: , Let m be the attention weight matrix between the m-th modality and the text. Let d be the transpose of the m-th modal key vector matrix, where d is the feature dimension. To scale attention scores; , This represents the association between the m-th modality and the text. Let be the value vector matrix of the m-th modality.

[0015] Preferably, text enhancement features Separately with semantic enhancement features Reasoning Enhancement Features and image description enhancement features The correlation representations of the three modalities are fused separately to obtain Next, the features are concatenated and fused according to their feature dimensions, and then multimodal fused features are obtained by mapping through an FFN feedforward neural network. The expression is as follows: Where concat represents concatenating vector knowledge according to the feature dimension, and b represents the bias vector. This is the weight matrix. This represents the FFN feedforward neural network mapping.

[0016] Preferably, the news fake news detection combined model constructs a main classification loss L that includes fake news identification and probability prediction. cls Consistency discrimination calculation loss I / T Loss and expert equilibrium loss L b -Loss is used as a joint loss function to constrain model training.

[0017] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) The news fake news detection combined model of the present invention integrates a multimodal feature extraction and knowledge enhancement module, a multi-domain expert learning module, a cross-modal attention fusion module and a prediction optimization module. In the multimodal feature extraction and knowledge enhancement module, the text encoder extracts the text features of the text data in the image and text data, the image and text visual encoder extracts the image semantic features of the image data in the image and text data, and uses a large language model to perform text logic reasoning and common sense reasoning on the image and text data and performs semantic complementary fusion enhancement through a bidirectional cross attention mechanism. The multi-domain expert learning module obtains expert selection weights and weighted fusion processing based on the Gumbel-Softmax gating mechanism. The cross-modal attention fusion module uses a multi-head attention mechanism to perform scaling dot product attention calculation, splicing and fusion processing on image semantic enhancement features, text enhancement features, reasoning enhancement features and image description enhancement features, and obtains multimodal fusion features through FFN feedforward neural network mapping. The prediction optimization module uses a classifier to identify fake news and predict its probability, which significantly improves the multimodal fake news detection capability. The present invention can make full use of the core semantic information of each modality and significantly improve the overall accuracy and stability of multimodal fake news detection.

[0018] (2) This invention uses the DV-CCM module to perform cross-attention modeling and fusion of textual logical reasoning and common sense reasoning information, so that the two types of reasoning features complement each other and improve the reliability and effectiveness of reasoning enhancement features from the structural level; the multi-domain expert learning module combined with domain embedding vectors enables the model to adapt to the semantic differences of different domains according to the domain features of the input samples. After expert selection weights and weighted fusion processing, the cross-domain generalization ability is improved, and the detection performance of the model in cross-domain and complex social scenarios is significantly improved. Attached Figure Description

[0019] Figure 1 This is a flowchart of the social news fake news detection method of the present invention; Figure 2 This is a schematic diagram illustrating the principle of the social news fake news detection method in the embodiment; Figure 3 This is a schematic diagram illustrating the principle of the news false detection combined model in the embodiment; Figure 4 This is a graph showing the prediction performance evaluation results of the news fake news detection combinatorial model trained in the example. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to embodiments: Example like Figure 1 As shown, a method for detecting fake social news based on multimodal reasoning and domain expert learning includes the following steps: S1. Obtain image and text data, such as... Figures 1-3 As shown, a combined model for news fake news detection is constructed, comprising modules for multimodal feature extraction and knowledge enhancement, multi-domain expert learning, cross-modal attention fusion, and prediction optimization. An image-text label sample dataset is constructed, containing image and text sample data and corresponding fake labels. The image-text sample data includes image sample data and / or text data, while the fake labels are information indicating whether the image-text sample data is fake, as well as domain labels. The combined news fake news detection model is trained using this image-text label sample dataset for feature extraction, fake news identification, and probability prediction. The combined news fake news detection model constructs a main classification loss L that includes fake news identification and probability prediction. cls Consistency discrimination calculation loss I / T Loss and expert equilibrium loss L b The joint loss function of -Loss is used as a constraint for model training. The total loss function of the news fake news detection combined model is: ,in For the total loss, To determine the loss for consistency, , These are the loss weighting coefficients, To balance the losses for experts; The primary classification loss measures the difference between the predicted probability and the true label, and is expressed as follows: Where N is the number of training samples, For the label of sample i, Let be the predicted probability of sample i.

[0021] The news fake news detection combinatorial model trained in this embodiment performs well on core metrics such as Accuracy, Macro-F1, and F1. Figure 4 As shown, the trained news fake news detection combinatorial model maintains a consistent trend under different evaluation dimensions and data distributions, indicating that the model is not a random fit but has good stability.

[0022] S2. Input the image and text data to be detected (including image data and text data) into the multimodal feature extraction and knowledge enhancement module of the news fake news detection combined model, such as... Figure 3As shown, the multimodal feature extraction and knowledge enhancement module extracts image semantic features, text features, text inference enhancement features, and image description features from the image and text data, respectively. Internally, the multimodal feature extraction and knowledge enhancement module includes a text encoder (in this embodiment, a pre-trained BERT-base model is used, which contains a 12-layer Transformer structure with 12 attention heads), a BLIP image-text visual encoder (containing a 12-layer visual Transformer structure with 12 attention heads), and a large language model. The text encoder extracts text features from the text data in the image and text data. Text features contain semantic information, and the image-text visual encoder extracts the image semantic features from the image data in the image-text data. Image semantic features include semantic information and visual feature information; in this embodiment, text features... Image semantic features The feature vector dimension is 768. A large language model (in this embodiment, the ChatGLM3-6B model or deepseek model is used, with a context window length of 8192 and a generation temperature parameter set to 0.7) performs text logic reasoning and common sense reasoning on the image and text data, obtaining the text logic reasoning TD view and common sense reasoning CS view respectively. The large language model identifies and extracts image description information (ImaDesc) from the image and text data. The text logic reasoning TD view focuses on the language and logic features of the news text itself, analyzing dimensions including language style (objective / subjective, whether exaggerated), logical coherence (timeline, causal relationship rationality), and information completeness (whether key details and sources are clear). The output requires first extracting text features, then deriving the tendency towards authenticity, and labeling the judgment results. This allows the large language model to make a judgment on the authenticity of the news based on the text's own logic, focusing on the internal logical consistency of the text. The common sense reasoning CS view only relates to common sense, objective laws, and known facts, and does not rely on text language features. The analysis dimensions include common sense matching degree and domain common sense fit degree (such as policy requirements and industry practices); the output should first clearly state the relevant common sense basis, then compare the fit degree between the news content and common sense and explain the reasons (50-120 words), and mark the judgment result, so that the large language model can combine external common sense to make a judgment on the authenticity of the news, focusing on the consistency between the text and common sense.

[0023] The text logic reasoning (TD view) and common sense reasoning (CS view) are encoded by a text encoder to obtain features. and ; Features and Text reasoning enhancement features are obtained through semantic complementarity fusion enhancement using a bidirectional cross-attention mechanism. In some embodiments, this embodiment constructs a dual-view complementary module DV-CCM (such as...). Figure 3 The DV-CCM module shown uses a DV-CCM model to interactively model semantic-level reasoning and cognitive-level reasoning information, enabling the two types of reasoning features to complement each other and reducing the influence of irrelevant reasoning information. The final representation T is obtained. reason (For further modeling in subsequent models), the Dual-View Complementary Module (DV-CCM) contains a bidirectional cross-attention mechanism, which uses the bidirectional cross-attention mechanism to focus on features. and Interactive modeling is performed using a bidirectional cross-attention mechanism based on features. For query vector Query, based on features Calculate the key vector Key and value vector Value from the features The corresponding augmented representation Bidirectional cross-attention mechanism with features For query vectors, based on features Calculate the features from the key vector and value vector. The corresponding augmented representation This will enhance the representation , After pooling and concatenation along feature dimensions, a joint representation is formed. ; and complementary feature enhancement is achieved by constructing an FFN feedforward neural network consisting of two MLP layers. , Text inference enhancement features are obtained through residual connections and LayerNorm normalization. The expression is as follows: , LayerNorm normalization is applied. The image description information ImaDesc is encoded using a text encoder to obtain the image description features. Image description features The feature dimension is 768.

[0024] The multimodal feature extraction and knowledge enhancement module also includes an image-text consistency loss module (I / T-Discriminator) to ensure the reliability of image description information. The image-text consistency loss module (i.e....) Figure 3 The I / T-Discriminator in the image describes features. Image semantic features The consistency judgment is used to calculate the I / T Loss and apply it to the dynamic loss constraint.

[0025] Image semantic features, text features, text reasoning enhancement features, and image description features (each with a feature dimension of 768) are input into a multi-domain expert learning module. Based on the Gumbel-Softmax gating mechanism, expert selection weights are obtained, and weighted fusion processing is performed to yield image semantic enhancement features. Text enhancement features Reasoning Enhancement Features and image description enhancement features The multi-domain expert learning module has N domain categories for domain attributes (example domains include politics, society, entertainment, health, economics, military, education, science and technology, disaster, etc.) and N corresponding domain experts. The module extracts the domain attribute information to which the image and text data belong. It constructs a gating network of dedicated experts and shared experts corresponding to the domain attribute information of the image and text data (e.g., the gating network includes 6 domain experts and 4 shared experts; the gating network takes the concatenation result of the domain embedding vector and pooled content features as input, and generates the corresponding logits for each expert through a multilayer perceptron). Based on the Gumbel-Softmax gating mechanism, it obtains the expert selection weights for all experts in the gating network. These expert selection weights are then used to perform weighted fusion processing on image semantic features, text features, text inference enhancement features, and image description features to obtain the weighted fusion processed image semantic enhancement features. Text enhancement features Reasoning Enhancement Features and image description enhancement features The expression for expert selection weights using the Gumbel-Softmax gating mechanism in conjunction with a gated network is as follows: , The selection weight for expert k (if the number of experts in the gating network is 10, then expert k is the kth expert among the 10 experts, and k is the expert number). , These represent the weights of the original information and the noise (Gumbel noise) in the current expert selection process, respectively. It is an exponential function; These are parameters used to control the smoothness of the weight distribution. If the gating network has 10 experts, then the image semantic enhancement features... Text enhancement features Reasoning Enhancement Features and image description enhancement features The expressions are as follows: ; ; ; in, The output of expert k represents the image semantic enhancement feature. Text enhancement features Reasoning Enhancement Features and image description enhancement features All features are 768-dimensional feature vectors, which can fully capture semantic features from different domains and improve cross-domain generalization ability. To avoid overuse of some experts and underutilization of others, an expert load balancing loss is introduced. This is used to optimize the model loss and ensure a balanced load among experts. The specific formula is as follows: ,in, Here, represents the weights of sample i and expert k, N is the number of training samples, and the expert load balancing loss is... Its purpose is to penalize situations where the weight distribution is too concentrated, so that the average weight of each expert is as close as possible to the ideal value, thereby improving the utilization rate of experts and the stability of the model.

[0026] S3, the cross-modal attention fusion module employs a multi-head attention mechanism to enhance image semantic features. Text enhancement features Reasoning Enhancement Features and image description enhancement features Scaling dot product attention calculations, concatenation and fusion processing, and mapping via FFN feedforward neural network to obtain multimodal fusion features. Multi-head attention mechanism in cross-modal attention fusion module for text enhancement features query vector (i.e., Query), image semantic enhancement features Reasoning Enhancement Features and image description enhancement features as a key vector (i.e., Key) and value vector (i.e., Value), perform a linear mapping to obtain the corresponding mapping vector, as shown in the following formula: , , , Image semantic enhancement features Reasoning Enhancement Features and image description enhancement features The three types of modal enhancement features, , , These are the weight matrices; the correlation between the text and each modality is calculated using scaled dot product attention (the correlation between the text and each modality is calculated using scaled dot product attention. The formula is as follows: first, multiply by the transpose matrix to obtain the original association score between the text and each token of the modality; then divide by...). Where d represents the feature dimension, which is scaled; finally, softmax normalization is applied to obtain attention weights that sum to 1. The expression is as follows: , Let m be the attention weight matrix between the m-th modality and the text. Let d be the transpose of the m-th modal key vector matrix, where d is the feature dimension. To scale attention scores; , This represents the association between the m-th modality and the text. Let A be the value vector matrix of the m-th modality. The resulting attention weights A... m With V m Multiplication filters out text-related information from the modalities, where highly relevant tokens contribute more semantics, while irrelevant tokens are weakened. Simultaneously, to avoid losing core text semantics during modal information fusion, the original text enhancement vector T is... enh Modal-related representation O m To add the vectors, we introduce LayerNorm to normalize them, as shown in the following expression: .

[0027] Text enhancement features Separately with semantic enhancement features Reasoning Enhancement Features and image description enhancement features The correlation representations of the three modalities are fused separately to obtain (These correspond to the outputs of the three modes, respectively) Corresponding reasoning enhanced features , Corresponding image description enhancement features , Corresponding image semantic enhancement features Then, the features are concatenated and fused according to their feature dimensions, and multimodal fused features are obtained by mapping through an FFN feedforward neural network. (As the final multimodal fusion feature of the model, it can fully integrate the core semantic information of text, image, reasoning, and image description, and has strong discriminative power for subsequent prediction tasks.) The expression is as follows: Where concat represents concatenating vector knowledge according to the feature dimension, and b represents the bias vector. This is the weight matrix. This represents the FFN feedforward neural network mapping.

[0028] S4. The prediction optimization module uses a classifier containing a multilayer perceptron (MLP) and a sigmoid activation function to process the input multimodal fusion features. The probability prediction of fake news (probability interval [0,1]) is expressed as follows: , This indicates the probability that the news is fake news.

[0029] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting fake social news based on multimodal reasoning and domain expert learning, characterized in that: The methods include: S1. Obtain image and text data, and construct a combined model for detecting fake news, including a multimodal feature extraction and knowledge enhancement module, a multi-domain expert learning module, a cross-modal attention fusion module, and a prediction optimization module. S2, the multimodal feature extraction and knowledge enhancement module extracts image semantic features, text features, text reasoning enhancement features, and image description features from the image and text data, respectively, and inputs them into the multi-domain expert learning module. Based on the Gumbel-Softmax gating mechanism, expert selection weights are obtained, and weighted fusion processing is performed to obtain image semantic enhancement features. Text enhancement features Reasoning Enhancement Features and image description enhancement features The multi-domain expert learning module has N domain categories for domain attributes, each corresponding to N domain experts. The module extracts the domain attribute information to which the text and image data belongs. It then constructs a gating network of dedicated experts and shared experts corresponding to the domain attribute information of the text and image data. The expression for the expert selection weights using the Gumbel-Softmax gating mechanism, combined with the gating network, is as follows: , The selection weights for expert k, , These represent the weights of the original information and the noise in the current expert selection process, respectively. It is an exponential function; These are parameters used to control the smoothness of the weight distribution; S3, the cross-modal attention fusion module employs a multi-head attention mechanism to enhance image semantic features. Text enhancement features Reasoning Enhancement Features and image description enhancement features Scaling dot product attention calculations, concatenation and fusion processing, and mapping via FFN feedforward neural network to obtain multimodal fusion features. ; S4. The prediction optimization module uses a classifier containing a multilayer perceptron (MLP) and a sigmoid activation function to process the input multimodal fusion features. The probability prediction for fake news is expressed as follows: , This indicates the probability that the news is fake news.

2. The social news fake news detection method based on multimodal reasoning and domain expert learning according to claim 1, characterized in that: A text-image label sample dataset is constructed, which contains text-image sample data and corresponding false labels. The text-image sample data includes image sample data and / or text data. The false labels are label information indicating whether the text-image sample data is false and domain labels. The news false detection combined model uses the text-image label sample dataset for feature extraction, false identification and probability prediction model training.

3. The social news fake news detection method based on multimodal reasoning and domain expert learning according to claim 1, characterized in that: The multimodal feature extraction and knowledge enhancement module includes a text encoder, a visual encoder, and a large language model. The text encoder extracts text features from the text data in the visual data. Text features contain semantic information, and the image-text visual encoder extracts the image semantic features from the image data in the image-text data. Image semantic features include semantic information and visual feature information; The large language model performs text-based logical reasoning and common-sense reasoning on the image and text data, obtaining the text-based logical reasoning TD view and the common-sense reasoning CS view, respectively. The large language model also identifies and extracts image description information (ImaDesc) from the image and text data. Finally, the text-based logical reasoning TD view and the common-sense reasoning CS view are encoded by a text encoder to obtain features. and ; Features and Text reasoning enhancement features are obtained through semantic complementarity fusion enhancement using a bidirectional cross-attention mechanism. The image description information ImaDesc is encoded using a text encoder to obtain image description features. .

4. The social news fake news detection method based on multimodal reasoning and domain expert learning according to claim 3, characterized in that: The multimodal feature extraction and knowledge enhancement module also includes an image-text consistency loss module, which incorporates image description features. Image semantic features Consistency judgment is performed to calculate the I / T Loss and it is used for dynamic loss constraints.

5. The social news fake news detection method based on multimodal reasoning and domain expert learning according to claim 3, characterized in that: The bidirectional cross-attention mechanism is characterized by For query vectors, based on features Calculate the features from the key vector and value vector. The corresponding augmented representation Bidirectional cross-attention mechanism with features For query vectors, based on features Calculate the features from the key vector and value vector. The corresponding augmented representation This will enhance the representation , After pooling and concatenation according to feature dimensions, complementary features are enhanced using an FFN feedforward neural network. Text inference enhancement features are obtained through residual connections and LayerNorm normalization. The expression is as follows: .

6. The social news fake news detection method based on multimodal reasoning and domain expert learning according to claim 1, characterized in that: Based on the Gumbel-Softmax gating mechanism, the expert selection weights of all experts in the gating network are obtained. These expert selection weights are then used to perform weighted fusion processing on image semantic features, text features, text inference enhancement features, and image description features, resulting in weighted fusion of image semantic enhancement features. Text enhancement features Reasoning Enhancement Features and image description enhancement features .

7. The social news fake news detection method based on multimodal reasoning and domain expert learning according to claim 1, characterized in that: The multi-head attention mechanism in the cross-modal attention fusion module uses text enhancement features. query vector Image semantic enhancement features Reasoning Enhancement Features and image description enhancement features as a key vector AND value vector A linear mapping is performed to obtain the corresponding mapping vector, as shown in the following formula: , , , Image semantic enhancement features Reasoning Enhancement Features and image description enhancement features The three types of modal enhancement features, , , These are the weight matrices; the correlation between the text and each modality is calculated using scaled dot product attention, as shown in the following expression: , Let m be the attention weight matrix between the m-th modality and the text. Let d be the transpose of the m-th modal key vector matrix, where d is the feature dimension. To scale attention scores; , This represents the association between the m-th modality and the text. Let be the value vector matrix of the m-th modality.

8. The social news fake news detection method based on multimodal reasoning and domain expert learning according to claim 6, characterized in that: Text enhancement features Separately with semantic enhancement features Reasoning Enhancement Features and image description enhancement features The correlation representations of the three modalities are fused separately to obtain Next, the features are concatenated and fused according to their feature dimensions, and then multimodal fused features are obtained by mapping through an FFN feedforward neural network. The expression is as follows: Where concat represents concatenating vector knowledge according to the feature dimension, and b represents the bias vector. This is the weight matrix. This represents the FFN feedforward neural network mapping.

9. The social news fake news detection method based on multimodal reasoning and domain expert learning according to claim 2, characterized in that: The news fake news detection combined model is constructed by including a main classification loss L for fake news identification and probability prediction. cls Consistency discrimination calculation loss I / T Loss and expert equilibrium loss L b -Loss is used as a joint loss function to constrain model training.

Images