Multi-modal field adaptive false information detection method and device

Abstract

The invention discloses a multi-modal field adaptive false information detection method and device, and the method comprises the following steps: S1, combining a text part and an image part of a news sample based on a large visual language model and a large language model, and generating comment information; s2, extracting features of news samples from three perspectives of images, texts and multiple modes; s3, extracting comment features of the comment information, and setting a hierarchical comment guide fusion mechanism to fuse the comment features with text features and visual features respectively; s4, constructing a hybrid expert network to extract and fuse specific domain knowledge and other cross-domain knowledge of the news sample; carrying out single-view-angle prediction based on the features of each view angle and carrying out adaptive instance normalization; and performing weighted splicing on the features of different visual angles to obtain an aggregation feature, and inputting the aggregation feature into a multi-layer perceptron to obtain a classification detection result. The method has the technical effect of accurate cross-modal semantics.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, and in particular to a multi-modal field adaptive false information detection method and device. BACKGROUND

[0002] In the information explosion era of the Internet, false information is rampant, seriously damaging the health of the information ecosystem, affecting the public's trust in information, and even possibly causing negative impacts on social stability and economic development. Traditional manual review methods are difficult to cope with the massive news data, so there is an urgent need for automated false information detection technology. With the development of multimedia technology, news forms have changed from simple text to multimedia content containing pictures, videos, etc. Multi-modal news is more attractive to readers and more likely to spread false information. Multi-modal detection methods that integrate visual content and text information have become the main trend in false information detection.

[0003] False information detection faces many challenges, mainly in the following three aspects. First, single modal information cannot support accurate judgment, and the detection effect is limited. Current false news detection methods mainly rely on single modal data input, such as using only news text as the analysis object. However, with the continuous evolution of false information dissemination means, many false information enhances its misleadingness and dissemination power through various forms such as text and image combination, audio and video assistance. Relying solely on text content for judgment often leads to one-sided information and makes it difficult to fully capture the traces of forgery, thereby limiting the recognition accuracy and robustness of the detection system. Second, multi-modal data is highly heterogeneous, and the fusion and feature extraction mechanism is imperfect. Multi-modal false news usually contains text, images, comments and other types of information, which have significant differences in structure, semantic expression and representation. Existing technologies generally use independent modal modeling followed by simple concatenation or weighted fusion, lacking effective modeling of cross-modal semantic association. In addition, some methods fail to fully extract deep semantic information from text or key text content from images during feature extraction, resulting in incomplete feature expression and affecting the final discrimination effect. Third, the model's generalization ability is insufficient, making it difficult to adapt to complex and changing real-world application scenarios. Most current false information detection models are trained for specific domains or platforms, while in real-world applications, false information covers multiple topics, language styles and dissemination channels, showing high diversity and dynamic change characteristics. Traditional models often show poor adaptability and generalization performance when faced with new types of false information across domains, platforms or languages, making it difficult to meet the stability and scalability requirements of detection systems in real-world environments.

[0004] To address the first challenge, existing research has attempted to introduce multimodal misinformation detection methods, improving recognition capabilities from the perspectives of image-text matching and cross-modal fusion. However, these methods still have limitations when dealing with misinformation in complex scenarios. Some methods attempt to use large models to simulate user behavior and generate comment information to assist the detection process, but overall, they lack a deep understanding of multimodal news semantics. Regarding the second challenge, existing research largely focuses on image-text consistency analysis, emphasizing the interaction between text and images, but neglecting the important role of comment information in misinformation identification. Comment information contains a wealth of user-perspective feedback; however, current methods are still insufficient in integrating comments with other modal information, failing to achieve deep multimodal collaborative learning. Regarding the third challenge, although existing research has explored the possibility of multi-domain misinformation detection, practical applications still face problems such as incomplete domain labels and modal dependency differences, and effective solutions have not yet been developed. Therefore, there is an urgent need to construct a more universal and adaptable detection framework to cope with the increasingly complex environment of misinformation dissemination.

[0005] In summary, existing misinformation detection technologies still have many shortcomings when dealing with complex, multi-domain, and multimodal scenarios, especially in cross-modal semantic understanding, comment information fusion, and model generalization capabilities, where they still face significant challenges. Summary of the Invention

[0006] In view of this, it is necessary to provide a multimodal domain adaptive method and apparatus for detecting fake information, so as to effectively solve the technical problem of difficult cross-modal semantic understanding in the current fake information detection.

[0007] This invention provides a multimodal domain adaptive method for detecting fake information, comprising the following steps: Step S1: Based on the large visual language model and the large language model, combine the text and image parts of the news sample to generate multimodal perception comment information; Step S2: Extract features from the news sample from three perspectives: image, text, and multimodal, to obtain visual features, text features, and multimodal features; Step S3: Extract the comment features of the comment information, and set up a hierarchical comment guidance fusion mechanism to fuse the comment features with the text features and the visual features respectively; Step S4: Construct a hybrid expert network to extract and fuse specific domain knowledge and other cross-domain knowledge of the news sample to obtain a domain-aware knowledge representation of the news sample; perform single-view prediction based on features from each perspective and perform adaptive instance normalization; weight and concatenate the features from different perspectives to obtain aggregated features; input the aggregated features into a multilayer perceptron for classification to obtain classification detection results.

[0008] Preferably, step S1 specifically comprises: A large visual language model is used to describe the image portion of the news sample, generating a visual description corresponding to the image portion; By combining a pre-defined user attribute library, a large language model is used to combine the text portion with the visual description to generate multimodal perceived comment information.

[0009] Preferably, step S2 specifically comprises: The first pre-trained model is used to extract visual features of the image portion; the second pre-trained model is used to extract text features of the text portion; cross-attention is used to perform cross-modal alignment of the visual features and the text features; and the third pre-trained model is used to simultaneously encode the image portion and the text portion to extract multimodal features. Cross-modal alignment of the visual features and the text features is performed using cross-attention, specifically as follows: ; ; ; ; in, The first, representing the textual perspective One point of attention, The first representing the visual perspective One's attention, Represents the query vector. Represents the key vector. Represents a value vector, in calculation hour, , , In calculation hour, , , , Represents the attention function. For a learnable parameter matrix, The query parameter matrix representing text features. The key parameter matrix representing text features, The matrix of values ​​representing text features The query parameter matrix representing visual features. Key parameter matrix representing visual features, The matrix of values ​​representing visual features. Representing text features, This represents the features of the aligned text. Indicates visual characteristics, This represents the visual features after alignment. This indicates the multi-head attention mechanism module. This indicates a matrix concatenation operation.

[0010] Preferably, step S3 specifically comprises: The comment features of the comment information are extracted using a fourth pre-trained model; A mask module for interactive fusion of comments is set up, and the comment features are interactively fused with aligned text features and aligned visual features based on the mask module. A hierarchical comment guidance fusion mechanism is set up, which treats the mask module as a sub-module. By stacking the sub-modules layer by layer, deep semantic fusion is performed between the comment features and the visual features, and between the comment features and the text features, to obtain the fused text features and fused visual features for comment guidance.

[0011] Preferably, step S4 specifically comprises: Step S41: Introduce learnable domain embeddings, assign a domain expert to each domain, and simultaneously set up a domain gating network to construct the hybrid expert network; calculate the importance weights of the representation dimensions for each perspective, aggregate the importance weights with the original features of the corresponding perspective, and concatenate them with the domain embeddings before inputting them into the domain gating network to obtain the hybrid expert network selection weights; simultaneously input the features of each perspective into the hybrid expert network, combine the hybrid expert network selection weights to extract specific domain knowledge, and fuse cross-domain knowledge to obtain a domain-aware knowledge representation; perform one-hot encoding on news samples labeled with domain tags to obtain domain tags after label smoothing, which are used to supervise the training of the domain gating network; Step S42: Perform single-view prediction for features from various perspectives that have extracted specific domain knowledge and integrated cross-domain knowledge, and supervise the prediction by using binary cross-entropy loss to obtain the mean and variance of the single-view prediction scores. Based on the mean and variance, further adjust the feature weights within each perspective by adaptive instance normalization. Step S43: Project the features from different perspectives onto a unified dimension through a linear layer and then concatenate them. Perform cross-attention calculation on the concatenated features and the domain embedding to obtain the feature weights for each perspective. Based on the feature weights, perform weighted concatenation of the features from each perspective to obtain aggregated features. Step S44: Input the aggregated features into a multilayer perceptron for classification to obtain the predicted labels of the news, and determine the authenticity of the news based on the predicted labels.

[0012] Preferably, in step S41, the importance weight of the representation dimension is calculated for each viewpoint. This importance weight is then aggregated with the original features of the corresponding viewpoint and concatenated with the domain embedding before being input into the hybrid expert network to obtain the hybrid expert network selection weights. Simultaneously, the original features of each viewpoint are input into the hybrid expert network, and the domain-aware knowledge representation is calculated by combining the hybrid expert network selection weights. Specifically: in, The output is a domain-aware knowledge representation. For domain-gated networks, Represents the primitive features of a mode. , To integrate text representations, To fuse image representations, For the multimodal features, Representing modes Importance weights This indicates the domain embedding. Indicates splicing, For domain expert network; In step S41, the news samples labeled with domain tags are subjected to one-hot encoding to obtain domain tags after tag smoothing, specifically as follows: ; in, For smoothed domain labels, The domain labels before smoothing. For smoothing coefficients, For the number of domains.

[0013] Preferably, step S42 specifically includes: Single-view prediction is performed on features from various perspectives that have extracted specific domain knowledge and integrated cross-domain knowledge; the visual perspective prediction is as follows: ; in, The true / false prediction score represents the visual perspective. This represents a two-layer multilayer perceptron. Knowledge representation for the visual perspective domain; Supervision is performed using binary cross-entropy loss, and the mean and variance of the single-view prediction scores are calculated using two specific multilayer perceptrons: ; ; in, For the loss of true / false classification from a visual perspective, As a label for the authenticity of news, The true / false prediction score is based on the visual perspective. To predict the mean of the scores, For multilayer perceptrons used to calculate the mean, To predict the variance of the scores, For use in calculating variance, a multilayer perceptron; Based on the mean and variance, the feature weights within each viewpoint are further adjusted through adaptive instance normalization: ; in, The features are normalized from the visual perspective. Knowledge representation for the visual perspective domain. The mean of knowledge representation for the visual perspective domain. Variance of knowledge representation for the visual perspective domain.

[0014] Preferably, step S43 specifically includes: Domain-aware features from different perspectives are projected onto a unified dimension through a linear layer and then concatenated to obtain concatenated features: ; in, As a feature in the visual perspective. For visual perspective, linear layer The features are normalized from the visual perspective. This represents an intermediate feature from a textual perspective. A linear layer from a textual perspective. The normalized features of the text perspective For intermediate features from a multimodal perspective, For multimodal view linear layers, These are the normalized features from a multimodal perspective; The concatenated features are then subjected to cross-attention calculation with the domain embedding to obtain the feature weights for each viewpoint: ; in, For feature weights, For the Softmax function, Embedding for learnable domains, Indicates matrix concatenation. Indicates feature dimension; Based on the aforementioned feature weights, the features from each perspective are weighted and concatenated to obtain aggregated features: ; in, As an aggregation feature, For the feature weights of the visual perspective, For the feature weights from the text perspective, The feature weights are from a multimodal perspective.

[0015] Preferably, step S44 specifically includes: ; ; in, For predicting labels, For actual labels, As an aggregation feature, The final true or false prediction loss.

[0016] The present invention also provides a multimodal domain adaptive fake information detection device, including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements the multimodal domain adaptive fake information detection method.

[0017] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention proposes a multimodal domain-adaptive fake information detection method based on social generation enhancement. Through the collaborative cooperation of a large visual language model and a large language model, multimodal user comments are generated, and news features are extracted from three perspectives: image, text, and multimodal. Hierarchical comment guidance and fusion are used to achieve deep fusion of comment information with news images and text, mining deep semantics. A domain-adaptive knowledge collaborative refinement mechanism is introduced, and domain-aware hybrid experts are used to achieve the extraction and fusion of specific domain and cross-domain knowledge. A domain-aware attention module is used to adaptively aggregate features from different perspectives based on domain information, thereby achieving accurate detection of complex fake information. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 is a flowchart of an embodiment of a multimodal domain adaptive fake information detection method provided by the present invention; Detailed Implementation

[0019] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0020] The technical terminology of this invention will be explained and described below: Large Language Model (LMM): A large language model is a deep neural network model with a very large number of parameters, specifically designed for processing and generating natural language tasks such as text understanding, translation, question answering, summarizing, and dialogue.

[0021] Large Vision-Language Model (LVLM): LVLM is a multimodal deep learning model that integrates visual and linguistic information, possessing cross-modal understanding and generation capabilities. It can describe image content using language, and can also generate images based on text or determine image relevance.

[0022] Comment generation: This is the process of automatically generating semantically logical, stylistically consistent, and expressive text comments using natural language processing and deep learning techniques. This process typically simulates the subjective opinions or emotional inclinations of human users based on given contextual information, outputting a comment text in natural language form.

[0023] Visual features: Numerical representations extracted after encoding image content through image processing algorithms or deep learning models, used to capture visual information such as color, texture, shape, objects and their spatial relationships in an image.

[0024] Textual features are numerical representations extracted from natural language text through encoding, used to characterize the semantics, syntax, structure, and contextual information of the text.

[0025] Multimodal Features: Numerical representations extracted by jointly encoding text and images through a pre-trained multimodal model. These features can simultaneously capture semantic associations and interaction information between different modalities, enabling cross-modal understanding and reasoning.

[0026] Cross-modal cross-attention alignment is a technique that uses attention mechanisms to establish relationships between different modalities. By calculating the attention weights of one modality, such as text, to another modality, such as an image, feature alignment and information exchange between the two modalities are achieved.

[0027] Mixture of Experts (MoE) is a model structure design method that improves the expressive power and efficiency of a model by combining multiple "expert" sub-models and having a "gating network" determine which experts should process the input data.

[0028] Multi-head Attention: An attention mechanism that maps queries and key-value pairs to outputs, capturing different types of dependencies through parallel computation of multiple attention heads.

[0029] Example 1 Please see Figure 1 The multimodal domain adaptive fake information detection method in this embodiment specifically includes the following steps: Step S1: Based on the large visual language model and the large language model, combine the text and image parts of the news sample to generate multimodal perception comment information; Step S2: Extract features from the news sample from three perspectives: image, text, and multimodal, to obtain visual features, text features, and multimodal features; Step S3: Extract the comment features of the comment information, and set up a hierarchical comment guidance fusion mechanism to fuse the comment features with the text features and the visual features respectively; Step S4: Construct a hybrid expert network to extract and fuse specific domain knowledge and other cross-domain knowledge of the news sample to obtain the domain label of the news sample; perform single-view prediction based on the features of each perspective and perform adaptive instance normalization; weight and concatenate the features of different perspectives to obtain aggregated features; input the aggregated features into a multilayer perceptron for classification to obtain the classification detection result.

[0030] like Figure 1 As shown, this embodiment provides a multimodal domain-adaptive fake information detection method based on social generation enhancement, enabling rapid and accurate detection of fake information on social media. The method first generates user comment information with multimodal perception capabilities through the collaborative operation of a large visual language model and a large language model, thereby enhancing the understanding and judgment of news content. Based on this, news features are extracted from three perspectives: image, text, and multimodal. A hierarchical comment guidance fusion mechanism is designed to achieve deep semantic fusion between comment information and news images and text, uncovering potential fake information clues. Furthermore, this embodiment introduces a domain-adaptive knowledge collaborative refinement mechanism, employing a domain-aware hybrid expert network to effectively extract and fuse knowledge representations from specific and cross-domain fields. By constructing a domain-aware attention module, feature representations from different perspectives can be adaptively aggregated according to the domain of the input news, thereby improving the model's generalization ability and detection accuracy in complex cross-domain application scenarios.

[0031] Specifically, step S1 is as follows: A large visual language model is used to describe the image portion of the news sample, generating a visual description corresponding to the image portion; By combining a pre-defined user attribute library, a large language model is used to combine the text portion with the visual description to generate multimodal perceived comment information.

[0032] Step S1 constructs a fine-grained attribute-aware comment generation module for image description generation: using the Qwen2.5-VL large visual language model to describe news images and generate corresponding visual descriptions; it is also used for multimodal perception user comment generation: combining a pre-designed user attribute library, the Qwen2.5 large language model combines news text and visual descriptions of images to generate multimodal perception user-generated comments.

[0033] Specifically, step S2 is as follows: The first pre-trained model is used to extract visual features of the image portion; the second pre-trained model is used to extract text features of the text portion; cross-attention is used to perform cross-modal alignment of the visual features and the text features; and the third pre-trained model is used to simultaneously encode the image portion and the text portion to extract multimodal features. Cross-modal alignment of the visual features and the text features is performed using cross-attention, specifically as follows: ; ; ; ; in, The first, representing the textual perspective One's attention, The first representing the visual perspective One's attention, Represents the query vector. Represents the key vector. Represents a value vector, in calculation hour, , , In calculation hour, , , , Represents the attention function. For a learnable parameter matrix, The query parameter matrix representing text features. The key parameter matrix representing text features, The matrix of values ​​representing text features The query parameter matrix representing visual features. Key parameter matrix representing visual features, The matrix of values ​​representing visual features. Representing text features, This represents the features of the aligned text. Indicates visual characteristics, This represents the visual features after alignment. This indicates the multi-head attention mechanism module. This indicates a matrix concatenation operation.

[0034] Step S2 constructs a multi-view cross-alignment representation learning module for visual feature extraction: the pre-trained model MAE is used to extract visual features of the news; text feature extraction: the pre-trained model BERT is used to extract text features of the news; multimodal feature extraction: the pre-trained model CLIP is used to encode news images and text simultaneously to extract multimodal features; cross-modal cross-attention alignment: cross-attention is used to perform cross-modal alignment between image and text modalities.

[0035] Specifically, step S3 is as follows: The comment features of the comment information are extracted using a fourth pre-trained model; A mask module for interactive fusion of comments is set up, and the comment features are interactively fused with aligned text features and aligned visual features based on the mask module. A hierarchical comment guidance fusion mechanism is set up, which treats the mask module as a sub-module. By stacking the sub-modules layer by layer, deep semantic fusion is performed between the comment features and the visual features, and between the comment features and the text features, to obtain the fused text features and fused visual features for comment guidance.

[0036] Comment feature extraction uses the pre-trained model BERT to extract features of generated comments.

[0037] Taking the comment and text fusion branch as an example, the masking module specifically includes: ; ; ; ; ; in, For the text features after interactive fusion, For normalization layer, It is a feedforward neural network. These are intermediate features in the fusion process; in, , Representing text features, For the mask matrix, This indicates element-wise multiplication. For the Softmax function, For query vector, For key vectors, For value vectors, For feature dimension, , To query the parameter matrix, For the key parameter matrix, For the value parameter matrix, Indicates comment characteristics, For text feature dimensions, For threshold coefficient, For the Sigmoid function, It is a multilayer perceptron. The comment features are after average pooling. For the threshold, For the number of comments, Text lexical units Comments similarity, Text lexical units Comments The mask, This is a similarity matrix between text and comments.

[0038] The aforementioned mask Transformer is treated as a sub-module, and deep semantic fusion is performed through a layer-by-layer stacking approach. The hierarchical comment guidance fusion mechanism is specifically as follows: ; ; in, For the After layer stacking Layered text representation It is a hierarchical stacked function. for Layered text representation As a comment feature, For the After layer stacking Layer fusion image representation, for Layer fusion image representation.

[0039] The principles of the comment and image fusion branches are the same as those of the comment and text fusion branches, and will not be elaborated here.

[0040] Specifically, step S4 is as follows: Step S41: Introduce learnable domain embeddings, assign a domain expert to each domain, and simultaneously set up a domain gating network to construct the hybrid expert network; calculate the importance weights of the representation dimensions for each perspective, aggregate the importance weights with the original features of the corresponding perspective, and concatenate them with the domain embeddings before inputting them into the domain gating network to obtain the hybrid expert network selection weights; simultaneously input the features of each perspective into the hybrid expert network, combine the hybrid expert network selection weights to extract specific domain knowledge, and fuse cross-domain knowledge to obtain a domain-aware knowledge representation; perform one-hot encoding on news samples labeled with domain tags to obtain domain tags after label smoothing, which are used to supervise the training of the domain gating network; Step S42: Perform single-view prediction for features from various perspectives that have extracted specific domain knowledge and integrated cross-domain knowledge, and supervise the prediction by using binary cross-entropy loss to obtain the mean and variance of the single-view prediction scores. Based on the mean and variance, further adjust the feature weights within each perspective by adaptive instance normalization. Step S43: Project the features from different perspectives onto a unified dimension through a linear layer and then concatenate them. Perform cross-attention calculation on the concatenated features and the domain embedding to obtain the feature weights for each perspective. Based on the feature weights, perform weighted concatenation of the features from each perspective to obtain aggregated features. Step S44: Input the aggregated features into a multilayer perceptron for classification to obtain the predicted labels of the news, and determine the authenticity of the news based on the predicted labels.

[0041] Step S4 constructs a domain-adaptive knowledge collaborative refinement module, including a domain-aware hybrid expert: A learnable domain embedding is introduced, and a domain expert is assigned to each domain. This enables domain-specific knowledge extraction and cross-domain knowledge fusion based on news domain information. For each modality, the importance weights of the token dimension are first calculated, then aggregated with the original features, and concatenated with the domain embedding before being input into the domain-aware hybrid expert network. Domain gating supervision: For news samples with domain labels, it can be one-hot encoded as… Domain labels are obtained after label smoothing. Single-view prediction and adaptive instance normalization: For features extracted from various perspectives that incorporate specific domain knowledge and cross-domain knowledge, single-view prediction is performed. Supervision is provided through binary cross-entropy loss, and the mean and variance of the single-view prediction scores are obtained. Adaptive instance normalization further adjusts the feature weights within each perspective. Domain-aware feature aggregation: First, features from different perspectives are projected onto a unified dimension through a linear layer, then concatenated together. Cross-attention calculation is performed with domain embeddings to obtain the feature weights for each perspective. Finally, a weighted concatenation is performed to obtain the final aggregated features. Final true / false news classification: The aggregated features are input into a multilayer perceptron for classification, predicting the true / false labels of news articles. Specifically, in step S41, the importance weight of the representation dimension is calculated for each viewpoint. This importance weight is then aggregated with the original features of the corresponding viewpoint and concatenated with the domain embedding before being input into the domain gating network to obtain the hybrid expert network selection weights. Simultaneously, the original features of each viewpoint are input into the hybrid expert network, and the domain-aware knowledge representation is calculated by combining the hybrid expert network selection weights. Specifically: in, The output is a domain-aware knowledge representation. For domain-gated networks, Represents the primitive features of a mode. , To integrate text representations, To fuse image representations, For the multimodal features, Representing modes Importance weights This indicates the domain embedding. Indicates splicing, For domain expert network; In step S41, the news samples labeled with domain tags are subjected to one-hot encoding to obtain domain tags after tag smoothing, specifically as follows: ; in, For smoothed domain labels, The domain labels before smoothing. For smoothing coefficients, For the number of domains.

[0042] The domain-gated network Training is performed using a loss function, which is: in, For domain gating loss, To smooth out labels in the news field, Output for the domain-gated network.

[0043] Specifically, step S42 includes: Single-view prediction is performed on features from various perspectives that have extracted specific domain knowledge and integrated cross-domain knowledge; the visual perspective prediction is as follows: ; in, The true / false prediction score represents the visual perspective. This represents a two-layer multilayer perceptron. Knowledge representation for the visual perspective domain; Supervision is performed using binary cross-entropy loss, and the mean and variance of the single-view prediction scores are calculated using two specific multilayer perceptrons: ; ; in, For the loss of true / false classification from a visual perspective, As a label for the authenticity of news, The true / false prediction score is based on the visual perspective. To predict the mean of the scores, For multilayer perceptrons used to calculate the mean, To predict the variance of the scores, For use in calculating variance, a multilayer perceptron; Based on the mean and variance, the feature weights within each viewpoint are further adjusted through adaptive instance normalization: ; in, The features are normalized from the visual perspective. Knowledge representation for the visual perspective domain. The mean of knowledge representation for the visual perspective domain. Variance of knowledge representation for the visual perspective domain.

[0044] The prediction method from the text perspective is the same as that from the visual perspective, and will not be repeated here.

[0045] Specifically, step S43 is as follows: Domain-aware features from different perspectives are projected onto a unified dimension through a linear layer and then concatenated to obtain concatenated features: ; in, As a feature in the visual perspective. For visual perspective, linear layer The features are normalized from the visual perspective. This represents an intermediate feature from a textual perspective. A linear layer from a textual perspective. The normalized features of the text perspective For intermediate features from a multimodal perspective, For multimodal view linear layers, These are the normalized features from a multimodal perspective; The concatenated features are then subjected to cross-attention calculation with the domain embedding to obtain the feature weights for each viewpoint: ; in, For feature weights, For the Softmax function, Embedding for learnable domains, Indicates matrix concatenation. Indicates feature dimension; Based on the aforementioned feature weights, the features from each perspective are weighted and concatenated to obtain aggregated features: ; in, As an aggregation feature, For the feature weights of the visual perspective, For the feature weights from the text perspective, The feature weights are from a multimodal perspective.

[0046] Specifically, step S44 is as follows: ; ; in, For predicting labels, For actual labels, As an aggregation feature, The final true or false prediction loss.

[0047] Specifically, the final true / false prediction loss, the domain gating supervision loss, and the single-view prediction loss are combined as the overall loss function of the domain-aware hybrid expert network for training: ; in, For the overall loss function, To predict the final true or false loss, For single-view prediction of loss, For the loss of domain gate monitoring, The weights for the single-view prediction loss, The weight of the domain gating supervision loss; Training was performed using the AdamW optimizer with an initial learning rate of 1e-4, and cosine annealing was used to restart and adjust the learning rate.

[0048] This method was validated on two standard datasets, Weibo and Weibo-21. Experimental results show that this method outperforms existing techniques in both detection accuracy and generalization performance. Details are shown in Table 1.

[0049] Table 1. Results of the verification experiment In summary, this embodiment has the following technical effects: High detection accuracy: By introducing a large visual language model and a large language model to collaboratively generate multimodal user comment information, the problem of missing or insufficiently utilized comment information in traditional methods is overcome. Simultaneously, news features are extracted from three perspectives—image, text, and multimodal—and combined with a hierarchical comment guidance fusion mechanism, deep semantic fusion of comment information and original content is achieved, thus more comprehensively capturing false clues and significantly improving the accuracy of false information identification.

[0050] High robustness: A cross-modal semantic alignment and fusion module is designed to effectively address the heterogeneity problem among multimodal data and improve the model's stability under interference such as complex forgery methods and inconsistent modal information. In addition, by introducing a domain-adaptive knowledge collaborative refinement mechanism, the system can maintain good discrimination ability under different noise environments and input perturbations, thus enhancing the overall robustness of the system.

[0051] High versatility: It adopts a domain-aware hybrid expert network and attention module, which can dynamically adjust the feature extraction and fusion strategy according to the domain of the input sample. It can achieve adaptive detection in multiple domains such as politics, science and technology, entertainment and different language styles, effectively solving the problem of insufficient generalization ability of existing models, and has strong versatility and cross-domain transfer ability.

[0052] Highly scalable: The overall architecture features a well-designed modular structure with low coupling between functional components, facilitating the flexible integration of new modalities such as audio and video, and enabling the introduction of more powerful pre-trained models or expansion of the user attribute library. Furthermore, the model structure supports incremental learning and online updates, enabling rapid adaptation to new modes of misinformation dissemination and demonstrating strong sustainability and system scalability.

[0053] Example 2 This embodiment provides a multimodal domain-adaptive fake information detection device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, it implements the multimodal domain-adaptive fake information detection method described in Embodiment 1.

[0054] The multimodal domain adaptive fake information detection device provided in this embodiment is used to implement the multimodal domain adaptive fake information detection method. Therefore, the multimodal domain adaptive fake information detection device also possesses the technical effects of the multimodal domain adaptive fake information detection method, and will not be described again here.

[0055] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of the present invention.

Claims

1. A multimodal domain adaptive method for detecting false information, characterized in that, Includes the following steps: Step S1: Based on the large visual language model and the large language model, combine the text and image parts of the news sample to generate multimodal perception comment information; Step S2: Extract features from the news sample from three perspectives: image, text, and multimodal, to obtain visual features, text features, and multimodal features; Step S3: Extract the comment features of the comment information, and set up a hierarchical comment guidance fusion mechanism to fuse the comment features with the text features and the visual features respectively; Step S4: Construct a hybrid expert network to extract and fuse specific domain knowledge and other cross-domain knowledge of the news sample to obtain a domain-aware knowledge representation of the news sample; perform single-view prediction based on features from each perspective and perform adaptive instance normalization; and weight and concatenate the features from different perspectives to obtain aggregated features. The aggregated features are input into a multilayer perceptron for classification to obtain the classification and detection results.

2. The multimodal domain adaptive false information detection method according to claim 1, characterized in that, Step S1 specifically involves: A large visual language model is used to describe the image portion of the news sample, generating a visual description corresponding to the image portion; By combining a pre-defined user attribute library, a large language model is used to combine the text portion with the visual description to generate multimodal perceived comment information.

3. The multimodal domain adaptive false information detection method according to claim 1, characterized in that, Step S2 specifically involves: The first pre-trained model is used to extract visual features of the image portion; the second pre-trained model is used to extract text features of the text portion; cross-attention is used to perform cross-modal alignment of the visual features and the text features; and the third pre-trained model is used to simultaneously encode the image portion and the text portion to extract multimodal features. Cross-modal alignment of the visual features and the text features is performed using cross-attention, specifically as follows: ； ； ； ； in, The first, representing the textual perspective One's attention, The first representing the visual perspective One's attention, Represents the query vector. Represents the key vector. Represents a value vector, in calculation hour, , , In calculation hour, , , , Represents the attention function. For a learnable parameter matrix, The query parameter matrix representing text features. The key parameter matrix representing text features, The matrix of values ​​representing text features The query parameter matrix representing visual features. Key parameter matrix representing visual features, The matrix of values ​​representing visual features. Representing text features, This represents the features of the aligned text. Indicates visual characteristics, This represents the visual features after alignment. This indicates the multi-head attention mechanism module. This indicates a matrix concatenation operation.

4. The multimodal domain adaptive false information detection method according to claim 1, characterized in that, Step S3 specifically involves: The comment features of the comment information are extracted using a fourth pre-trained model; A mask module for interactive fusion of comments is set up, and the comment features are interactively fused with aligned text features and aligned visual features based on the mask module. A hierarchical comment guidance fusion mechanism is set up, which treats the mask module as a sub-module. By stacking the sub-modules layer by layer, deep semantic fusion is performed between the comment features and the visual features, and between the comment features and the text features, to obtain the fused text features and fused visual features for comment guidance.

5. The multimodal domain adaptive false information detection method according to claim 1, characterized in that, Step S4 specifically involves: Step S41: Introduce learnable domain embeddings, assign a domain expert to each domain, and simultaneously set up a domain gating network to construct the hybrid expert network; calculate the importance weights of the representation dimensions for each perspective, aggregate the importance weights with the original features of the corresponding perspective, and concatenate them with the domain embeddings before inputting them into the domain gating network to obtain the hybrid expert network selection weights; simultaneously input the features of each perspective into the hybrid expert network, combine the hybrid expert network selection weights to extract specific domain knowledge, and fuse cross-domain knowledge to obtain a domain-aware knowledge representation; perform one-hot encoding on news samples labeled with domain tags to obtain domain tags after label smoothing, and use the domain tags to supervise the training of the domain gating network; Step S42: Perform single-view prediction for features from various perspectives that have extracted specific domain knowledge and integrated cross-domain knowledge, and supervise the prediction by using binary cross-entropy loss to obtain the mean and variance of the single-view prediction scores. Based on the mean and variance, further adjust the feature weights within each perspective by adaptive instance normalization. Step S43: Project the features from different perspectives onto a unified dimension through a linear layer and then concatenate them. Perform cross-attention calculation on the concatenated features and the domain embedding to obtain the feature weights for each perspective. Based on the feature weights, perform weighted concatenation of the features from each perspective to obtain aggregated features. Step S44: Input the aggregated features into a multilayer perceptron for classification to obtain the predicted labels of the news, and determine the authenticity of the news based on the predicted labels.

6. The multimodal domain adaptive false information detection method according to claim 1, characterized in that, In step S41, importance weights for the representation dimensions are calculated for each viewpoint. These importance weights are then aggregated with the original features of the corresponding viewpoint and concatenated with the domain embedding before being input into the domain gating network to obtain the hybrid expert network selection weights. Simultaneously, the original features of each viewpoint are input into the hybrid expert network, and the domain-aware knowledge representation is calculated by combining the hybrid expert network selection weights. Specifically: in, The output is a domain-aware knowledge representation. For domain-gated networks, Represents the primitive features of a mode. , To integrate text representations, To fuse image representations, For the multimodal features, Representing modes Importance weights This indicates the domain embedding. Indicates splicing, For domain expert network; In step S41, the news samples labeled with domain tags are subjected to one-hot encoding to obtain domain tags after tag smoothing, specifically as follows: ； in, For smoothed domain labels, The domain labels before smoothing. For smoothing coefficients, For the number of domains.

7. The multimodal domain adaptive false information detection method according to claim 1, characterized in that, Step S42 specifically includes: Single-view prediction is performed on features from various perspectives that have extracted specific domain knowledge and integrated cross-domain knowledge; the visual perspective prediction is as follows: ； in, The true / false prediction score represents the visual perspective. This represents a two-layer multilayer perceptron. Knowledge representation for the visual perspective domain; Supervision is performed using binary cross-entropy loss, and the mean and variance of the single-view prediction scores are calculated using two specific multilayer perceptrons: ； ； in, For the loss of true / false classification from a visual perspective, As a label for the authenticity of news, The true / false prediction score is based on the visual perspective. To predict the mean of the scores, For multilayer perceptrons used to calculate the mean, To predict the variance of the scores, For use in calculating variance, a multilayer perceptron; Based on the mean and variance, the feature weights within each viewpoint are further adjusted through adaptive instance normalization: ； in, The features are normalized from the visual perspective. Knowledge representation for the visual perspective domain. The mean of knowledge representation for the visual perspective domain. Variance of knowledge representation for the visual perspective domain.

8. The multimodal domain adaptive false information detection method according to claim 1, characterized in that, Step S43 specifically involves: Domain-aware features from different perspectives are projected onto a unified dimension through a linear layer and then concatenated to obtain concatenated features: ； in, As a feature in the visual perspective. For visual perspective, linear layer The features are normalized from the visual perspective. This represents an intermediate feature from a textual perspective. A linear layer from a textual perspective. The normalized features of the text perspective For intermediate features from a multimodal perspective, For multimodal view linear layers, These are the normalized features from a multimodal perspective; The concatenated features are then subjected to cross-attention calculation with the domain embedding to obtain the feature weights for each viewpoint: ； in, For feature weights, For the Softmax function, Embedding for learnable domains, Indicates matrix concatenation. Indicates feature dimension; Based on the aforementioned feature weights, the features from each perspective are weighted and concatenated to obtain aggregated features: ； in, As an aggregation feature, For the feature weights of the visual perspective, For the feature weights from the text perspective, The feature weights are from a multimodal perspective.

9. The multimodal domain adaptive false information detection method according to claim 1, characterized in that, Step S44 specifically involves: ； ； in, For predicting labels, For actual labels, As an aggregation feature, The final true or false prediction loss.

10. A multimodal domain adaptive fake information detection device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the multimodal domain-adaptive fake information detection method as described in any one of claims 1-9.

Images