Multimodal fake information detection methods, devices, and equipment based on geometric differences

Abstract

This application relates to a method, apparatus, and device for multimodal misinformation detection based on geometric dissimilarity. The method models textual and visual features as probability distributions to capture modality-specific uncertainties; and achieves multimodal fusion supporting uncertainty perception by quantifying cross-modal inconsistencies on a statistical manifold through geometric dissimilarity quantification based on an improved Fisher-Rao distance. The method has demonstrated robustness and effectiveness in detecting fake news on multiple social media platforms.

Description

Technical Field

[0001] This invention belongs to the field of information detection technology, and relates to a method, apparatus and equipment for detecting multimodal fake information based on geometric differences. Background Technology

[0002] With the rapid development of the internet and social media platforms, barriers to information dissemination have been significantly reduced, allowing people to easily access and share information. However, inadequate regulation has exacerbated the large-scale spread of fake news through these channels, which can mislead public opinion. Therefore, automatic misinformation detection (MD) has become a key research topic in the field of Natural Language Processing (NLP). This task aims to efficiently identify textual or multimodal false content that contradicts objective facts, while reducing the cost and workload of manual annotation.

[0003] Existing fake information detection methods can be categorized into unimodal and multimodal methods based on the input modality. Early unimodal methods employed traditional machine learning techniques and trained shallow classifiers for accuracy evaluation. Subsequently, deep learning models were introduced for text data processing. However, relying solely on text features limited further improvements in detection accuracy. The introduction of visual modalities revealed the crucial role of visual features in multimodal detection, thus driving the development of multimodal methods. These methods can be broadly classified into early fusion, late fusion, and hybrid fusion based on their fusion strategies. Late fusion is widely adopted due to its simple structure and ease of implementation. In contrast, early fusion integrates features from a specific modality in the initial stage and then performs shared processing to learn a joint representation. Intermediate strategies such as mid-process fusion and intermediate fusion are collectively referred to as hybrid fusion in this work.

[0004] The Fisher-Rao distance defines a principled Riemannian metric on the statistical manifold of the probability distribution, differing from the Kullback-Leibler divergence in three aspects. The Fisher-Rao distance is invariant under reparameterization, remains sensitive to both the mean and covariance of the distribution, and fully conforms to all the axioms of an appropriate metric. For a unimodal Gaussian distribution, the resulting Fisher-Rao metric, with its closed-form expression determined by the difference between the mean and the log-variance, is computationally feasible in deep learning. In the proposed framework, textual and visual features are explicitly modeled as diagonal Gaussian distributions in a shared latent space, and cross-modal inconsistencies are quantified by the geometric differences derived from this metric. This method inherently incorporates modality-specific uncertainties during the fusion process, overcoming the limitations of traditional methods. Figure 1 The limitations of the Euclidean distance are illustrated. These limitations are explained by a univariate normal distribution, where the mean... Standard deviation Four univariate normal distributions were compared, each consisting of a pair of... Unique parameterization. Figure 1 (a) and Figure 1 In (b), gap1 and gap2 show that the statistical difference between the distributions corresponding to C and D is smaller than the statistical difference between A and B. Figure 1 Euclidean metrics are applied in the parameter space of (c), but this application fails to faithfully reflect the statistical differences mentioned above. This shortcoming highlights the necessity of geometrically perceptive distance metrics for distribution comparisons. Summary of the Invention

[0005] To address the problems existing in the above-mentioned traditional methods, this invention proposes a multimodal misinformation detection method, apparatus, and device based on geometric differences. This method models text and visual features as probability distributions to capture modality-specific uncertainties; and on the statistical manifold, it quantifies cross-modal inconsistencies based on geometric differences using an improved Fisher-Rao distance, thereby achieving multimodal fusion that supports uncertainty perception.

[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: On the one hand, a multimodal fake information detection method based on geometric differences is provided, including the following steps: Step 1: Input the acquired raw image into the visual semantic parsing module, and use a large visual language model to extract entity information and image description information; Step 2: The original image, corresponding original text, entity information and image description information are processed using a single-modal encoder module to obtain visual embedding vector, text embedding vector, entity embedding vector and description embedding vector; Step 3: Input the visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector into the cross-modal attention module. Use a multi-head attention mechanism to establish fine-grained alignment between the text embedding vector and other embedding vectors to generate cross-modal embedding vectors. Step 4: Input the visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector into the geometric difference module. They are modeled as a diagonal Gaussian distribution in the shared latent space and obtained as geometric uncertainty embedding vectors by improving the geometric difference measure of Fisher-Rao distance on the statistical manifold. Step 5: Input the cross-modal embedding vector and the geometric uncertainty embedding vector into the prediction module to obtain the false information detection results.

[0007] On the other hand, a multimodal fake information detection device based on geometric differences is also provided, comprising: The visual semantic parsing unit is used to input the acquired raw image into the visual semantic parsing module, and to extract entity information and image description information using a large visual language model.

[0008] The single-modal embedding vector determination unit is used to process the original image, corresponding original text, entity information and image description information using the single-modal encoder module to obtain visual embedding vector, text embedding vector, entity embedding vector and description embedding vector.

[0009] The cross-modal embedding vector determination unit is used to input visual embedding vectors, text embedding vectors, entity embedding vectors and description embedding vectors into the cross-modal attention module. It uses a multi-head attention mechanism to establish fine-grained alignment relationships between text embedding vectors and other embedding vectors to generate cross-modal embedding vectors.

[0010] The geometric uncertainty embedding vector determination unit is used to input visual embedding vectors, text embedding vectors, entity embedding vectors, and description embedding vectors into the geometric difference module. They are modeled as diagonal Gaussian distributions in a shared latent space and the geometric uncertainty embedding vectors are obtained on the statistical manifold by a geometric difference metric based on an improved Fisher-Rao distance.

[0011] The false information detection result unit is used to input cross-modal embedding vectors and geometric uncertainty embedding vectors into the prediction module to obtain false information detection results.

[0012] In another aspect, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above-described multimodal spoofing detection methods based on geometric differences.

[0013] One of the above technical solutions has the following advantages and beneficial effects: The aforementioned multimodal misinformation detection method, apparatus, and device based on geometric dissimilarity model the textual and visual features as probability distributions to capture modality-specific uncertainties. Furthermore, on a statistical manifold, cross-modal inconsistencies are quantified by geometric dissimilarity based on an improved Fisher-Rao distance, achieving multimodal fusion that supports uncertainty perception. This method has demonstrated robustness and effectiveness in detecting fake news on multiple social media platforms. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of a normalized univariate normal distribution in the data space and parameter space in the prior art, where... Figure 1 Figure (a) shows the probability density function curves of two normal distributions, A and B. Figure 1 Figure (b) shows the probability density function curves of two normal distributions, C and D. Figure 1 (c) represents four normal distributions. A schematic diagram of coordinates in parameter space; Figure 2 This is a schematic diagram of a multimodal misinformation detection method based on geometric differences in one embodiment; Figure 3 This is a schematic diagram of a multimodal fake information detection model based on geometric differences in one embodiment; Figure 4 This is a structural diagram of the internal components of a CMAM in one embodiment, wherein... Figure 4 In the middle (a), the multi-head sparse attention mechanism is shown. Figure 4 (b) is the multi-word attention submodule. Figure 4 (c) represents the alignment & normalization module. Figure 4 (d) represents the hierarchical feature fusion submodule; Figure 5 This is a structural diagram of a Gaussian encoder in a GDM in one embodiment; Figure 6 This is a schematic diagram of the statistical manifold induced by a family of Gaussian distributions in one embodiment; Figure 7 This is a schematic diagram of geometric dissimilarity during the training phase in one embodiment (Twitter dataset), where Figure 7 (a) is a schematic diagram of the average geometric difference in the Twitter dataset based on the authenticity of news articles. Figure 7 (b) is a schematic diagram of the normalized variance distribution corresponding to the round with the best accuracy on the Twitter dataset. Figure 8 This is a schematic diagram of geometric dissimilarity during the training phase in one embodiment (Weibo dataset), where Figure 8 (a) is a schematic diagram of the average geometric difference based on the authenticity of news in the Weibo dataset. Figure 8(b) is a schematic diagram of the normalized difference distribution corresponding to the round with the best accuracy on the Weibo dataset; Figure 9 Here is a visualization of the latent features learned by the model in one embodiment (Twitter dataset), where Figure 9 (a) is a visualization of the initialization phase of attention features on the Twitter dataset. Figure 9 (b) is a visualization of the attention features corresponding to the optimal accuracy round on the Twitter dataset. Figure 10 This is a visualization of the latent features learned by the model in one embodiment (Weibo dataset). Figure 10 (a) is a visualization of the initialization phase of attention features on the Weibo dataset. Figure 10 (b) is a visualization of the attention features in the round with the best accuracy on the Weibo dataset. Figure 11 In one embodiment A diagram illustrating the impact on the accuracy of the two datasets, where Figure 11 (a) is A diagram illustrating the impact on the accuracy of the Twitter dataset; Figure 11 (b) is a schematic diagram illustrating the impact on accuracy in the Weibo dataset. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0018] It should be noted that, in this document, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments. The term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, including such combinations.

[0019] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0020] In one embodiment, such as Figure 1 As shown, a multimodal fake information detection method based on geometric differences is provided. This method transforms the multimodal fake information detection problem into a geometric difference minimization problem in a structured latent space; it may include the following processing steps 1 to 5: Step 1: Input the acquired raw image into the visual semantic parsing module, and use a large visual language model to extract entity information and image description information.

[0021] Specifically, the Visual Semantic Parsing Module (VSPM) processes the original image, extracts semantic entity text, and generates visual descriptive text. Specifically, given the original input image... The visual semantic parsing module uses a pre-trained large-scale visual language model to generate two types of structured text signals; one set consists of salient entities, denoted as... The other set is the overall image description information, denoted as... .

[0022] The input data for this method is multimodal input data, including the original images and the corresponding original text (i.e., text information). The original images are the image information in the dataset, and the original images can be, but are not limited to, news images.

[0023] Step 2: The original image, corresponding original text, entity information and image description information are processed using a single-modal encoder module to obtain visual embedding vector, text embedding vector, entity embedding vector and description embedding vector.

[0024] Specifically, the Unimodal Encoder Module (UEM) is used to extract fixed-dimensional feature representations from visual and text inputs using a dedicated unimodal encoder. Specifically, the unimodal encoder module processes the original image and various types of text (including the corresponding original text, entity information, and image description information) into compact, modality-specific feature representations.

[0025] Step 3: Input the visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector into the cross-modal attention module. Use a multi-head attention mechanism to establish fine-grained alignment between the text embedding vector and other embedding vectors to generate cross-modal embedding vectors.

[0026] Specifically, the Cross-Modal Attention Module (CMAM) uses a multi-head attention mechanism to establish fine-grained alignment between visual and textual features, generating a fused semantic representation.

[0027] Step 4: Input the visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector into the geometric difference module. They are modeled as a diagonal Gaussian distribution in the shared latent space and obtained as geometric uncertainty embedding vectors on the statistical manifold by using a geometric difference measure based on the improved Fisher-Rao distance.

[0028] Specifically, this method abandons the traditional approach of treating feature representations as deterministic point estimates and instead adopts a probabilistic perspective. In the shared latent space, each unimodal feature is modeled as an independent diagonal Gaussian distribution.

[0029] This step models textual and visual features as probability distributions on a shared statistical manifold to directly quantify cross-modal inconsistency via a geometric difference metric. Cross-modal inconsistency is naturally described as the geometric difference between the corresponding distributions.

[0030] The core component of the geometric discrepancy module is a custom geometric discrepancy metric derived from a local approximation of the Fisher-Rao metric. This metric serves a dual purpose: first, as an auxiliary regularization term during training; and second, as an uncertainty-aware signal for the final truth prediction. This dual design results in a more robust and interpretable multimodal fusion mechanism.

[0031] The Geometric Dissimilarity Module (GDM) is used to independently model the unimodal feature representations as diagonal Gaussian distributions in a shared latent space and to compute the geometric dissimilarity between source and target modal correspondences for realism assessment.

[0032] Step 5: Input the cross-modal embedding vector and the geometric uncertainty embedding vector into the prediction module to obtain the false information detection results.

[0033] Specifically, the final authenticity prediction integrates complementary signals from the cross-modal attention module and the geometric difference module, and incorporates the cross-modal embedding vector. Embedded vectors with geometric uncertainty By concatenating the components, we obtain the union representation as follows:

[0034] in, For joint representation, These are the dimensions of the cross-modal embedding vector and the geometric uncertainty embedding vector, respectively.

[0035] The joint representation z is mapped to the log odds of binary classification using a two-dimensional linear classifier.

[0036] A multimodal fake information detection model based on geometric dissimilarity includes: a visual semantic parsing module, a unimodal encoder module, a cross-modal attention module, a geometric dissimilarity module, and a prediction module. The structure of the geometric dissimilarity-based multimodal fake information detection model is as follows: Figure 3 As shown, this model is an end-to-end framework called GeoFake. It models unimodal features as a diagonal Gaussian distribution, enriches visual input with semantic cues, achieves fine-grained cross-modal semantic alignment, and uses a Fisher-Rao distance-inspired geometric dissimilarity metric as a regularization term in the training process and an interpretability evaluation indicator for multimodal consistency. In the geometric dissimilarity-based multimodal fake information detection model: a large visual language model (LVLM) is used to generate image description information from the original image and extract structured semantic entities from the image. These signals, along with the original text and the original image, constitute one visual stream and three text streams. Each stream is encoded into compact, modality-specific features through a dedicated unimodal encoder module (UEM). To capture fine-grained cross-modal semantic correspondences, a cross-modal attention module (CMAM) is introduced. A multi-head attention mechanism is used to perform bidirectional feature optimization, generating a fused representation that preserves both intramodal and intermodal dependencies. The Geometric Dissimilarity Module (GDM) computes a customized geometric dissimilarity metric to quantify intra-sample multimodal inconsistencies between the text and visual modalities. The scalar dissimilarity scores are averaged to obtain the geometric consistency loss, which is jointly optimized with the main classification loss for model training.

[0037] The aforementioned multimodal fake news detection method based on geometric dissimilarity models textual and visual features as probability distributions to capture modality-specific uncertainties. Furthermore, it quantifies cross-modal inconsistencies on a statistical manifold by improving the geometric dissimilarity of the Fisher-Rao distance, thus achieving multimodal fusion that supports uncertainty perception. This method has demonstrated its robustness and effectiveness in detecting fake news on multiple social media platforms.

[0038] In one embodiment, the large-scale visual language model is the Tongyi Qianwen 2.5-VL-72B model.

[0039] Specifically, by inputting task-specific instructions into a large visual language model, it outputs results in a standard format: .in, This is a noun phrase representing the original image. Objects or scenes that have semantic meaning within the context; To form coherent sentences that describe the overall visual context of an image.

[0040] In this embodiment, the visual semantic parsing module uses the Tongyi Qianwen 2.5-VL-72B model to extract structured semantic signals from the original image and text input. During training, the parameters of this large-scale visual language model are frozen to ensure the stability of semantic extraction and prevent catastrophic forgetting of its general visual language knowledge.

[0041] In one embodiment, the unimodal encoder module includes an image encoder and three text encoders; step 2 includes: inputting the original image into the image encoder, processing it using a pre-trained VGG-19 network with the final classification layer removed, and obtaining a visual embedding vector; inputting the original text into the first text encoder, encoding it using a BERT model, and obtaining a text embedding vector; inputting entity information into the second text encoder, encoding it using a BERT model, and obtaining an entity embedding vector; and inputting image description information into the third text encoder, encoding it using a BERT model, and obtaining a description embedding vector.

[0042] Specifically, in the single-modal encoder module, for the visual modality, a pre-trained VGG-19 network with the final classification layer removed is used: features are extracted from the output of the second fully connected layer to obtain a 4096-dimensional embedding vector for each image. The input image is resized to 224×224, normalized using the mean and standard deviation of the ImageNet dataset, and then input into the network to obtain the visual embedding vector. For the text modality, the BERT model is used to encode the input sentence. For sequences of up to 60 words, mean pooling is performed on the word-level output along the sequence dimension to generate a sentence-level representation. The same pre-trained language model is used to encode the original input text, image descriptions, and entities, respectively, resulting in text embedding vectors of dimension 768. , and These unimodal embedding vectors form the basis for subsequent cross-modal interaction and geometric difference modeling.

[0043] In one embodiment, the cross-modal attention module includes: an alignment and normalization module, two multi-head sparse attention mechanisms, two multi-word attention sub-modules, a hierarchical feature fusion sub-module, and a fully connected layer; step 3 includes: processing the visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector respectively using the alignment and normalization module to obtain aligned visual embedding vectors, text embedding vectors, entity embedding vectors, and description embedding vectors; the alignment and normalization module is used to project the input features to a shared latent space of a preset dimension through linear transformation, and then perform layer normalization to obtain aligned representations; cross-modal bidirectional attention calculation and hierarchical feature fusion are performed on the text-visual, text-description, and text-entity modal pairs respectively to obtain text-visual feature vectors, visual-text feature vectors, text-entity feature vectors, entity-text feature vectors, text-description feature vectors, and description-text feature vectors; the text-visual feature vectors, visual-text feature vectors, and text-entity feature vectors are then combined. The concatenated feature vectors, entity-text feature vectors, text-description feature vectors, and description-text feature vectors are input into a fully connected layer to obtain cross-modal embedding vectors. The cross-modal bidirectional attention calculation and hierarchical feature fusion process for the text-visual modality pair includes: using a multi-head sparse attention mechanism to independently calculate the query matrix, key matrix, and value matrix for the aligned visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector; inputting the query matrix of the text embedding vector and the key matrix and value matrix of the visual embedding vector into the first multi-terminal attention submodule to obtain the positive cross-modal interaction result; inputting the query matrix of the visual embedding vector and the key matrix and value matrix of the text embedding vector into the second multi-terminal attention submodule to obtain the negative cross-modal interaction result; and inputting the concatenated positive bidirectional cross-modal interaction result and the negative cross-modal interaction result into the hierarchical feature fusion submodule to obtain the text-visual feature vector and the visual-text feature vector.

[0044] Specifically, the cross-modal attention module enhances the semantic coherence between unimodal representations through structured pairwise interactions. This is achieved for the four embedding vectors generated by the unimodal encoder module. , , and First, each vector is processed by the Alignment and Normalization (A&N) module. Specifically, the single-modal encoder module projects the features to the dimension through a linear transformation. The shared latent space is then subjected to layer normalization to obtain the aligned representation, denoted as follows: , , , Subsequently, the multi-head sparse attention (MHS) mechanism is applied to each modality. Independently compute the query matrix Key matrix Sum matrix The specific calculation formula is as follows:

[0045] Where c represents the mode The input feature vector; , , It is a learnable linear projection matrix; This is a binary mask for elements with values ​​{0, 1}.

[0046] The cross-modal attention module does not perform full pairwise cross-modal attention computation. Instead, it restricts the interaction to three semantic-based modal pairs: text-visual, text-description, and text-entity. This design stems from the key fact that both descriptive text and entities are extracted from the same visual input. Directly aligning them with the text modality enables more reasonable and concise feature fusion. For each modal pair, the multi-word attention (MTA) submodule computes bidirectional attention outputs:

[0047] in, , These are the positive cross-modal attention output and the negative cross-modal attention output, respectively. The features of mode B are obtained by calculating the features using a multi-head sparse attention mechanism. , and matrix, The feature information of mode A is calculated using a multi-head sparse attention mechanism. , and The matrix, MTA stands for Multi-Word Attention Mechanism.

[0048] In the Multi-Word Attention (MTA) submodule, the similarity between the query vector and the key vector is first calculated, and then the value vector is aggregated based on this similarity to achieve compact cross-modal interaction. To enhance context modeling capabilities, a lightweight depthwise separable convolution is introduced, and the attention logarithm is optimized before aggregation.

[0049] To fully utilize the bidirectional cross-modal interaction results and their joint representation, a hierarchical feature fusion (HFF) submodule is proposed. Specifically, for two optimized features... and The hierarchical feature fusion submodule first obtains the fused representation through concatenation. Then, through hierarchical gating and residual fusion mechanisms, the triplet is... By performing joint processing to capture complementary information at different levels of abstraction, six optimized feature vectors are finally obtained. , , , , , These vectors are concatenated along the feature dimension and then fed into a fully connected layer to generate the final cross-modal embedding vector. .

[0050] The internal modular structure of CMAM is as follows: Figure 4 As shown, where Figure 4 In the middle (a), the multi-head sparse attention mechanism is shown. Figure 4 (b) is the multi-word attention submodule. Figure 4 (c) represents the alignment & normalization module. Figure 4 In the middle (d), there is a hierarchical feature fusion submodule.

[0051] In one embodiment, the geometric difference module includes a Gaussian encoder and a geometric difference encoder; step 4 includes: projecting the visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector onto a shared latent space of a preset dimension and then inputting them into the Gaussian encoder, representing each modality as a diagonal Gaussian distribution; inputting the diagonal Gaussian distributions of all modalities into the geometric difference encoder, and using a geometric difference metric method based on the improved Fisher-Rao distance to calculate the pairwise geometric differences between the text modality and the other three modalities, obtaining three scalar differences and three vector differences; concatenating the three vector differences along the feature dimension and then processing them using a fully connected layer to obtain the geometric uncertainty embedding vector.

[0052] In one embodiment, in the Gaussian encoder: the input features of the input Gaussian encoder are processed by a fully connected layer to obtain the mean. The mean is processed through a fully connected layer, layer regularization, and the GELU function. The result is then processed again through a fully connected layer to obtain the log-variance. .

[0053] In one embodiment, the geometric difference measure based on Fisher-Rao distance specifically includes defining the expression for the vector difference measure between the diagonal Gaussian distributions of two modes as follows:

[0054] in, It is a measure of the difference between the diagonal Gaussian distributions of two modes. The mean of the diagonal Gaussian distribution corresponding to the first mode. The Dimensional data, The diagonal Gaussian distribution corresponding to the second mode. mean The Dimensional data, The standard deviation of the diagonal Gaussian distribution corresponding to the first mode. The Dimensional data, The standard deviation of the diagonal Gaussian distribution corresponding to the second mode. The Dimensional data, , , Let be the dimension of the diagonal Gaussian distribution corresponding to the two modes.

[0055] Based on the vector difference metric between the two modes, the scalar difference between the two modes is determined as follows:

[0056] in, This is a scalar difference between two modes, used to represent the overall degree of cross-modal inconsistency; As a dimension, express The k dimension.

[0057] Specifically, this method defines the Fisher-Rao distance on the statistical manifold of a multivariate Gaussian distribution. The probability density function of a multivariate normal distribution The definition is as follows:

[0058] in, A vector of variables; It is the mean vector; Let covariance matrix be the variance matrix. express n×n The set of positive definite symmetric matrices. The set of all such distributions constitutes a statistical model. The model is a smooth manifold with dimension . According to Burbea's research findings, for any , express n non-singular matrix space of order, mapping It is a statistical manifold Isometric transformations on the statistical manifold. Therefore, the statistical manifold The two parameters and The Fisher-Rao distance between them satisfies the following property:

[0059] in, for and Fisher-Rao distance between the corresponding distributions. This shows that the Fisher-Rao distance is invariant under affine transformations, and its value depends only on the intrinsic statistical structure of the distribution, regardless of the choice of coordinate system.

[0060] The geometric discriminant module quantifies multimodal inconsistencies within samples through probabilistic modeling in the Riemannian hidden space. First, it models the four single-modal embedding vectors output by the single-modal encoder module. , , and Projected to dimension The shared space is then input to a Gaussian encoder, which outputs the mean value for each mode. Sum of logarithmic variance The formula is as follows:

[0061] Where GE stands for Gaussian encoder (e.g.) Figure 5 (As shown). Each mode is represented as a diagonal Gaussian distribution. The Geometric Difference Encoder (GDE) calculates the pairwise geometric differences between the text modality and the other three modalities. For each modality pair (T, )( Output the three scalar difference scores. , , and the difference scores of the three vectors , , These outputs are obtained using a metric based on the improved Fisher-Rao distance, which, under the assumption of diagonal covariance, simultaneously considers the mean difference and variance mismatch.

[0062] Three vector descriptors , and By piecing them together, we obtain Then through the output dimension as Projecting the fully connected layer, we obtain... Scalar fraction , , The proposed geometric difference metric method is described in detail below to define the geometric consistency loss.

[0063] set up For a parametric statistical model, where the parameter space As shown in Figure 7, immersion mapping statistical models Mapped to Fisher-Rao metric Riemannian structure . and The Fisher-Rao distance between the corresponding distributions can be described as:

[0064] in, Fisher-Rao distance represents the distance between two univariate Gaussian distributions, and univariate normal distribution. and Fisher-Rao distance The parsing expression:

[0065] in, Let be the standard Euclidean norm in R2 space.

[0066] According to Nielsen's research, and The exact Fisher-Rao distance between covariances does not have a closed-form expression, making it impossible to compute directly in end-to-end deep learning. To achieve engineering deployment, this study employs a local metric approximation method, which preserves the geometry of the Fisher-Rao framework while ensuring differentiability and efficiency. The standard practice is to reparameterize the covariance in logarithmic space, letting... Then line element Transformed into:

[0067] in, For line elements, n This represents the dimension of the distribution.

[0068] Assuming the Fisher metric changes slowly along the geodesic, it can be approximated by integrating it at an intermediate point. A commonly used intermediate point is the arithmetic mean of the variances at the endpoints, i.e. This leads to the classic local approximation formula:

[0069] in, d Let be the dimension of the two distributions.

[0070] Based on empirical observations, the relative weights of each term in the above formula are adjusted to improve training stability and semantic coherence while preserving the geometric interpretation. Specifically, the distribution is defined as follows: and The differences between them are measured as follows:

[0071] in, For distribution and The difference between them is measured.

[0072] The coefficients in the formula are empirically adjusted values ​​used to balance the contributions of mean difference and log-variance difference: the coefficient of the mean term is adjusted from 2 to 1, and the coefficient of the log-variance term is adjusted from 0.5 to 0.125 to ensure that semantic shift and modality-specific uncertainty have a considerable impact in the final difference measure.

[0073] To capture both fine-grained and global cross-modal differences, the distance is decomposed into vector differences and global scalar differences. Specifically, vector differences... , , The difference between the source mode and the target mode is calculated using the above formula. t , e , v , d (These correspond to text, visual, entity, and descriptive modalities, respectively). The formula for calculating global scalar difference is as follows:

[0074] in, Indicates the overall degree of cross-modal inconsistency. express The k dimension.

[0075] After the Gaussian encoder generates four diagonal Gaussian distributions in the geometric difference module, the geometric difference encoder calculates three sets of pairwise scalar differences. , , and the corresponding three sets of vector differences , , The differences between the three vectors are concatenated along the feature dimension and input into a fully connected layer to generate a compact embedding vector with uncertainty. Then combine this vector with the cross-modal embedding vector The data is pieced together to enrich the final log-odds used for prediction. A schematic diagram of the statistical manifold induced by the Gaussian distribution family is shown below. Figure 6 As shown.

[0076] In one embodiment, a multimodal misinformation detection model based on geometric dissimilarity includes: a visual semantic parsing module, a unimodal encoder module, a cross-modal attention module, a geometric dissimilarity module, and a prediction module; the overall training objective of the multimodal misinformation detection model based on geometric dissimilarity is to minimize the total loss; the total loss is a weighted sum of the main classification loss and the geometric consistency loss; the main classification loss is calculated using cross-entropy loss; the expression for the geometric consistency loss is:

[0077] in, For geometric consistency loss, For a set of cross-modal pairs, This indicates the number of cross-modal pairs in the set. These are text-entity modal pairs, text-visual modal pairs, and text-description modal pairs, respectively. and Modal pairs The average of the three scalar differences between the real news sample and the fake news sample.

[0078] Specifically, the cross-entropy loss is used to calculate the main classification loss. Simultaneously, the geometric difference scores obtained from the geometric difference module are aggregated into an auxiliary loss. This is to enhance semantic coherence. The overall training objective of the model is to minimize the weighted sum of the two losses:

[0079] in, This is a hyperparameter used to balance task-oriented detection accuracy and geometric coherence.

[0080] In a validation embodiment, experiments were conducted on two widely used multimodal fake news detection datasets: the Weibo dataset (collected from the social media platform Weibo) and the Twitter dataset (from the MediaEval multimedia validation benchmark dataset). Following standard preprocessing procedures, the data underwent targeted filtering: news samples containing videos and those without text or images were removed. Ultimately, the Weibo dataset was divided into training and test sets in an 8:2 ratio, while the Twitter dataset was divided in a 12:1 ratio. More detailed dataset statistics are shown in Table 1.

[0081] (1) Comparison with baseline model To evaluate the performance of this method, this embodiment compares it with recent state-of-the-art methods in the field of fake information detection, which are mainly divided into two paradigms: unimodal and multimodal.

[0082] Table 1. Statistical table of Weibo dataset and Twitter dataset.

[0083] (a) Single-modal method Spoofing detection is performed using only single-modal data. Specifically, BERT is used to extract text features, and VGG-19 is used to extract visual features. These features are then input into a fully connected layer, followed by a softmax activation layer to complete the scam detection. In addition, a Visual Transformer (ViT) is introduced as an additional baseline model. This model is based on a block-based pure Transformer encoder that processes the image into a 16×16 word sequence.

[0084] (b) Multimodal approach This study integrates text and visual data for fake information detection. EANN eliminates specific event bias and improves model generalization through adversarial event learning; CAFE adaptively aggregates single-modal features and cross-modal correlations, using KL divergence to quantify the ambiguity between text and visual modalities; MRML is a metric-based triplet learning framework that captures the intramodal relationship between fake and real information within each modality, while combining contrastive pairwise learning to model cross-modal interactions; AMPLE injects emotional cues through cue learning and aligns text and visual features using a cross-attention mechanism; BMR decouples multimodal inputs into three perspectives: text, image semantics, and image patterns. It utilizes single-view prediction results and cross-modal consistency to adaptively reweight and bootstrap optimize feature representations, achieving interpretable and robust fake information detection; MoPeD mines the factors for judging fake information through a dual-encoding strategy, jointly models modality-specific and modality-consistent features, and then adaptively fuses single-modal and cross-modal representations through a heterogeneity perception module. This module assigns weights to features based on the KL divergence between text and image distributions.

[0085] In addition, the experiment also introduced several recent large-scale multimodal language models as baselines, including Qwen3-VL, GLM-4V and DeepSeek-VL2.

[0086] (2) Evaluation indicators To comprehensively evaluate the performance of our method and other baseline models on the Weibo and Twitter datasets, the experiments employed a set of standard evaluation metrics, including accuracy (acc), precision (pre), recall (rec), and F1 score (f1). Accuracy reflects the overall prediction accuracy of the model; however, considering the potential class imbalance problem in misinformation detection tasks, the experiments supplemented this with class-specific metrics to address this issue: precision measures the proportion of samples predicted as belonging to a certain class that are actually belonging to that class; recall measures the proportion of actual samples belonging to a certain class that are correctly predicted; and the F1 score is the harmonic mean of precision and recall, providing a more balanced single evaluation metric when class distribution is imbalanced.

[0087] (3) Parameter settings In text processing, the word length is set to 60, and each word is mapped to a 768-dimensional word embedding space; visual features are encoded as 4096-dimensional vectors. During training, the batch size is set to 128, and the learning rate is... The hyperparameter λ was set to 1.2 on the Twitter dataset and 3.3 on the Weibo dataset. An early stopping strategy was used: training was terminated if the validation set accuracy showed no improvement for seven consecutive training epochs. A dropout rate of 0.5 was set for the regularized model. Adam was chosen as the optimizer, and [the following parameters were introduced]. Weight decay; the hidden layer dimension of the model is set to 768.

[0088] All experiments were conducted using the PyTorch 2.0.0 framework on a Linux server equipped with an Nvidia GeForce RTX 3090 graphics card. To ensure statistical reliability, all experiments were repeated 5 times using different random seeds, and the final reported results are the average of the 5 independent experiments.

[0089] (4) Experimental results and analysis (a) Performance comparison with advanced methods Experiments were conducted on two benchmark datasets, Twitter and Weibo, comparing our proposed method with various other methods. Evaluation metrics included overall accuracy, as well as category-specific precision, recall, and F1 score for both real and fake news samples. As shown in Table 2, our method achieved state-of-the-art or competitive performance on all evaluation metrics across both datasets.

[0090] Our method outperforms all comparable methods in terms of accuracy, achieving a top accuracy of 0.7187 on the Twitter dataset and 0.8597 on the Weibo dataset. Compared to the best baseline models on these two datasets, the accuracy improvements are 2.08% and 0.26%, respectively. Furthermore, our method ranks highly on almost all evaluation metrics, validating its effectiveness in the task of detecting misinformation.

[0091] While this method does not perform optimally on all individual metrics, it effectively avoids the metric trade-offs present in some comparative methods. For example, on the Twitter dataset, VGG-19 performs best in recall for the fake news category (0.7838), but its precision is significantly lower (0.6218), ultimately leading to a decrease in the F1 score (0.6920). In contrast, this method achieves a more balanced performance between precision and recall for both classes, thus obtaining a consistently competitive F1 score.

[0092] Single-modal methods (such as text-based BERT, image-based ViT, and VGG-19) perform poorly in fake information detection, primarily because they can only process information from a single modality. As shown in Table 2, the performance of text-based BERT on both datasets is consistently lower than that of the best-performing multimodal methods because it cannot utilize complementary visual input information. Similarly, ViT and VGG-19, which only process images, also yield poor experimental results. ViT's performance is further limited by insufficient training data, a known limitation of Transformer-based visual models, which require large-scale datasets to achieve effective generalization.

[0093] The difference in accuracy between VGG-19 on the Weibo dataset (0.8548) and the Twitter dataset (0.6004) reflects a significant modal bias in the model across different datasets: visual content is more discriminative on the Weibo dataset, while text input plays a dominant role in misinformation detection on the Twitter dataset. This finding is consistent with the distinct characteristics of the two platforms: Weibo content often contains rich and contextually relevant images, while Twitter posts rely more heavily on text to convey misinformation.

[0094] Multimodal methods, by fusing textual and visual inputs, generally outperform single-modal methods, highlighting the value of cross-modal information in fake information detection. However, due to inherent limitations in fusion design and contextual modeling, their performance still falls short of this method: late-stage fusion methods (such as EANN and CAFE) process each modality separately, making it difficult to achieve fine-grained alignment between text and images; hybrid fusion models also have various problems: AMPLE's sentiment-aware cues can cause predictions to favor sentimental content, ignoring neutral but genuine posts; MRML overemphasizes intramodal structure, resulting in poor cross-modal consistency in noisy social media scenarios; BMR's decoupling strategy may over-separate complementary modal signals; and while MoPeD's fusion mechanism aims to handle cross-modal heterogeneity, it is still susceptible to dominant misleading modal cues.

[0095] While large-scale multimodal language models have a massive number of parameters, they do not explicitly consider contextual factors such as the domain-specific features and rules inherent in social media platforms, which are crucial for assessing the credibility of online content. The large models evaluated in this experiment all have around 8 billion parameters, but their effectiveness in misinformation detection is limited. The core reason is that their pre-training objectives focus on general semantic alignment, rather than domain-specific reasoning tasks such as credibility assessment and narrative coherence. Furthermore, these large models directly use pre-trained native versions without architectural adaptation or task-specific fine-tuning, thus failing to address the unique challenges of misinformation detection, such as subtle cross-modal inconsistencies and position-sensitive signals.

[0096] In contrast, this method integrates multimodal semantics and contextual information within a unified architecture. It employs a probabilistic perspective to model each unimodal feature as an independent diagonal Gaussian distribution in a shared latent space (rather than traditional deterministic point estimation), capturing both the semantic content of each modality and representing its inherent uncertainty. The geometric discrepancy module quantifies the intra-sample inconsistency between visual and textual inputs by calculating a customized metric between these distributions. This discrepancy is incorporated into the geometric consistency loss and jointly optimized with the classification objective, enhancing cross-modal coherence during model training. This design allows the method to not only detect spurious information through surface signals but also identify intermodal inconsistencies, ultimately improving the model's robustness and generalization ability.

[0097] Table 2. Comparison results of our method with other baseline models on the Twitter and Weibo datasets.

[0098] (b) Ablation test To evaluate the contribution of each component in our proposed method, comprehensive ablation experiments were conducted on the Twitter and Weibo datasets, and the results are shown in Table 3. Using our proposed method as a baseline (which performs best across all metrics), we constructed ablation variants by removing core components one by one from the framework. These included the visual semantic parsing module (w / o VSPM), hierarchical feature fusion (w / o HHF) in the cross-modal attention module, multi-word attention (w / o MTA) in the cross-modal attention module, multi-head sparse attention (w / o MHS) in the cross-modal attention module, and the geometric difference module (w / o GDM). All ablation variants were trained under the same experimental conditions to isolate the effect of each module. Overall, our proposed method achieves state-of-the-art performance in both overall accuracy and F1 scores for both real and fake news samples.

[0099] The performance of all ablation variants showed a significant decline compared to this method, highlighting the necessity of each component: 1) The variant with the visual semantic parsing module removed (w / o VSPM) showed a significant performance degradation on both datasets, indicating that decomposing the raw visual input into semantic-based entities and description streams is key to establishing a meaningful cross-modal correspondence between them and textual claims; 2) Removing core components of the cross-modal attention module will continue to affect the quality of feature representation: Removing variants of hierarchical feature fusion, multi-head sparse attention, and multi-terminal attention all lead to a weakening of cross-modal fusion effect. These components enable fine-grained modal alignment, efficient attention computation of salient regions, and feature fusion at multiple semantic levels, verifying that robust multimodal reasoning requires composite attention design rather than a single attention mechanism. (3) The variant with the removal of the geometric difference module (w / o GDM) showed a significant performance drop, indicating that abandoning the probabilistic modeling of single-modal features and the associated geometric consistency optimization objective would result in the loss of a core signal that is crucial for detecting the typical subtle intra-sample inconsistencies in deceptive content.

[0100] In summary, these experimental results validate the effectiveness of each component in this method.

[0101] Table 3 Ablation study of the impact of each component of this method on the two datasets.

[0102] (c) Learning feature representation and convergence behavior analysis To evaluate the ability of this method to learn discriminative cross-modal feature representations, experiments were conducted to analyze the evolution of geometric differences and latent feature distributions during training.

[0103] From the geometric difference trend chart ( Figure 7(a) Figure 8 As shown in (a) of the dataset, on the Twitter and Weibo datasets, the average geometric difference of all modality pairs is consistently higher for fake news samples than for real news samples, and the gap continues to widen. This indicates that under supervised learning, the model learns to amplify cross-modal inconsistencies in fake samples while preserving cross-modal consistency in real samples. The gradual separation of the two classes of samples in geometric space improves the model's discriminative ability. To further verify this trend, a violin diagram ( Figure 7 (b) Figure 8 (b) shows the geometric difference distribution on the Twitter and Weibo datasets when the model achieves the best accuracy in the rounds. The results show that the median geometric difference of fake news is higher than that of real news across all modal pairs, confirming that the method can learn unique geometric features for authenticity classification.

[0104] The t-SNE visualization results are shown as follows: Figure 9 and Figure 10 As shown, the attention features learned by the model continuously improve their ability to distinguish between real and fake news during training: in the initialization phase ( Figure 9 (a) Figure 10 In (a), the features of the two types of samples highly overlap, indicating that the model has almost no discriminative signal at this point; while in the round with the best accuracy ( Figure 9 (b) Figure 10 In (b), the features of the two types of samples show a clear separation (especially significant on the Weibo dataset), which indicates that this method effectively learns discriminative feature representations through geometric difference regularization.

[0105] The average geometric difference between real and fake news categories showed a clear differentiation pattern throughout the training epochs. Figure 7 (a) Figure 8 In the initial stage, the geometric differences between the two types of samples were low and almost negligible, indicating that the model had not yet learned to distinguish modal-level inconsistencies. After a certain number of training rounds, the geometric differences between fake news and real news in all modal pairs increased sharply, indicating that the model gradually learned to detect the subtle cross-modal misalignments unique to fake information.

[0106] (d) Analysis of the impact of hyperparameters The experiment systematically evaluated the sensitivity of the proposed method to the geometric regularization weight λ, adjusting λ from 0.0 to 4.5 in incremental steps, and recording its accuracy on the Twitter and Weibo datasets. The results are as follows: Figure 11 As shown. From Figure 11 As can be seen in (a), the accuracy on the Twitter dataset is... It reaches its peak at [location], and then gradually declines; from [location] Figure 11 As can be seen in (b), the accuracy gradually increases on the Weibo dataset and... The situation tends to stabilize.

[0107] On the Twitter dataset, as λ increases from 0.0, accuracy initially decreases slightly, then steadily increases, reaching a peak of 0.7187 at λ=1.2; beyond this optimal value, accuracy decreases again. Overall, accuracy fluctuates around 0.66 within the test range. This trend suggests that moderate geometric regularization can improve the model's ability to capture cross-modal inconsistencies, but excessive regularization introduces noise that weakly aligns cross-modal signals, thus reducing the overall model performance.

[0108] Conversely, model performance on the Weibo dataset... λ The adjusted response varied: as λ increased from 0.0 to 3.3, the accuracy steadily increased, peaking at 0.8597 at λ=3.3; when λ>3.3, the accuracy remained stable without significant fluctuations. This indicates that stronger geometric constraints are more beneficial to the Weibo dataset, and the continuous performance improvement stems from the more discriminative visual content on this dataset. This type of content can be effectively combined with geometric discriminative modeling, strengthening cross-modal coherence.

[0109] The results above confirm that the optimal value of λ is dataset-dependent, but the method still maintains robust performance over a wide range of hyperparameters.

[0110] The experimental results, on the Weibo and Twitter datasets, demonstrate that our proposed method achieves state-of-the-art performance compared to other baseline models, thus validating its effectiveness.

[0111] It should be understood that, although the above Figure 2 The steps are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated in this document, there is no strict order in which these steps are executed; they can be performed in other orders. Furthermore, the above... Figure 2 At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0112] In one embodiment, a multimodal spoofing detection device based on geometric differences is also provided, comprising: The visual semantic parsing unit is used to input the acquired raw image into the visual semantic parsing module, and to extract entity information and image description information using a large visual language model.

[0113] The single-modal embedding vector determination unit is used to process the original image, corresponding original text, entity information and image description information using the single-modal encoder module to obtain visual embedding vector, text embedding vector, entity embedding vector and description embedding vector.

[0114] The cross-modal embedding vector determination unit is used to input visual embedding vectors, text embedding vectors, entity embedding vectors and description embedding vectors into the cross-modal attention module. It uses a multi-head attention mechanism to establish fine-grained alignment relationships between text embedding vectors and other embedding vectors to generate cross-modal embedding vectors.

[0115] The geometric uncertainty embedding vector determination unit is used to input visual embedding vectors, text embedding vectors, entity embedding vectors, and description embedding vectors into the geometric difference module. They are modeled as diagonal Gaussian distributions in a shared latent space and the geometric uncertainty embedding vectors are obtained on the statistical manifold by a geometric difference metric based on an improved Fisher-Rao distance.

[0116] The false information detection result unit is used to input cross-modal embedding vectors and geometric uncertainty embedding vectors into the prediction module to obtain false information detection results.

[0117] In one embodiment, the large visual language model in the visual semantic parsing unit is the Tongyi Qianwen 2.5-VL-72B model.

[0118] In one embodiment, the unimodal encoder module includes an image encoder and three text encoders; the unimodal embedding vector determination unit is further configured to input the original image into the image encoder, process it using a pre-trained VGG-19 network with the final classification layer removed, and obtain a visual embedding vector; input the original text into the first text encoder, encode it using a BERT model, and obtain a text embedding vector; input entity information into the second text encoder, encode it using a BERT model, and obtain an entity embedding vector; and input image description information into the third text encoder, encode it using a BERT model, and obtain a description embedding vector.

[0119] In one embodiment, the cross-modal attention module includes: an alignment and normalization module, two multi-head sparse attention mechanisms, two multi-word attention sub-modules, a hierarchical feature fusion sub-module, and a fully connected layer; the cross-modal embedding vector determination unit is further configured to process the visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector respectively using the alignment and normalization module to obtain aligned visual embedding vectors, text embedding vectors, entity embedding vectors, and description embedding vectors; the alignment and normalization module is configured to project the input features to a shared latent space of a preset dimension through linear transformation, and then perform layer normalization to obtain aligned representations; cross-modal bidirectional attention calculation and hierarchical feature fusion are performed on text-visual, text-description, and text-entity modal pairs respectively to obtain text-visual feature vectors, visual-text feature vectors, text-entity feature vectors, entity-text feature vectors, text-description feature vectors, and description-text feature vectors; and the text-visual feature vectors, visual-text feature vectors, and text-entity feature vectors are then combined. The entity feature vector, entity-text feature vector, text-description feature vector, and description-text feature vector are concatenated and input into a fully connected layer to obtain a cross-modal embedding vector. The cross-modal bidirectional attention calculation and hierarchical feature fusion process for the text-visual modality pair includes: using a multi-head sparse attention mechanism to independently calculate the query matrix, key matrix, and value matrix for the aligned visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector; inputting the query matrix of the text embedding vector, the key matrix of the visual embedding vector, and the value matrix of the visual embedding vector into the first multi-terminal attention submodule to obtain a positive cross-modal interaction result; inputting the query matrix of the visual embedding vector, the key matrix of the text embedding vector, and the value matrix of the text embedding vector into the second multi-terminal attention submodule to obtain a negative cross-modal interaction result; and inputting the concatenated positive bidirectional cross-modal interaction result and the negative cross-modal interaction result into the hierarchical feature fusion submodule to obtain the text-visual feature vector and the visual-text feature vector.

[0120] In one embodiment, the geometric difference module includes a Gaussian encoder and a geometric difference encoder; the geometric uncertainty embedding vector determination unit is further configured to project the visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector onto a shared latent space of a preset dimension and then input them into the Gaussian encoder, representing each modality as a diagonal Gaussian distribution; input the diagonal Gaussian distributions of all modalities into the geometric difference encoder, and use a geometric difference metric method based on the improved Fisher-Rao distance to calculate the pairwise geometric differences between the text modality and the other three modalities, obtaining three scalar differences and three vector differences; concatenate the three vector differences along the feature dimension and then process them using a fully connected layer to obtain the geometric uncertainty embedding vector.

[0121] In one embodiment, the geometric uncertainty embedding vector determination unit is further configured to: process the input features of the input Gaussian encoder through a fully connected layer to obtain the mean. The mean is processed through a fully connected layer, layer regularization, and the GELU function. The result is then processed again through a fully connected layer to obtain the log-variance. .

[0122] In one embodiment, the geometric difference measure method based on the improved Fisher-Rao distance in the geometric uncertainty embedding vector determination unit specifically includes: defining a vector difference measure between the diagonal Gaussian distributions of two modes as shown in the expression for the vector difference measure above; determining the scalar difference between the two modes based on the vector difference measure between the two modes; the scalar difference between the two modes is shown in the expression for the scalar difference between the two modes above.

[0123] In one embodiment, the geometrically discriminant-based multimodal misinformation detection model includes: a visual semantic parsing module, a unimodal encoder module, a cross-modal attention module, a geometrically discriminant module, and a prediction module; the overall training objective of the geometrically discriminant-based multimodal misinformation detection model is to minimize the total loss; the total loss is a weighted sum of the main classification loss and the geometric consistency loss; the main classification loss is calculated using cross-entropy loss; the geometric consistency loss is as shown in the expression for the geometric consistency loss above.

[0124] It is understood that for a detailed explanation of the multimodal misinformation detection device based on geometric differences, please refer to the corresponding explanations of the embodiments of the multimodal misinformation detection method based on geometric differences mentioned above, and will not be repeated here. Each module in the above-mentioned multimodal misinformation detection device based on geometric differences can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in hardware or independently of a device with data processing capabilities, or stored in software in the memory of the aforementioned device, so that the processor can call and execute the operations corresponding to each module. The aforementioned device can be, but is not limited to, various types of data processing computer devices already existing in the art.

[0125] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0126] It is understood that, in addition to the memory and processor mentioned above, the computer equipment described above also includes other hardware and software components not listed in this specification. The specific components can be determined according to the model of the image processing computer in different application scenarios, and will not be listed and described in detail in this specification.

[0127] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and all such modifications and improvements fall within the scope of protection of this application.

Claims

1. A multi-modal fake information detection method based on geometric discrepancy, characterized in that, Including the following steps: Step 1: Input the acquired raw image into the visual semantic parsing module, and use a large visual language model to extract entity information and image description information; Step 2: The original image, corresponding original text, entity information and image description information are processed using a single-modal encoder module to obtain visual embedding vector, text embedding vector, entity embedding vector and description embedding vector; Step 3: Input the visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector into the cross-modal attention module. Use a multi-head attention mechanism to establish fine-grained alignment between the text embedding vector and other embedding vectors to generate cross-modal embedding vectors. Step 4: Input the visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector into the geometric difference module. They are modeled as a diagonal Gaussian distribution in the shared latent space and obtained as geometric uncertainty embedding vectors on the statistical manifold through a geometric difference measure based on Fisher-Rao distance. Step 5: Input the cross-modal embedding vector and geometric uncertainty embedding vector into the prediction module to obtain the false information detection result; The geometric difference module includes a Gaussian encoder and a geometric difference encoder; The visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector are projected onto a shared latent space of a preset dimension and then input into the Gaussian encoder, representing each mode as a diagonal Gaussian distribution; The diagonal Gaussian distribution of all modalities is input into the geometric difference encoder. The pairwise geometric differences between the text modality and the other three modalities are calculated using the geometric difference metric method based on Fisher-Rao distance, resulting in three scalar differences and three vector differences. The differences between the three vectors are concatenated along the feature dimension and then processed by a fully connected layer to obtain a geometric uncertainty embedding vector; Specifically, the geometric difference measurement methods based on Fisher-Rao distance include: Define the vector difference metric between the diagonal Gaussian distributions corresponding to two modes as: in, It is a measure of the difference between the diagonal Gaussian distributions of two modes. The mean of the diagonal Gaussian distribution corresponding to the first mode. The Dimensional data, The diagonal Gaussian distribution corresponding to the second mode. mean The Dimensional data, The standard deviation of the diagonal Gaussian distribution corresponding to the first mode. The Dimensional data, The standard deviation of the diagonal Gaussian distribution corresponding to the second mode. The Dimensional data, , , Let be the dimension of the diagonal Gaussian distribution corresponding to the two modes; Based on the vector difference metric between the two modes, the scalar difference between the two modes is determined as follows: wherein, is a scalar difference between two modalities, used to represent the overall degree of cross-modal inconsistency; is a dimension, represents the first k dimension. 2.The multi-modal fake information detection method based on geometric discrepancy according to claim 1, wherein, The large-scale visual language model is the Tongyi Qianwen 2.5-VL-72B model. 3.The multi-modal fake information detection method based on geometric discrepancy according to claim 1, wherein, The single-modal encoder module includes an image encoder and three text encoders; step 2 includes: The original image is input into an image encoder and processed using a pre-trained VGG-19 network with the final classification layer removed to obtain a visual embedding vector. The original text is input into the first text encoder and encoded using the BERT model to obtain the text embedding vector; The entity information is input into the second text encoder and encoded using the BERT model to obtain the entity embedding vector; The image description information is input into a third text encoder, which is then encoded using the BERT model to obtain the description embedding vector. 4.The multi-modal fake information detection method based on geometric discrepancy according to claim 1, wherein, The cross-modal attention module includes: an alignment and normalization module, two multi-head sparse attention mechanisms, two multi-lexical attention sub-modules, a hierarchical feature fusion sub-module, and a fully connected layer; step 3 includes: The alignment and normalization module is used to process the visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector respectively to obtain aligned visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector; the alignment and normalization module is used to project the input features to a shared latent space of a preset dimension through linear transformation, and then perform layer normalization to obtain the aligned representation; Cross-modal bidirectional attention calculation and hierarchical feature fusion are performed on text-visual, text-description, and text-entity modal pairs respectively to obtain text-visual feature vector, visual-text feature vector, text-entity feature vector, entity-text feature vector, text-description feature vector, and description-text feature vector; The text-visual feature vector, the visual-text feature vector, the text-entity feature vector, the entity-text feature vector, the text-description feature vector, and the description-text feature vector are concatenated and input into a fully connected layer to obtain a cross-modal embedding vector. The cross-modal bidirectional attention computation and hierarchical feature fusion process for text-visual modal pairs includes: A multi-head sparse attention mechanism is used to independently compute the query matrix, key matrix, and value matrix for the aligned visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector; The query matrix of the text embedding vector, the key matrix and the value matrix of the visual embedding vector are input into the first multi-word attention submodule to obtain positive cross-modal interaction results; The query matrix of the visual embedding vector, the key matrix and the value matrix of the text embedding vector are input into the second multi-word attention submodule to obtain the reverse cross-modal interaction result; The concatenated result of the positive cross-modal interaction and the result of the negative cross-modal interaction, as well as the result of the positive cross-modal interaction and the result of the negative cross-modal interaction, are input into the hierarchical feature fusion submodule to obtain the text-visual feature vector and the visual-text feature vector. 5.The multi-modal fake information detection method based on geometric discrepancy according to claim 1, wherein, In the Gaussian encoder: After the input feature of the input Gaussian encoder is processed by the full connection layer, the mean value ; The mean is processed through a fully connected layer, layer regularization, and the GELU function. The result is then processed through another fully connected layer to obtain the logarithmic variance. 6.The multi-modal fake information detection method based on geometric discrepancy according to claim 1, wherein, The multimodal fake information detection model based on geometric dissimilarity includes: the visual semantic parsing module, the unimodal encoder module, the cross-modal attention module, the geometric dissimilarity module, and the prediction module; the overall training objective of the multimodal fake information detection model based on geometric dissimilarity is to minimize the total loss; the total loss is a weighted sum of the main classification loss and the geometric consistency loss; the main classification loss is calculated using cross-entropy loss; the geometric consistency loss is: in, For geometric consistency loss, For a set of cross-modal pairs, This indicates the number of cross-modal pairs in the set. These are text-entity modal pairs, text-visual modal pairs, and text-description modal pairs, respectively. and Modal pairs The average scalar difference between real news samples and fake news samples.

7. A multimodal fake information detection device based on geometric differences, characterized in that, include: The visual semantic parsing unit is used to input the acquired raw image into the visual semantic parsing module, and to extract entity information and image description information using a large visual language model; The single-modal embedding vector determination unit is used to process the original image, corresponding original text, entity information and image description information using the single-modal encoder module to obtain visual embedding vector, text embedding vector, entity embedding vector and description embedding vector; The cross-modal embedding vector determination unit is used to input visual embedding vectors, text embedding vectors, entity embedding vectors and description embedding vectors into the cross-modal attention module. It uses a multi-head attention mechanism to establish fine-grained alignment relationships between text embedding vectors and other embedding vectors to generate cross-modal embedding vectors. The geometric uncertainty embedding vector determination unit is used to input visual embedding vectors, text embedding vectors, entity embedding vectors and description embedding vectors into the geometric difference module. They are modeled as diagonal Gaussian distributions in a shared latent space and the geometric uncertainty embedding vectors are obtained on the statistical manifold through a geometric difference metric based on Fisher-Rao distance. The false information detection result unit is used to input the cross-modal embedding vector and the geometric uncertainty embedding vector into the prediction module to obtain the false information detection result; The geometric difference module includes a Gaussian encoder and a geometric difference encoder; The visual embedding vector, text embedding vector, entity embedding vector, and description embedding vector are projected onto a shared latent space of a preset dimension and then input into the Gaussian encoder, representing each mode as a diagonal Gaussian distribution; The diagonal Gaussian distribution of all modalities is input into the geometric difference encoder. The pairwise geometric differences between the text modality and the other three modalities are calculated using the geometric difference metric method based on Fisher-Rao distance, resulting in three scalar differences and three vector differences. The differences between the three vectors are concatenated along the feature dimension and then processed by a fully connected layer to obtain a geometric uncertainty embedding vector; Specifically, the geometric difference measurement methods based on Fisher-Rao distance include: Define the vector difference metric between the diagonal Gaussian distributions corresponding to two modes as: in, It is a measure of the difference between the diagonal Gaussian distributions of two modes. The mean of the diagonal Gaussian distribution corresponding to the first mode. The Dimensional data, The diagonal Gaussian distribution corresponding to the second mode. mean The Dimensional data, The standard deviation of the diagonal Gaussian distribution corresponding to the first mode. The Dimensional data, The standard deviation of the diagonal Gaussian distribution corresponding to the second mode. The Dimensional data, , , Let be the dimension of the diagonal Gaussian distribution corresponding to the two modes; Based on the vector difference metric between the two modes, the scalar difference between the two modes is determined as follows: wherein, is a scalar difference between two modalities, used to represent the overall degree of cross-modal inconsistency; is a dimension, represents the first k dimension. 8.A computer device, comprising a memory and a processor, and characterized in that, The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the multimodal fake information detection method based on geometric differences as described in any one of claims 1 to 6.

Images