Method and system for detecting false information based on cross-modal difference classification

Abstract

The present invention belongs to the technical field of error information detection, and in particular relates to an error information detection method and system based on cross-modal difference classification. The method comprises the following steps: obtaining training data and pre-processing the data to obtain data set data; performing graphic and text modality feature extraction on the data set data to obtain the image modality and text modality of the data set data; performing inter-modality difference feature extraction to obtain difference information; generating prototype features based on the image modality, text modality and difference information, and splicing the image modality, difference information and text modality to obtain sample features; training to obtain judgment prototype features based on the prototype features and sample features; judging whether the information is correct based on the prototype features of the data to be detected. The present invention solves the technical problem that the classifier method has high requirements on the modal integrity of the data and inaccurate predictions, and achieves the technical effect of accurate predictions with low modal integrity.

Description

Technical Field

[0001] The present invention belongs to the technical field of error information detection, and in particular relates to an error information detection method and system based on cross-modal difference classification. Background Art

[0002] With the rapid development of the internet and social media, the spread of false information (such as fake news and satirical content) on online platforms has become more widespread and rapid, resulting in significant negative impacts on society and individuals. Traditional false information detection methods primarily rely on information from a single modality (such as text), but with the surge in multimedia content, multimodal false information detection has gradually become a research hotspot. Existing multimodal false information detection methods typically map multimodal data such as images and text into a shared embedding space and perform classification through intermodal information fusion. However, these methods may weaken the differential expression between image and text modalities during the information fusion process, which is often important in detecting false content such as satire and misleading information. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention provides a method and system for detecting false information based on cross-modal difference classification, aiming to solve the technical problem that the MLP classifier method has high requirements for the modal integrity of data and inaccurate prediction.

[0004] The present invention provides a method for detecting false information based on cross-modal difference classification, comprising the following steps:

[0005] S1: Acquire training data and preprocess the data to obtain data set data; perform image and text modality feature extraction on the data set data to obtain image modality and text modality of the data set data;

[0006] S2: extracting difference features between the image modality and the text modality to obtain difference information;

[0007] S3: generating a prototype feature according to the image modality, text modality, and difference information, and concatenating the image modality, difference information, and text modality to obtain a sample feature;

[0008] S4: Calculating a similarity score based on the prototype feature and the sample feature, adjusting the prototype feature based on the actual results of the training data, and obtaining a judgment prototype feature;

[0009] S5: Extract the image modality of the data to be detected, the text modality of the data to be detected, and the difference information of the data to be detected, calculate the similarity score of the data to be detected based on the image modality of the data to be detected, the text modality of the data to be detected, the difference information of the data to be detected, and the judgment prototype features, and judge whether the information is correct based on the similarity score of the data to be detected.

[0010] According to a method for detecting error information based on cross-modal difference classification provided by the present invention, the text data of the data set is encoded using a deep learning model encoder to obtain a text modality; the text data of the data set is encoded using a deep learning model encoder to obtain a text modality; and the image data of the data set is encoded using a convolutional neural network encoder to obtain an image modality:

[0011]

[0012]

[0013] in, is the image modality; For text mode; Encode the convolutional neural network encoder; Encoding deep learning model encoders; is the image data; For text data.

[0014] According to the error information detection method based on cross-modal difference classification provided by the present invention, the deep learning model encoder is a BERT (Bidirectional Encoder Representations from Transformers, a bidirectional encoder using Transformer) encoder, and the convolutional neural network encoder is a ResNet34 encoder.

[0015] According to a method for detecting false information based on cross-modal difference classification provided by the present invention, step S2 includes:

[0016] S21: Input the image mode and the text mode into Combined encoder, we get Image modality in embedding space and Text modal with embedded space :

[0017]

[0018] in, for Joint encoder processing function;

[0019] S22: Input the image mode and the text mode into Combined encoder, we get Image modality in embedding space and Text modal with embedded space :

[0020]

[0021] in, for Joint encoder processing function;

[0022] S23: Calculation The difference in modal information in the embedding space and Modal difference information in embedding space:

[0023]

[0024]

[0025] in, for Modal difference information in the embedding space, for Modal difference information in the embedding space;

[0026] S24: Calculate modal difference information :

[0027] ;

[0028] S25: Use a multi-layer perceptron to align the modal difference information to obtain the difference information :

[0029] .

[0030] According to the present invention, a method for detecting erroneous information based on cross-modal difference classification is provided. The joint encoder contains u-layer linear layers and v-layer activation layers. The activation layer of the joint encoder uses the ReLU function;

[0031] described The joint encoder contains m linear layers and n activation layers. The activation layer of the joint encoder uses the ReLU function.

[0032] According to the present invention, a method for detecting erroneous information based on cross-modal difference classification, step S23, includes:

[0033] like When Normalize using the infinity norm:

[0034]

[0035] in, is the comparison threshold, is the infinite norm;

[0036] like When Normalize using the infinity norm:

[0037] ;

[0038] like When Normalize using the infinity norm:

[0039] ;

[0040] like When Normalize using the infinity norm:

[0041] .

[0042] According to the present invention, a method for detecting false information based on cross-modal difference classification, step S3, includes:

[0043] S31: Generate initial prototype features using Gaussian initialization based on the image modality, text modality and vector length of the difference information ,

[0044]

[0045] in, is the initial image prototype feature, is the initial difference prototype feature, is the initial text prototype feature;

[0046] S32: Perform Schmidt orthogonalization on the initial prototype features so that the initial image prototype features, the initial difference prototype features and the initial text prototype features are orthogonal to each other to obtain the prototype features :

[0047]

[0048] in, is the image prototype feature, is the difference prototype feature, It is the prototype feature of the text;

[0049] S33: combining the image modality, the text modality, and the difference information to obtain the sample feature :

[0050] .

[0051] According to the present invention, a method for detecting false information based on cross-modal difference classification, step S4, includes:

[0052] S41: Calculating the similarity score :

[0053]

[0054] S42: Using an optimizer to adjust the prototype features according to the actual results of the training data to obtain judgment prototype features.

[0055] According to a method for detecting false information based on cross-modal difference classification provided by the present invention, step S41 includes:

[0056] When the training data lacks image modality, the similarity score calculation formula is:

[0057]

[0058] When the training data lacks text modality, the similarity score calculation formula is:

[0059] .

[0060] The present invention also provides a system for detecting false information based on cross-modal difference classification, comprising:

[0061] Preprocessing module: Acquires training data and preprocesses the data to obtain data set data; extracts image and text modality features from the data set data to obtain image modality and text modality of the data set data; extracts intermodality difference features from the image modality and the text modality to obtain difference information; generates prototype features based on the image modality, text modality and difference information, and splices the image modality, difference information and text modality to obtain sample features;

[0062] A judgment prototype feature training module calculates a similarity score based on the prototype feature and the sample feature, and adjusts the prototype feature according to the actual results of the training data to obtain a judgment prototype feature;

[0063] Prediction module: extracts the image modality of the data to be detected, the text modality of the data to be detected, and the difference information of the data to be detected, calculates the similarity score of the data to be detected based on the image modality of the data to be detected, the text modality of the data to be detected, the difference information of the data to be detected and the judgment prototype features, and judges whether the information is correct based on the similarity score of the data to be detected.

[0064] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0065] This paper provides a method and system for detecting false information based on cross-modal difference classification. By leveraging information differences between image and text modalities and combining them with a prototype-based classifier, it achieves stable detection in modality-absent conditions and improves the model's ability to interpret features from different modalities.

[0066] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0068] Figure 1 3 is a flow chart of the error information detection method based on cross-modal difference classification provided by the present invention.

[0069] Figure 2 This is a structural block diagram of the error information detection device based on cross-modal difference classification provided by the present invention.

[0070] Reference numerals:

[0071] 101. Preprocessing module; 102. Prototype feature training module; 103. Prediction module. DETAILED DESCRIPTION

[0072] To make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below. Obviously, the embodiments described are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0073] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0074] Example

[0075] The following combination Figure 1 and Figure 2 The present invention is described.

[0076] like Figure 1 As shown, the present invention provides a method for detecting false information based on cross-modal difference classification, the steps of which are as follows:

[0077] S1: Acquire training data and preprocess the data to obtain data set data; perform image and text modality feature extraction on the data set data to obtain image modality and text modality of the data set data;

[0078] S2: extracting difference features between the image modality and the text modality to obtain difference information;

[0079] S3: generating a prototype feature according to the image modality, text modality, and difference information, and concatenating the image modality, difference information, and text modality to obtain a sample feature;

[0080] S4: Calculating a similarity score based on the prototype feature and the sample feature, adjusting the prototype feature based on the actual results of the training data, and obtaining a judgment prototype feature;

[0081] S5: Extract the image modality of the data to be detected, the text modality of the data to be detected, and the difference information of the data to be detected, calculate the similarity score of the data to be detected based on the image modality of the data to be detected, the text modality of the data to be detected, the difference information of the data to be detected, and the judgment prototype features, and judge whether the information is correct based on the similarity score of the data to be detected.

[0082] To verify the effectiveness and robustness of the present invention, three public multimodal datasets were used for training in the embodiments of the present invention, including:

[0083] 1. Weibo Dataset: The Weibo dataset is a public dataset for multimodal disinformation detection in Chinese. It contains image and text pairs from social media platforms. The dataset is divided into two categories: rumor samples and non-rumor samples. The training set contains 3,749 rumor samples and 3,783 non-rumor samples.

[0084] 2. Pheme dataset: The Pheme dataset is an English multimodal false information detection dataset that focuses on the detection of rumors on social media. This dataset contains 590 rumor samples and 1,428 non-rumor samples, and is suitable for evaluating the false information detection ability of the model in a multimodal environment. The Pheme dataset contains missing cases of different modalities. For example, approximately 36.75% of the samples lack image modality. The present invention uses the Pheme dataset to verify the stability of the prototype-based classifier in the case of modality missing, and proves the effectiveness of the dynamic mask mechanism in the scenario of incomplete modality.

[0085] 3. Sarcasm Dataset: The Sarcasm dataset is an English multimodal sarcasm detection dataset that contains both sarcastic and non-sarcastic image-text pairs. The training set contains 8,642 sarcastic examples and 11,174 non-sarcastic examples, and the test set contains 959 sarcastic examples and 1,450 non-sarcastic examples.

[0086] To ensure the comparability of the experimental results, the embodiment of the present invention needs to first perform data partitioning on the three data sets. The Weibo data set and the Sarcasm data set use the LogicDM (Logical Data Model) partitioning method; the Pheme data set uses the MFAN (Multi-modal Feature-enhanced Attention Network) partitioning method to obtain training data.

[0087] Then, the present invention obtains data set data by preprocessing the training data; extracting image and text modality features from the data set data to obtain image modality and text modality of the data set data;

[0088] The text data of the data set data is encoded using a deep learning model encoder to obtain a text modality; the image data of the data set data is encoded using a convolutional neural network encoder to obtain an image modality:

[0089]

[0090]

[0091] in, is the image modality; For text mode; Encode the convolutional neural network encoder; Encoding deep learning model encoders; is the image data; The deep learning model encoder in the embodiment of the present invention is a BERT encoder, and the convolutional neural network encoder in the embodiment of the present invention is a ResNet34 encoder.

[0092] Specifically, in order to obtain the difference between image and text features, the embodiment of the present invention first maps them to the same semantic space to a certain extent through a joint encoder, thereby facilitating subsequent processing of features from different modalities and obtaining difference information. The specific steps are as follows:

[0093] Input the image modality and the text modality into Combined encoder, we get Image modality in embedding space and Text modal with embedded space :

[0094]

[0095] in, for Joint encoder processing function;

[0096] Input the image modality and the text modality into Combined encoder, we get Image modality in embedding space and Text modal with embedded space :

[0097]

[0098] in, for Joint encoder processing function. In particular, the embodiment of the present invention The joint encoder consists of u linear layers and v activation layers; The joint encoder includes m linear layers and n activation layers. Joint encoder and The number of linear layers of the joint encoder is 4, and the number of activation layers is 1. Joint encoder and The activation layers of the joint encoder all use the ReLU function.

[0099] calculate The difference in modal information in the embedding space and Modal difference information in embedding space:

[0100]

[0101]

[0102] in, for Modal difference information in the embedding space, for Modal difference information in the embedding space;

[0103] Calculate modal difference information :

[0104] ;

[0105] Use a multi-layer perceptron to align the modal difference information to obtain the difference information :

[0106] .

[0107] Since the model may have different degrees of bias for features of different modalities for different samples and different data sets, the weights of the generated features of different modalities may be inconsistent. When the weight distribution differences between modalities are very large, it may have an adverse effect on the calculation of the difference information, resulting in the final difference information not really obtaining the difference features between different modalities, but relying too much on one of the modalities. Therefore, in the embodiment of the present invention, the features with too large differences between modalities are constrained by the ∞ norm, and the numerical value range of the features with too large differences is constrained to The specific processing methods are:

[0108] like When Normalize using the infinity norm:

[0109] ;

[0110] like When Normalize using the infinity norm:

[0111] ;

[0112] like When Normalize using the infinity norm:

[0113] ;

[0114] like When Normalize using the infinity norm:

[0115] ;

[0116] in, is the comparison threshold, in the embodiment of the present invention , is an infinite norm.

[0117] Prototype features are generated based on the image modality, text modality and difference information, and the image modality, difference information and text modality are spliced to obtain sample features; similarity scores are calculated based on the prototype features and the sample features, and the prototype features are adjusted according to the actual results of the training data to obtain judgment prototype features.

[0118] According to the image modality, text modality and the vector length of the difference information, Gaussian initialization is used to generate the initial prototype features. ,

[0119]

[0120] in, is the initial image prototype feature, is the initial difference prototype feature, is the initial text prototype feature;

[0121] Perform Schmidt orthogonalization on the initial prototype features so that the initial image prototype features, initial difference prototype features and initial text prototype features are orthogonal to each other to obtain the prototype features. :

[0122]

[0123] in, is the image prototype feature, is the difference prototype feature, It is the prototype feature of the text;

[0124] Splicing the image modality, the text modality and the difference information to obtain the sample feature :

[0125] .

[0126] Calculate the similarity score :

[0127]

[0128] According to the actual results of the training data, the prototype features are adjusted using the optimizer to obtain the judgment prototype features. The similarity score calculation formula is adjusted using the prototype classifier:

[0129] When the training data lacks image modality, the similarity score calculation formula is:

[0130]

[0131] When the training data lacks text modality, the similarity score calculation formula is:

[0132] .

[0133] Specifically, the embodiment of the present invention determines whether the data to be detected is correct by extracting the image modality of the data to be detected, the text modality of the data to be detected, and the difference information of the data to be detected, and calculating the similarity score of the data to be detected based on the image modality of the data to be detected, the text modality of the data to be detected, the difference information of the data to be detected, and the judgment prototype features. When the similarity score is less than the judgment threshold, it is judged to be false information; when the similarity score is greater than the judgment threshold, it is judged to be true information.

[0134] The effectiveness and advantages of the embodiments of the present invention are demonstrated below:

[0135] Table 1 Performance comparison of different detection methods on Weibo and Pheme datasets

[0136]

[0137] Table 2 Performance comparison of different detection methods on the Sarcasm dataset

[0138]

[0139] Among them, EANN (Event adversarial neural networks); MVAE (Multimodal variational autoencoder); Spotfake; Spotfake+; MCAN (Multimodal fusion with co-attention network); BMR (Bootstrapping multi-view representations); LogicDM+ (Interpretable multimodal misinformation detection with logic reasoning+); FSRU (Frequency Spectrum Representation and fUsion network); MFAN+ (Multi-modal Feature-enhanced Attention Network+); MMCAN-Res (Multimodal Matching-aware Co-Attention Network-Res); Bert (Bidirectional Encoder Representations from Transformers); ViT (Vision Other misinformation detection technologies include Transformer (Visual Transformer), HFM (Hierarchical Fusion Model), D&R Net (Decomposition and Relation Network), Att-Bert, InCrossMGs (in-modal and cross-modal graphs), HCM (Hierarchical Congruity Modeling), and LogicDM (Interpretable multimodal misinformation detection with logicreasoning).

[0140] As shown in Tables 1 and 2, on the Weibo, Pheme, and Sarcasm datasets, the embodiments of the present invention outperform existing advanced methods in multiple metrics, including accuracy, precision, recall, and F1 score, demonstrating excellent detection capabilities. For example, on the Weibo dataset, the F1 score of the embodiment of the present invention reached 92.5%, a 1% improvement over the traditional MLP classifier. On the Pheme and Sarcasm datasets, the F1 scores of the embodiments of the present invention improved by 1.7% and 4.4%, respectively. In particular, the embodiments of the present invention demonstrated strong generalization capabilities in the sarcasm detection task, validating the model's adaptability in different scenarios and language environments.

[0141] Since the embodiment of the present invention has a very strong feature decoupling capability, it has a very strong effect on sample classification even when the sample modality is missing. Therefore, the embodiment of the present invention tests the adaptability and better performance of the model of the embodiment of the present invention in the sample modality missing environment from two aspects.

[0142] First, the embodiment of the present invention is tested from the perspective of the number of features used. The embodiment of the present invention tests the model's capabilities in three datasets under different usage scenarios for three features: image, text, and information difference features. On the one hand, this can test the feature decoupling capabilities of the embodiment of the present invention, and on the other hand, it can also reflect the model's dependence on different categories of features and the effectiveness of the features. The experimental results are shown in Table 3.

[0143] Table 3 Performance comparison of modality missing environment and different features

[0144]

[0145] in, Indicates the presence of this feature type.

[0146] Table 3 shows the results of feature ablation experiments on the model of the present invention on three datasets: Weibo, Pheme and Sarcasm, which are used to analyze the contribution of image features, text features and inter-modal difference features to classification performance. The experiment shows that when only a single feature is used, the classification performance of the inter-modal difference feature on all datasets is better than that of the image feature or text feature, indicating that the inter-modal difference feature can effectively capture the contradictory information in multimodal data and plays an important role. In the joint feature experiment, the combination of any two features can significantly improve the model performance, especially the combination of difference features with text or image features, which further verifies the importance of difference features in enhancing the expression of multimodal information. When the image, text and inter-modal difference features are used together, the classification accuracy and F1 score of the model on all datasets reach the highest, fully demonstrating the advantages of multimodal feature fusion. The results also show that the model of the present invention can effectively decouple feature contributions through a dynamic mask mechanism, thereby showing high classification performance and robustness under different feature combinations.

[0147] Secondly, to demonstrate the strong feature decoupling capabilities of this embodiment of the present invention, while maintaining the same model structure, the final prototype classifier was replaced with an MLP classifier. This not only verifies the effective utilization of the inter-modal information difference capability of this embodiment of the present invention, but also better demonstrates that the prototype-based classification method of this embodiment of the present invention can better decouple features from different categories and achieve better results in the absence of modalities. The final results are shown in Table 4.

[0148] Table 4 Comparison of modality loss performance between MLP structure and prototype classifier

[0149]

[0150] Table 4 shows the experimental results of the embodiment of the present invention in a modality-missing scenario on the Weibo dataset, used to evaluate the classification performance of the model when the image or text modality is missing. In the complete modality scenario, the prototype classifier achieved the highest accuracy and F1 score, outperforming the MLP-based classifier, validating the advantages of the prototype classifier in processing multimodal input. In the unimodal scenario, the embodiment of the present invention still demonstrated strong adaptability. In particular, when only text input was present, the classification accuracy only decreased by 1.4% compared to the complete modality scenario, significantly outperforming the scenario with only image input. This demonstrates that the text modality provides more discriminative information in false information detection. In addition, compared with the traditional MLP classifier, the performance improvement of the embodiment of the present invention in the unimodal scenario was even more significant. When only image input was present, the accuracy increased by 12%, further demonstrating the robustness and applicability of the dynamic mask mechanism and prototype classifier in the modality-missing scenario. Overall, the experimental results show that the model of the present invention can effectively handle scenarios with incomplete modalities. In particular, when the text modality is retained, the detection performance is close to that of the complete modality scenario, demonstrating strong practical application value.

[0151] Table 5. Comparison of model structure ablation performance

[0152]

[0153] Among them, -w / o is the abbreviation of without, β is the β joint encoder, and ∞-norm is the infinite norm normalization.

[0154] The ablation experiments in Table 5 show that each module significantly improves the overall performance of the model. Bidirectional discrepancy computation (β-joint encoder) effectively captures inconsistencies in inter-modal information, while the ∞-norm normalization constraint balances the numerical range of modal features. The prototype-based classifier significantly outperforms the traditional MLP classifier in terms of classification accuracy and adaptability. The synergistic effect of these modules enables the model to achieve excellent performance in both complete and missing modalities.

[0155] like Figure 2 As shown, the present invention also provides a system comprising:

[0156] Preprocessing module 101: Acquires training data and preprocesses the data to obtain dataset data; performs image-text modality feature extraction on the dataset data to obtain image modality and text modality of the dataset data; performs inter-modality difference feature extraction on the image modality and the text modality to obtain difference information; generates prototype features based on the image modality, text modality, and difference information, and splices the image modality, difference information, and text modality to obtain sample features;

[0157] The judgment prototype feature training module 102 calculates a similarity score based on the prototype feature and the sample feature, and adjusts the prototype feature according to the actual results of the training data to obtain the judgment prototype feature;

[0158] Prediction module 103: extracts the image modality of the data to be detected, the text modality of the data to be detected, and the difference information of the data to be detected, calculates the similarity score of the data to be detected based on the image modality of the data to be detected, the text modality of the data to be detected, the difference information of the data to be detected, and the judgment prototype features, and judges whether the information is correct based on the similarity score of the data to be detected.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for detecting false information based on cross-modal difference classification, characterized in that: The following steps are involved: S1: Acquire training data and preprocess the data to obtain data set data; perform image and text modality feature extraction on the data set data to obtain image modality and text modality of the data set data; S2: extracting difference features between the image modality and the text modality to obtain difference information; S3: Generate prototype features based on the image modality, text modality and difference information, and splice the image modality, difference information and text modality to obtain sample features; Step S3 includes: S31: Generate initial prototype features using Gaussian initialization based on the image modality, text modality and vector length of the difference information , in, is the initial image prototype feature, is the initial difference prototype feature, is the initial text prototype feature; S32: Perform Schmidt orthogonalization on the initial prototype features so that the initial image prototype features, the initial difference prototype features and the initial text prototype features are orthogonal to each other to obtain the prototype features : in, is the image prototype feature, is the difference prototype feature, It is the prototype feature of the text; S33: combining the image modality, text modality, and the difference information to obtain the sample feature : in, is the image modality; For text mode; For difference information; S4: Calculating a similarity score based on the prototype feature and the sample feature, adjusting the prototype feature based on the actual results of the training data, and obtaining a judgment prototype feature; S5: Extract the image modality of the data to be detected, the text modality of the data to be detected, and the difference information of the data to be detected, calculate the similarity score of the data to be detected based on the image modality of the data to be detected, the text modality of the data to be detected, the difference information of the data to be detected, and the judgment prototype features, and judge whether the information is correct based on the similarity score of the data to be detected.

2. The error information detection method based on cross-modal difference classification according to claim 1 is characterized in that: The text data of the data set data is encoded using a deep learning model encoder to obtain a text modality; the image data of the data set data is encoded using a convolutional neural network encoder to obtain an image modality: in, Encode a convolutional neural network encoder for image processing; Encoding a deep learning encoder for text modality processing; is the image data; For text data.

3. The error information detection method based on cross-modal difference classification according to claim 2 is characterized in that: The deep learning model encoder is a BERT encoder, and the convolutional neural network encoder is a ResNet34 encoder.

4. The error information detection method based on cross-modal difference classification according to claim 2 is characterized in that: Step S2 includes: S21: Input the image mode and the text mode into Combined encoder, we get Image modality in embedding space and Text modal with embedded space : in, for Joint encoder processing function; S22: Input the image mode and the text mode into Combined encoder, we get Image modality in embedding space and Text modal with embedded space : in, for Joint encoder processing function; S23: Calculation The difference in modal information in the embedding space and Modal difference information in embedding space: in, for Modal difference information in the embedding space, for Modal difference information in the embedding space; S24: Calculate modal difference information : ； S25: Use a multi-layer perceptron to align the modal difference information to obtain the difference information : 。 5. The error information detection method based on cross-modal difference classification according to claim 4 is characterized in that: described The joint encoder contains u-layer linear layers and v-layer activation layers. The activation layer of the joint encoder uses the ReLU function; described The joint encoder contains m linear layers and n activation layers. The activation layer of the joint encoder uses the ReLU function.

6. The error information detection method based on cross-modal difference classification according to claim 4 is characterized in that: Step S23 includes: like When Normalize using the infinity norm: in, is the comparison threshold, is the infinite norm; like When Normalize using the infinity norm: ； like When Normalize using the infinity norm: ； like When Normalize using the infinity norm: 。 7. The error information detection method based on cross-modal difference classification according to claim 6, characterized in that: Step S4 includes: S41: Calculating the similarity score : S42: Using an optimizer to adjust the prototype features according to the actual results of the training data to obtain judgment prototype features.

8. The error information detection method based on cross-modal difference classification according to claim 7 is characterized in that: Step S41 includes: When the training data lacks image modality, the similarity score calculation formula is: When the training data lacks text modality, the similarity score calculation formula is: 。 9. A system for detecting error information based on cross-modal difference classification, for executing the error information detection method based on cross-modal difference classification according to any one of claims 1 to 8, characterized in that: include: Preprocessing module: obtains training data and preprocesses the data to obtain data set data; Performing image-text modality feature extraction on the data set data to obtain the image modality and text modality of the data set data; performing inter-modality difference feature extraction on the image modality and the text modality to obtain difference information; generating prototype features based on the image modality, text modality and difference information, and splicing the image modality, difference information and text modality to obtain sample features; generating initial prototype features using Gaussian initialization based on the vector lengths of the image modality, text modality and difference information , in, is the initial image prototype feature, is the initial difference prototype feature, is the initial text prototype feature; Perform Schmidt orthogonalization on the initial prototype features so that the initial image prototype features, initial difference prototype features and initial text prototype features are orthogonal to each other to obtain the prototype features. : in, is the image prototype feature, is the difference prototype feature, It is the prototype feature of the text; Splicing the image modality, the text modality and the difference information to obtain the sample feature : in, is the image modality; For text mode; For difference information; A judgment prototype feature training module calculates a similarity score based on the prototype feature and the sample feature, and adjusts the prototype feature according to the actual results of the training data to obtain a judgment prototype feature; Prediction module: extracts the image modality of the data to be detected, the text modality of the data to be detected, and the difference information of the data to be detected, calculates the similarity score of the data to be detected based on the image modality of the data to be detected, the text modality of the data to be detected, the difference information of the data to be detected and the judgment prototype features, and judges whether the information is correct based on the similarity score of the data to be detected.

Images