Training method, detection method and system of multi-modal information detection model

Abstract

This disclosure relates to a training method, detection method, and system for a multimodal information detection model. The training method includes: acquiring a multimodal training sample set and sample annotation information; for each modality in the multimodal sample information, obtaining a single-modal sample set by masking information from other modalities outside that modality; inputting the multimodal training sample set and the single-modal sample set into the multimodal information detection model respectively to obtain multimodal prediction and single-modal prediction; determining a prediction loss based on the multimodal prediction, single-modal prediction, and sample annotation information; and training the multimodal information detection model using the prediction loss. The training method, detection method, and system for the multimodal information detection model disclosed herein can solve the problem of difficulty in improving model accuracy due to different learning levels of different modalities. It can fully learn the features of each single modality, making full use of the information of each modality during training, thereby improving the training effect and accuracy of the model.

Description

Technical Field

[0001] This disclosure relates to the fields of computer technology and artificial intelligence, and in particular to a training method, detection method and system for a multimodal information detection model. Background Technology

[0002] With the rapid expansion of multimodal content on online social media, technologies for automatically detecting the authenticity of information, such as multimodal fake news, have received widespread attention.

[0003] In related technologies, multimodal joint training is typically expected to benefit from fully utilizing cross-modal features. Examples include multimodal fake news detection methods that combine visual, textual, and social contextual features; methods that learn event-invariant features through adversarial training and add an event discriminator after feature extraction; and methods that utilize encoder-decoder architectures to learn shared representations of textual and visual features.

[0004] However, due to the heterogeneity of multimodal networks, the convergence speeds of different modalities vary significantly. Optimizing the single objective of multimodal joint training will inevitably cause the model to rely on a specific modality, while other modalities will be underoptimized and not fully utilized by the model, making it difficult to further improve the model's accuracy. Summary of the Invention

[0005] This disclosure provides a training method, detection method, and system for a multimodal information detection model, to at least address the problem in related technologies where different modalities have varying levels of learning, making it difficult to improve model accuracy. The technical solution of this disclosure is as follows:

[0006] According to a first aspect of this disclosure, a training method for a multimodal information detection model is provided. The training method includes: acquiring a multimodal training sample set and sample annotation information, wherein each training sample in the multimodal training sample set includes multimodal sample information, the multimodal sample information includes image information and text information, the modality includes an image modality and a text modality, and the sample annotation information characterizes the information authenticity of each training sample; for each modality in the multimodal sample information, by masking information of other modalities outside that modality in the multimodal sample information, a single-modal sample set corresponding to that modality is obtained; inputting the multimodal training sample set and the single-modal sample set corresponding to each modality into the multimodal information detection model to obtain a multimodal prediction for the multimodal training sample set and a single-modal prediction for each single-modal sample set; determining a prediction loss based on the multimodal prediction, each single-modal prediction, and the sample annotation information; and adjusting the parameters of the multimodal information detection model using the prediction loss to train the multimodal information detection model.

[0007] Optionally, for each modality in the multimodal sample information, a single-modal sample set corresponding to that modality is obtained by: determining the feature distribution of other modal information in the multimodal training sample set besides that modality; constructing a masking function based on the feature distribution, wherein the masking function can characterize the information-free noise data of the feature distribution; determining padding data for replacing the other modal information based on the other modal information and the masking function; and replacing the other modal information in the multimodal training sample set with the padding data to obtain the single-modal sample set.

[0008] Optionally, the prediction loss is determined by: determining a single-modal loss corresponding to each single-modal prediction based on each single-modal prediction; determining a multimodal loss corresponding to the multimodal prediction based on the multimodal prediction; and determining the prediction loss based on each single-modal loss and the multimodal loss, wherein, for each single-modal prediction, the single-modal loss is determined by: obtaining a single-modal teacher prediction of a pre-trained teacher model corresponding to that modality, wherein the single-modal teacher prediction is obtained by inputting a single-modal sample set corresponding to that modality into the pre-trained teacher model, and the value of the single-modal teacher prediction is a real number in the range [0,1]; and determining the single-modal loss based on the single-modal teacher prediction and the single-modal prediction.

[0009] Optionally, the step of determining the single-modal loss based on the single-modal teacher prediction and the single-modal prediction includes: determining a first loss based on the single-modal teacher prediction and the single-modal prediction; determining a second loss based on the single-modal prediction and the sample labeling information; and determining the single-modal loss based on the first loss and the second loss.

[0010] Optionally, the single-modal training samples in the single-modal sample set are divided into multiple sample clusters, each sample cluster corresponding to a corresponding information topic. Different sample clusters correspond to different information topics, and there are no overlapping samples between different sample clusters. The single-modal teacher prediction is obtained by: training the teacher model using other sample clusters besides the sample cluster for each sample cluster to obtain a trained teacher model; inputting the sample cluster into the trained teacher model to obtain the single-modal teacher prediction for the sample cluster.

[0011] Optionally, the step of determining the prediction loss based on each single-modal loss and the multimodal loss includes: for each modality, determining the contribution of that modality to the multimodal information detection model, wherein the contribution is determined based on the improvement in overall training accuracy of the multimodal information detection model by introducing that modality into all possible modal sets other than that modality; determining the underfit of each modality based on the contribution of each modality and the performance of the teacher model corresponding to each modality; determining the loss weight corresponding to each single-modal loss based on the underfit of each modality; weighting the corresponding single-modal losses based on the loss weights of each single-modal loss to determine the weighted single-modal loss; and determining the prediction loss based on the weighted single-modal loss and the multimodal loss.

[0012] According to a second aspect of this disclosure, a multimodal information detection method is provided, the multimodal information detection method comprising: acquiring target multimodal information, the target multimodal information including image information and text information; inputting the target multimodal information into a multimodal information detection model to obtain a detection result of the information authenticity of the multimodal information, wherein the multimodal information detection model is trained according to the training method of the multimodal information detection model described in the first aspect of this disclosure.

[0013] According to a third aspect of this disclosure, a training system for a multimodal information detection model is provided. The training system includes: a training acquisition unit configured to acquire a multimodal training sample set and sample annotation information, wherein each training sample in the multimodal training sample set includes multimodal sample information, the multimodal sample information including image information and text information, the modality including an image modality and a text modality, and the sample annotation information characterizing the information authenticity of each training sample; and a masking unit configured to, for each modality in the multimodal sample information, mask information of other modalities outside that modality in the multimodal sample information. The system obtains a single-modal sample set corresponding to the modality; a single-modal prediction unit is configured to input the multimodal training sample set and the single-modal sample set corresponding to each modality into the multimodal information detection model to obtain multimodal predictions for the multimodal training sample set and single-modal predictions for each single-modal sample set; a loss determination unit is configured to determine a prediction loss based on the multimodal predictions, each single-modal prediction, and the sample labeling information; and a training unit is configured to adjust the parameters of the multimodal information detection model using the prediction loss to train the multimodal information detection model.

[0014] According to a fourth aspect of this disclosure, a multimodal information detection system is provided, the multimodal information detection system comprising: an information acquisition unit configured to acquire target multimodal information, the target multimodal information including image information and text information; and an information detection unit configured to input the target multimodal information into a multimodal information detection model to obtain a detection result of the information authenticity of the multimodal information, wherein the multimodal information detection model is trained according to the training method of the multimodal information detection model described in the first aspect of this disclosure.

[0015] According to a fifth aspect of this disclosure, an electronic device is provided, the electronic device comprising: a processor; and a memory for storing processor-executable instructions, wherein, when executed by the processor, the processor-executable instructions cause the processor to perform a training method for a multimodal information detection model according to a first aspect of this disclosure or a multimodal information detection method according to a second aspect of this disclosure.

[0016] According to a sixth aspect of this disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform a training method for a multimodal information detection model according to a first aspect of this disclosure or a multimodal information detection method according to a second aspect of this disclosure.

[0017] According to a seventh aspect of this disclosure, a computer program product is provided, the computer program product including computer-executable instructions, which, when executed by at least one processor, implement a training method for a multimodal information detection model according to a first aspect of this disclosure or a multimodal information detection method according to a second aspect of this disclosure.

[0018] The technical solution provided in this disclosure brings at least the following beneficial effects:

[0019] By employing this disclosure, for each modality in the multimodal sample information of the multimodal training samples, information from other modalities outside that modality can be masked to obtain the corresponding single-modal sample. This allows for obtaining multimodal predictions for the multimodal training sample set and single-modal predictions for each single-modal sample set. Thus, the loss can be predicted based on both multimodal and single-modal predictions to train the multimodal information detection model. This not only allows the model to learn the correlation features between multimodalities but also fully learns the features of each single modality. Furthermore, the loss can be determined by combining the learning from both aspects. This approach fully utilizes the information from each modality during training, improving the training effect and accuracy of the model.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0022] Figure 1 This is a schematic flowchart of a training method for a multimodal information detection model according to an exemplary embodiment of the present disclosure.

[0023] Figure 2 This is a schematic flowchart illustrating the steps of obtaining a single-modal sample set in a training method for a multimodal information detection model according to an exemplary embodiment of the present disclosure.

[0024] Figure 3 This is a schematic block diagram of a training method for a multimodal information detection model according to exemplary embodiments of the present disclosure.

[0025] Figure 4 This is a schematic flowchart illustrating the step of determining the prediction loss in a training method for a multimodal information detection model according to an exemplary embodiment of the present disclosure.

[0026] Figure 5 This is a schematic flowchart illustrating the step of determining a single-modal loss in a training method for a multimodal information detection model according to an exemplary embodiment of the present disclosure.

[0027] Figure 6 This is a schematic flowchart illustrating the step of determining the prediction loss based on each unimodal loss and the multimodal loss in a training method for a multimodal information detection model according to an exemplary embodiment of the present disclosure.

[0028] Figure 7 This is a schematic flowchart illustrating the step of determining the degree of influence in a training method for a multimodal information detection model according to an exemplary embodiment of the present disclosure.

[0029] Figure 8 This is a schematic flowchart illustrating the steps of obtaining a single-modal teacher prediction in a training method for a multimodal information detection model according to an exemplary embodiment of the present disclosure.

[0030] Figure 9 This is a schematic diagram of training a teacher model in a training method for a multimodal information detection model according to an exemplary embodiment of the present disclosure.

[0031] Figure 10 This is a schematic flowchart of a multimodal information detection method according to an exemplary embodiment of the present disclosure.

[0032] Figure 11A This is a schematic diagram of experimental results using a multimodal information detection method according to an exemplary embodiment of the present disclosure.

[0033] Figure 11B This is a schematic diagram showing the single-modal performance analysis results of a multimodal information detection method according to an exemplary embodiment of the present disclosure.

[0034] Figure 12 This is a schematic block diagram of a training system for a multimodal information detection model according to exemplary embodiments of the present disclosure.

[0035] Figure 13 This is a schematic block diagram of a multimodal information detection system according to exemplary embodiments of the present disclosure.

[0036] Figure 14 This is a block diagram of an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0037] In order to enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0038] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0039] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. As another example, "performing at least one of step one and step two" indicates the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0040] As mentioned earlier, in the related technologies, during the training of multimodal information detection models, the convergence speeds of different modalities in the multimodal information of the training samples are different. This causes the final trained model to tend to rely on a specific modality and fail to fully learn the features of each modality, making it difficult to further improve the accuracy of the model.

[0041] What may be more serious is that, because these methods rely on only a single loss function, there may even be a situation where all modes are still underconverged after the multimodal loss function has converged.

[0042] Furthermore, multimodal information, such as that in multimodal fake news, typically only involves the tampering of one modality. However, sample annotation information can only indicate the authenticity of samples from a holistic perspective across the multimodalities, meaning it cannot determine the extent of tampering in each individual modality. Therefore, how to assign fine-grained responsibilities to each modality based on each sample remains a problem that related technologies cannot solve.

[0043] In view of the above problems, exemplary embodiments of this disclosure provide a training method for a multimodal information detection model, a multimodal information detection method, a training system for a multimodal information detection model, a multimodal information detection system, an electronic device, a computer-readable storage medium, and a computer program product, which can solve or at least alleviate the above problems.

[0044] In a first aspect of an exemplary embodiment of this disclosure, a method for training a multimodal information detection model is provided.

[0045] The training method for the multimodal information detection model according to the exemplary embodiments of the present disclosure can be applied to scenarios where users interact with software. For example, software can be loaded on a user terminal, and the user can input a multimodal training sample set and sample annotation information for training the model on the user terminal. The user terminal can train the model based on the multimodal training sample set and sample annotation information by executing the training method for the multimodal information detection model according to the exemplary embodiments of the present disclosure to obtain the final multimodal information detection model.

[0046] Specifically, the user terminal can acquire a multimodal training sample set and sample annotation information. Each training sample in the multimodal training sample set includes multimodal sample information, which includes image and text information. The modalities include image and text modalities. The sample annotation information characterizes the authenticity of each training sample. The user terminal can also, for each modality in the multimodal sample information, mask information from other modalities outside that modality to obtain a corresponding single-modal sample set. The user terminal can also input the multimodal training sample set and the corresponding single-modal sample set into the multimodal information detection model to obtain multimodal predictions for the multimodal training sample set and single-modal predictions for each single-modal sample set, respectively. The user terminal can also determine the prediction loss based on the multimodal predictions, each single-modal prediction, and the sample annotation information. The user terminal can also use the prediction loss to adjust the parameters of the multimodal information detection model for training.

[0047] The aforementioned user terminal can be such as a tablet computer, laptop computer, digital assistant, wearable device, etc. However, the implementation scenario of the above training method is only an example scenario. The training method of the multimodal information detection model according to the exemplary embodiment of this disclosure can also be applied to other application scenarios. For example, the user can request the model to be trained from the server via the network on the user terminal (e.g., mobile phone, desktop computer, tablet computer, etc.). The server can complete the model training by executing the training method of the multimodal information detection model according to the exemplary embodiment of this disclosure. Here, the server can be an independent server, a server cluster, a cloud computing platform, or a virtualization center.

[0048] The training method for the multimodal information detection model according to the exemplary embodiments of the present disclosure can learn the correlation features between multimodalities and fully learn the features of each single modality, avoiding the model's tendency to rely on a specific modality due to different convergence speeds of different modalities. This method can combine the learning from both aspects to determine the loss, and can make full use of the information of each modality during the training process, thereby improving the training effect and accuracy of the model.

[0049] The following will refer to Figure 1 Example steps of a training method for a multimodal information detection model according to embodiments of the present disclosure are described. For example... Figure 1 As shown, the training method for this multimodal information detection model may include the following steps:

[0050] In step S110, the multimodal training sample set and sample annotation information can be obtained.

[0051] Here, each training sample in the multimodal training sample set can include multimodal sample information, which can include image information and text information. The modality can include image modality and text modality. The sample annotation information can characterize the authenticity of the information of each training sample.

[0052] As an example, multimodal sample information can be, but is not limited to, multimodal news. Multimodal news can consist of text content and images. This multimodal information detection model can be used to detect the authenticity of multimodal information such as multimodal news, for example, whether multimodal news is fake news. Here, fake news can include situations such as falsified text content, falsified images, and text content not corresponding to images.

[0053] For example, given a multimodal dataset D = {(x i ,y i As the multimodal training sample set mentioned above, each training sample may include text content and a corresponding image. Each training sample may also have corresponding sample annotation information, which can be the true label y of the training sample. i ∈{0.1}, where i = 1, ..., n, and n is the total number of training samples. The goal of multimodal information detection is to output the probability of non-real information. Probability of true information Specifically in, The text and visual feature embeddings represent preprocessed text and visual features extracted by a pretrained model. For example, preprocessed text can be extracted from text content using existing Bidirectional Encoder Representations from Transformers (BERT) models, and visual features can be extracted from images using existing Residual Network (ResNet) models. However, the embodiments of text and visual feature extraction disclosed herein are not limited to these, and other methods or models can be used for extraction. This disclosure does not impose any particular limitations on these methods.

[0054] As an example, the general framework of a multimodal information detection model can be represented by the following equation (1):

[0055]

[0056] Where σ(·) is the Sigmoid function, θ represents the learnable parameters, t represents the text modality, v represents the image modality, c represents cross-modality, and f represents multimodal fusion.

[0057] In equation (1), This can represent a single-mode encoder, where m represents a single mode, for example, m can be an image mode v or a text mode t. Additional learning and transformation can be performed within each individual modality to better capture task-specific monomodal features.

[0058] ψ(·) can represent a cross-modal consistency learning module. ψ(·) can project single-modal pre-trained embeddings into a unified semantic space through contrastive learning to evaluate the degree of inconsistency between modalities. Taking fake news as an example, the fabricated text content of fake news generally lacks corresponding real images, inevitably leading to the use of irrelevant images. Using the module ψ(·) allows for the comparison of information from different modalities, determining the authenticity of the information by judging whether the information from different modalities is consistent. However, the module ψ(·) is not mandatory; it is generally used in hybrid fusion methods.

[0059] f(·) can represent a multimodal fusion module, which aims to fuse all acquired features and ultimately output a prediction. To fuse various modalities, for late-stage and hybrid fusion models, this module f(·) can be, for example, but not limited to, simple concatenation or weighted summation. In early-stage fusion models, f(·) can be, for example, but not limited to, a stack of cross-modal common attention blocks. After obtaining f(·), a multilayer perceptron (MLP) is typically used for final classification. To demonstrate this, we will use four powerful backbone methods employing the three fusion strategies described above in the following validation experiments.

[0060] The aforementioned general framework can be any existing multimodal information detection model, and its modules The specific forms of ψ(·) and f(·) can be set according to actual needs; for example, this disclosure does not impose any particular restrictions on them. The training method of the multimodal information detection model according to the embodiments of this disclosure can be applied to existing multimodal information detection models. For example, this method can be used as a plug-and-play module of the existing multimodal information detection backbone and inserted into the training process of the existing multimodal information detection model. This will be described in detail below with experimental examples.

[0061] In step S120, for each mode in the multimodal sample information, by masking the information of other modes outside of that mode in the multimodal sample information, a single-mode sample set corresponding to that mode can be obtained.

[0062] In related technologies, textual and visual features are typically fused before final prediction, making it impractical to separate unimodal predictions. To address this, in embodiments of this disclosure, unimodal predictions in a multimodal information detection model can be estimated by simulating the case of missing modalities. Specifically, other modalities can be masked to obtain a unimodal sample set containing only unimodal information for use in unimodal prediction.

[0063] As an example, such as Figure 2 As shown, for each modality in the multimodal sample information, the corresponding single-modal sample set can be obtained in the following way:

[0064] In step S210, the feature distribution of information of other modalities besides the given modality in the multimodal training sample set can be determined.

[0065] Here, feature distribution can refer to the data distribution of information in each modality. For example, for image modality, feature distribution can be the distribution of pixel values ​​in the image; for text modality, feature distribution can be the distribution of word vectors in the text.

[0066] In step S220, a masking function can be constructed based on the feature distribution.

[0067] Here, the masking function can characterize the information-free noise data of the feature distribution. For example, a Gaussian distribution that approximates the original input distribution can be constructed, which both keeps the distribution of other modal inputs within a reasonable range and masks the features of other modal information, ensuring that the sample set contains only information of a single modality.

[0068] In step S230, the padding data used to replace the other modal information can be determined based on the other modal information and the masking function.

[0069] As an example, taking the above equation (1) as an example, for any single mode m, the input of other modes besides the mode m can be replaced with information-free noise. For example, the padding data used to replace the information of other modes can be represented by the following equation (2):

[0070]

[0071] in, j represents the j-th mode among all modes except mode m, and k takes values ​​in the range [1, K], where K represents the feature length of each mode. A Gaussian distribution that approximates the original input distribution, i.e.

[0072] In step S240, the padding data can be used to replace the other modal information in the multimodal training sample set to obtain a single modal sample set.

[0073] In this step, for any single mode m, the single-mode prediction can be obtained by masking the input of other modes before the fusion stage. The framework of the single-mode prediction can be represented by the following equation (3):

[0074]

[0075] Among them, f s This represents the student model. The masking operation is denoted by "—", which indicates that the inputs of other modalities are replaced with information-free noise before modal fusion; that is, express The k-th dimension. When separating single-modal predictions, modal consistency features are always masked, such as... express The k-th dimension.

[0076] The above-described masking method achieves non-trivial masking. Compared to simple zero-padding, this masking method does not change the distribution of input data, thus avoiding the problem that simple zero-padding may cause to destroy the numerical stability of network parameters. In particular, for the BatchNorm layer in the model, the above masking method can maintain the numerical stability of the layer during operation, further improving the training effect.

[0077] However, the above-described shielding method is only a preferred embodiment, and the embodiments of this disclosure are not limited to this. For example, a simple zero-filling method can also be used, which can still solve the technical problems to be solved by this disclosure.

[0078] In step S130, the multimodal training sample set and the single-modal sample set corresponding to each modality can be input into the multimodal information detection model to obtain multimodal predictions for the multimodal training sample set and single-modal predictions for each single-modal sample set.

[0079] As an example, Figure 3 An example of a model training process according to an embodiment of the present disclosure is shown, in a multimodal information detection model, where unimodal predictions are obtained by masking inputs from other modalities before fusion. Figure 3 As shown, for a multimodal information detection model, the multimodal training sample set and each single-modal sample set can be input into the model to obtain the corresponding prediction results. For example, for the multimodal training sample set, a multimodal prediction can be obtained; for each single-modal training sample set, a corresponding single-modal prediction can be obtained. Figure 3 In this context, for image modalities, prediction results can be obtained from the image modal training sample set; for text modalities, prediction results can be obtained from the text modal training sample set.

[0080] In step S140, the prediction loss can be determined based on the multimodal prediction, each single-modal prediction, and sample labeling information.

[0081] In this step, the prediction loss can be obtained by fusing the loss from multimodal prediction and the loss from single-modal prediction, for example, by superimposing the various losses. Furthermore, according to embodiments of this disclosure, an example procedure for determining the prediction loss will be given below.

[0082] In step S150, the parameters of the multimodal information detection model can be adjusted using the prediction loss to train the multimodal information detection model.

[0083] The above training method allows for separate prediction of each single modality by masking other modalities. It also allows for the fusion of the loss from multimodal prediction and the loss from single-modal prediction. This approach ensures that while taking into account the learning of cross-modal correlation features between multimodalities, it also allows for the full learning of information from different single modalities, thereby improving the training effect of the model.

[0084] The following will combine Figures 4 to 8 Describe an example process for determining the predicted loss in step S140 above.

[0085] like Figure 4 As shown, the predicted loss can be determined in the following ways:

[0086] In step S410, the single-mode loss corresponding to each single-mode prediction can be determined based on each single-mode prediction.

[0087] In one example, the loss for single-modal prediction can be calculated using any existing loss calculation method, such as the loss function of an existing multimodal information detection model.

[0088] As another example, according to an embodiment of this disclosure, in step S410, for each unimodal prediction, the unimodal loss can be determined in the following manner: step S411, the unimodal teacher prediction of the pre-trained teacher model corresponding to the unimodal can be obtained; step S412, the unimodal loss can be determined based on the unimodal teacher prediction and the unimodal prediction.

[0089] Specifically, prediction-level knowledge distillation techniques can be used to enhance the learning performance of single-modal channels in a multimodal information detection model. In this example, the currently trained multimodal information detection model can serve as a student model.

[0090] In step S411, the unimodal teacher prediction can be obtained by inputting the unimodal sample set corresponding to the modality into the pre-trained teacher model. The value of the unimodal teacher prediction can be a real number in the range [0,1]. The training process of the teacher model will be described in detail below.

[0091] Here, the unimodal teacher prediction can be obtained during the training of the current multimodal information detection model, or it can be obtained in advance before training. In this step, the pre-obtained unimodal teacher prediction can be directly called.

[0092] Furthermore, as mentioned above, the value predicted by the unimodal teacher can be a real number in the range [0,1], which can represent the prediction probability of the teacher model, after being fully trained on the unimodal sample set, that the unimodal sample contains false information.

[0093] Specifically, the values ​​predicted by unimodal teachers are more granular than those predicted by multimodal information detection models. For example, the values ​​predicted by unimodal teachers can be continuously distributed within the range [0,1]. Unlike the truth labels in sample annotation information, they can take small values ​​between 0 and 1, rather than simply representing the prediction results with the binary values ​​"0" and "1". The magnitude of the value predicted by unimodal teachers is proportional to the degree of truthfulness (it can be directly or inversely proportional depending on the actual needs). For example, when a value of "0" indicates that the current sample is true information and a value of "1" indicates that the current sample is false information, a value of "0.5" can represent a neutral prediction result. Values ​​between "0" and "0.5" indicate that the prediction result is more inclined to indicate that the current sample is true information, and values ​​between "0.5" and "1" indicate that the prediction result is more inclined to indicate that the current sample is false information.

[0094] In step S412, the unimodal teacher prediction can be used to guide the unimodal prediction of the current multimodal information detection model. Specifically, the unimodal loss can be determined jointly based on both the unimodal teacher prediction of the teacher model and the unimodal prediction of the multimodal information detection model.

[0095] In step S412, for example, as Figure 5 As shown, the single-mode loss can be determined in the following way:

[0096] In step S510, the first loss can be determined based on the unimodal teacher prediction and the unimodal prediction.

[0097] Here, the first loss can be obtained based on the difference between the unimodal teacher prediction and the unimodal prediction.

[0098] In this step, since the same unimodal sample set can be used to obtain both the unimodal teacher prediction and the unimodal prediction, the difference between the unimodal teacher prediction and the unimodal prediction can be calculated for each training sample, and this difference can be statistically analyzed for all training samples to determine the first loss.

[0099] Here, the difference between unimodal teacher predictions and unimodal predictions can be represented by any mathematical method, such as using the mean squared error.

[0100] Specifically, when only valid information for a given modality m is provided, the mean squared error loss can be used to align the predictions of the teacher model and the multimodal information detection model (i.e., the student model). For example, the first loss can be determined by the following equation (4):

[0101]

[0102] in, Denotes the first loss of mode m. This represents the unimodal teacher prediction used for the teacher model in modality m. This represents the single-modal prediction of modality m by the multimodal information detection model, where n represents the number of samples in the single-modal sample set (or the multimodal training sample set).

[0103] It should be noted that the specific forms of the various formulas given in this article are only illustrative and can be adjusted according to actual needs. For example, the coefficients of each term can be adjusted or any term can be weighted. Alternatively, other forms of calculation formulas can be used. For example, the first loss of the above formula (4) can also be calculated using hinge loss, mean absolute error, cross entropy loss, etc.

[0104] In step S520, the second loss can be determined based on the single-modal prediction and sample labeling information.

[0105] In this step, to correct potential errors in the teacher model, the true labels of the samples can be used to guide the training of the single-modal channel. For example, the second loss can be determined by the following equation (5):

[0106]

[0107] in, This represents the second loss of mode m. This represents the label (e.g., 0 or 1) used for the i-th sample in the sample annotation information. This represents the single-modal prediction of modality m by the multimodal information detection model, where n represents the number of samples in the single-modal sample set.

[0108] In step S530, the single-mode loss can be determined based on the first loss and the second loss.

[0109] In this step, the first loss and the second loss can be fused to obtain the single-mode loss. As an example, the first loss and the second loss can be simply superimposed as the single-mode loss.

[0110] As another example, to balance the weighting relationship between the first and second losses, the first and / or second losses can be weighted. For example, the single-mode loss can be determined by the following equation (6):

[0111]

[0112] in, Let represent the single-mode loss for mode m, and β be a hyperparameter used to balance the weights between the two losses. Typically, considering that the teacher model is pre-trained and its prediction bias relative to the actual modality being manipulated is relatively small, a second loss is used during the stacking process to correct for potential errors in the teacher model. The corresponding weight β can take a relatively small value. In practical applications, the weight β can be set according to actual needs.

[0113] This disclosure reveals that, compared to feature-level knowledge distillation, the information in the prediction results is more compact. Therefore, prediction-level knowledge distillation does not impose too many restrictions on the learning of multimodal information detection models, enabling them to flexibly acquire necessary knowledge and be applicable to various network architectures. Thus, calculating the unimodal loss using the above method can optimize the training effect of unimodal channels.

[0114] In step S420, the multimodal loss corresponding to the multimodal prediction can be determined based on the multimodal prediction.

[0115] In this step, the multimodal loss can, for example, utilize the loss from existing multimodal joint training. For instance, in an example where the training method of a multimodal information detection model according to embodiments of this disclosure is inserted into the training process of an existing multimodal information detection model, the multimodal loss can be calculated using the loss function of the existing multimodal information detection model; or, in other words, the multimodal loss is the loss of the original training system. This disclosure does not impose any particular limitations on the calculation of the multimodal loss, and therefore will not be elaborated upon here.

[0116] In step S430, the prediction loss can be determined based on each single-mode loss and multi-mode loss.

[0117] In this step, the individual unimodal losses and multimodal losses can be fused to obtain the final prediction loss. By fusing the individual unimodal losses and multimodal losses, the predictive capabilities of each unimodal channel can be taken into account during the training of the model, thus balancing the learning between different modalities.

[0118] As an example, the individual unimodal losses and multimodal losses can be directly superimposed as the prediction loss. As another example, according to embodiments of this disclosure, the individual unimodal losses can be processed taking into account the convergence speed balance between different unimodals before being fused with the multimodal losses.

[0119] Specifically, the embodiments of this disclosure address the problem of inconsistent learning levels between different modalities, which hinders the improvement of model accuracy, by calculating the loss of each individual modality separately. Furthermore, this disclosure considers that since each modality contains different types of information, each modality exhibits a different learning speed. This may lead to overfitting in the dominant modality, while weak modalities remain in an underoptimized state, affecting the optimal overall performance of model training. It is also possible that this imbalance may cause the dominant modality to mask or even destroy the knowledge learned by weak modalities, i.e., mutual inhibition between modalities.

[0120] For example, when the modalities include image and text modalities, the model may converge faster in learning text content compared to image content. This could lead to the text modality becoming the dominant modality and the image modality becoming the weak modality, resulting in an imbalance between the single-modal learning of these two modalities. Here, the embodiments of this disclosure first balance the single-modal losses, increasing the weights corresponding to the weaker modal single-modal losses, and then fuse them with the multimodal losses. In this way, the utilization rate of various modalities can be dynamically balanced throughout the training process, further improving the problem of different learning levels between different modalities, thereby further improving the model's accuracy.

[0121] As an example, such as Figure 6 As shown, in step S430, the prediction loss can be determined in the following way:

[0122] In step S610, for each modality, the contribution of that modality to the multimodal information detection model can be determined, wherein the contribution is determined based on the degree to which the introduction of that modality improves the overall training accuracy of the multimodal information detection model under all possible modal sets other than that modality.

[0123] Specifically, for each modality, we can determine the extent to which the multimodal information detection model improves its overall training accuracy by further introducing that modal input, given an existing set of possible modal inputs. Here, for any given modality, we can determine the training accuracy of the multimodal detection model with other modal inputs besides that modality. Introducing information from the modality of interest will improve the multimodal training accuracy, differing from the model's training accuracy with other original modal inputs. By comparing the difference between these two levels of training accuracy, we can determine the contribution of the modality of interest to the overall model performance. The greater the difference in training accuracy, the greater the contribution of the modality of interest.

[0124] For example, for a modality set {A, B, C}, assuming the target modality is modality A, the other modalities besides modality A are modality B and modality C. The inputs of these other modalities can include four cases: {B}, {C}, {B, C}, and an empty set. The improvement of the model after adding modality A can be calculated in each of these four cases to measure the contribution of modality A. Specifically, V{A, B}-V{B}, V{A, C}-V{C}, V{A, B, C}-V{B, C}, and V{A}-V{empty set} can be calculated, where V{i} can represent the effect parameter characterizing the model's performance when the input is i, such as training accuracy. The calculation results of the effect parameter in these four cases can be combined to evaluate the contribution of modality A.

[0125] As an example, such as Figure 7 As shown, for any modality of interest (e.g., each modality), the degree of contribution can be determined in the following way:

[0126] In step S710, the first training accuracy of the multimodal information detection model under modal inputs other than the modality of interest, and the second training accuracy of the multimodal information detection model under the introduction of the modality of interest can be determined.

[0127] As an example, Shapley values ​​can be used to evaluate the contribution of individual unimodals in complex multimodal collaborations. Originating from cooperative game theory, Shapley values ​​aim to fairly measure each player's contribution to the collaboration. Unlike traditional game theory, which assumes players are not independent, Shapley values ​​take into account the cooperative relationships and mutual influences among players.

[0128] In this step, each modality can be considered as a player, and the value function V(S) can be defined as the training accuracy achieved by the model when it only receives input from modalities in set S, which can be expressed by the following equation (7):

[0129]

[0130] in, Let f represent a batch of training data, where f can be the multimodal information detection model f described above. s I(x) is a step function, that is, The Shapley value of mode m can be expressed by the following equation (8):

[0131]

[0132] Among them, V m (S) = V(S∪m) - V(S), which calculates the marginal contribution when the input of the mode of interest m is introduced in addition to the input of other existing modal inputs S. φ m This represents the weighted average of all marginal contributions of mode m. This represents the normalization factor.

[0133] Based on equations (7) and (8) above, the training accuracy of the multimodal information detection model under the original other modal inputs S, and the training accuracy of the multimodal information detection model after introducing the modality of interest input on this basis can be calculated. For example, when the model already contains an existing modal input, i.e., |S|=1, the first training accuracy V({v}) and V({t}) of the multimodal information detection model under the existing image modality and text modality, and the second training accuracy V({t,v}) of the multimodal information detection model after introducing the modality of interest can be calculated.

[0134] In step S720, a first comparison result can be obtained by comparing the first training accuracy and the second training accuracy.

[0135] In this step, for each modality, the difference between the first training accuracy and the second training accuracy can be calculated to obtain the first comparison result.

[0136] For example, for the text modality, the first comparison result can be represented as V({t, v}) - V({v}); for the image modality, the first comparison result can be represented as V({t, v}) - V({t}).

[0137] In step S730, a second comparison result can be obtained by comparing the third training accuracy of the multimodal information detection model under an empty training set (e.g., |S|=0 as described below) with the fourth training accuracy after introducing the modality of interest relative to the empty training set.

[0138] In this step, the difference between the third and fourth training accuracies can be calculated for each modality to obtain the second comparison result. Here, the empty training set refers to the training sample set without any modality information. For example, the masking method described above can be used to mask out the information of all modalities in the multimodal training sample set.

[0139] For example, in a bimodal example where the modalities include an image modality and a text modality, for the text modality, the second comparison result can be expressed as: For image modalities, the second comparison result can be expressed as:

[0140] In step S740, the contribution level of the mode of interest can be determined based on the first comparison result and the second comparison result.

[0141] In this step, statistical calculations can be performed on the first and second comparison results to determine the contribution of each modal multimodal detection model to the overall performance. For example, the average of the first and second comparison results can be used to characterize this contribution.

[0142] Taking the bimodal example above as an example, let's consider the contribution degree φ of the image modality. t and the degree of contribution to the text modality φ v This can be expressed by the following equations (9) and (10):

[0143]

[0144]

[0145] In step S620, the degree of underfitting for each modality can be determined based on the contribution level for each modality and the performance of the teacher model corresponding to each modality.

[0146] As an example, the amount of information carried by a modality can be measured based on the output of the teacher model. Specifically, the degree of underfitting corresponding to each single modality can be determined by: obtaining the single-modal teacher prediction of the pre-trained teacher model corresponding to that modality; and determining the degree of underfitting corresponding to each single modality based on the contribution of that modality to the multimodal detection system and the accuracy of the single-modal teacher prediction.

[0147] Here, as mentioned above, unimodal teacher prediction can be obtained by inputting the unimodal sample set corresponding to that modality into a pre-trained teacher model.

[0148] Specifically, the classification accuracy predicted by the unimodal teacher model can be compared with φ. m Combined, to estimate the degree of underoptimization (i.e., underfitting) γ of mode m.m :

[0149]

[0150] in, This represents the unimodal teacher prediction used for the teacher model in modality m. In the middle, in y i This indicates that the classification predicted by the unimodal teacher is correct. exist This indicates a classification error in the case of a unimodal teacher's prediction.

[0151] In step S630, the loss weight corresponding to each single-mode loss can be determined based on the degree of underfitting for each mode.

[0152] Loss weights can be used to weight the single-modal loss of each modality, and their role is to balance the convergence speed of each modality under multimodal joint training. For example, more weight can be given to modalities with more underfitting, that is, modalities that carry more information but contribute less to the multimodal detection system.

[0153] Here, the goal is not to maintain equal contributions from all modalities, but rather to consider the differences in information upper limits between different modalities. By combining the information upper limit of a modality with its current contribution to the multimodal detection system, we can identify the modality with the most severe underoptimization problem—that is, a modality with a high information upper limit but a low current contribution. For underoptimized modes, the weight of their single-modal loss should be increased to accelerate their convergence. Thus, the normalized loss weight λ of modality m can be obtained. m .

[0154]

[0155] It should be noted that the methods for determining loss weights are not limited to those mentioned above, and other methods can also be used. The purpose here is to introduce the concept of introducing loss weights to evaluate whether the information capacity carried by each single modality matches the current utilization rate of the modality in the multimodal detection system. The matching result is then incorporated into the final loss calculation. In this way, while considering the learning of each single modality, the information of each modality can be utilized to the maximum extent, further improving the training effect.

[0156] In step S640, the corresponding single-mode loss can be weighted based on the loss weight of each single-mode loss to determine the weighted single-mode loss.

[0157] In this step, the corresponding single-mode losses can be weighted using the aforementioned loss weights to obtain the weighted losses for each single mode.

[0158] In step S650, the prediction loss can be determined based on the weighted single-mode loss and multi-mode loss.

[0159] In this step, the weighted single-modal loss and multi-modal loss can be superimposed to obtain the prediction loss.

[0160] In one example, during the stacking process, the unimodal loss and / or multimodal loss can be weighted to balance the learning rate between the multimodal and unimodal losses. For example, the final prediction loss can be represented by the following equation (13):

[0161]

[0162] in, Let α represent the multimodal loss, where α is a hyperparameter balancing the classification loss and the unimodal loss learning rate. For example, as... Figure 3 As shown, when the modalities include image modality and text modality, the single-modal loss L of the text modality can be calculated separately. textual Single-mode loss L of image modality visual To obtain the final single-mode loss.

[0163] As described above, in the embodiments of this disclosure, it is understood that the model's utilization of different modalities can be balanced by measuring the contribution of a single modality to the model. A low contribution indicates low utilization, thus requiring a higher weight. Simultaneously, it is considered that different modalities carry different amounts of information, and each modality has an upper limit to its information content. This can be estimated using the accuracy of the teacher model; high teacher model accuracy indicates a high upper limit for the corresponding modality's information content. Therefore, the underfitting degree of a single modality can be obtained by the ratio of its upper limit to its contribution. Based on these two aspects, the higher the upper limit of the target modality's information content, and the lower its contribution to the multimodal model, the more underoptimized that modality is, requiring a higher weight, such as the aforementioned loss weight λ. m .

[0164] Based on the aforementioned prediction loss, multi-task learning can be used to simultaneously update the model through gradients from both the unimodal loss and the original multimodal classification loss, thereby optimizing the model's training.

[0165] The following will describe in detail a specific example of the training process for the teacher model for each modality mentioned above.

[0166] As an example, for each modality, the teacher model can be trained in the following way: obtain the single-modality sample set and sample labeling information mentioned above; input the single-modality sample set into the teacher model to obtain the teacher sample prediction; and train the teacher model based on the teacher sample prediction and sample labeling information.

[0167] Here, a single-modal sample set can include multiple single-modal training samples. The single-modal training samples can be divided into multiple sample clusters. Each sample cluster corresponds to a corresponding information topic. Different sample clusters correspond to different information topics, and there are no overlapping samples between different sample clusters.

[0168] As an example, such as Figure 8 As shown, a single-modal teacher prediction can be obtained in the following way: Step S810: For each sample cluster, the teacher model can be trained using other sample clusters besides the sample cluster to obtain a trained teacher model; Step S830: The sample cluster can be input into the trained teacher model to obtain a single-modal teacher prediction for the sample cluster.

[0169] Specifically, in the example of introducing a teacher model, the unimodal event-independent predictions of the teacher model can be obtained for modality-specific distillation. Here, the teacher model can use the same network structure as the student model to ensure generality. To enable the unimodal teacher model to focus on learning features from a specific modality, inputs from other modalities can still be masked during the training of the teacher model for modality m; that is, the unimodal sample set described above can be used to train the teacher model for the corresponding modality.

[0170] The teacher model's predictions can be represented by the following equation (14):

[0171]

[0172] in, Denotes the single-modal teacher prediction of mode m. Let m be the teacher model, and let m be the teacher model. Alternatively, binary cross-entropy loss can be used for training.

[0173] Generally, given false information such as rumors, it's likely that only one modality is manipulated, while other modalities appear normal. In this case, simply training each modality to correctly classify each sample can introduce a significant amount of noise. Therefore, to assign fine-grained tasks to each modality on a per-sample basis, the basic idea is to use the predictions of a unimodal teacher to distill multimodal students. However, a unimodal teacher trained directly on the entire training set can achieve near 100% training accuracy. Due to the limited size of the dataset, the model will inevitably memorize event-specific information from the training set, in addition to mining generalizable features. If such a domain-specific unimodal teacher model is directly used to teach a multimodal information detection model (i.e., the student model), the situation degenerates into distilling with hard labels (e.g., binary 0s and 1s), failing to provide more fine-grained predictive guidance (e.g., values ​​distributed in the range [0,1]) for the student model, thus losing the purpose of introducing the teacher model.

[0174] In response, when training the teacher model, to eliminate spurious connections related to event-related information, such as... Figure 9 As shown, the training dataset can be divided into k non-overlapping domains. Each domain can be viewed as a cluster of samples with similar themes. For example, in Figure 9 In the dataset Weibo21, official domain labels are provided, covering 9 domains. For the GossipCop dataset, KNN clustering can be performed using sentence embeddings generated based on Sentence BERT to divide the samples into 10 domains. Then, the teacher model is trained and validated on different domains of the training dataset using cross-validation. In the i-th iteration, d i The samples in the sample are retained for verification and used The samples in the dataset are used to train a randomly initialized teacher model from scratch. Then, teachers outside the event domain For in d i The prediction of the sample above is denoted as Where, j∈d i This can be viewed as event-independent knowledge of the teacher model about unseen events. This process is repeated k times to obtain the complete set of teacher predictions for the training dataset. These predictions can be stored as event-independent labels on disk for retrieval and use during unimodal distillation.

[0175] By using the above methods, a teacher model with more granular predicted labels can be trained, which can better guide the training of multimodal information detection models.

[0176] According to the training method of the multimodal information detection model according to embodiments of the present disclosure, a fully trained unimodal teacher model can be used, and then the features learned by the teacher model can be fully transferred to the multimodal information detection model as a student model through a modality-level knowledge distillation method. Specifically, the unimodal / multimodal output of the model is obtained by changing the input mode (unimodal / multimodal) of the multimodal information detection model, and the original multimodal joint loss function is still used for optimization of the multimodal output of the model.

[0177] Furthermore, this method uses the output of the unimodal teacher model to distill the unimodal output of the multimodal student model. Through this operation, the teacher model can supplement the missing unimodal knowledge in the student model, thereby achieving full learning of unimodal features.

[0178] Furthermore, to ensure the accuracy of the knowledge imparted by the teacher model—that is, to ensure it is unaffected by specific event information in the dataset—the original dataset can be divided into multiple unrelated event domains. The teacher model is then trained using cross-validation across different event domains. Finally, the output of the teacher model on the test set is used as the knowledge distilled from the student model. Predictions from the cross-domain teacher model can be considered event-irrelevant knowledge that is not associated with specific event information.

[0179] Furthermore, in this method, in order to balance the convergence speed among modes, the participation of each mode is dynamically monitored during training. In this way, more underoptimized modes can be identified and the single-mode distillation loss can be reweighted accordingly.

[0180] In a second aspect of exemplary embodiments of this disclosure, a method for detecting multimodal information is provided.

[0181] The multimodal information detection method according to exemplary embodiments of the present disclosure can be applied to scenarios where users interact with software. For example, software can be loaded on a user terminal, and the user terminal can acquire target multimodal information, which includes image information and text information. The user terminal can input the target multimodal information into a multimodal information detection model to obtain a detection result for the authenticity of the multimodal information. Here, the multimodal information detection model can be trained using the training method of the multimodal information detection model according to exemplary embodiments of the present disclosure.

[0182] The aforementioned user terminal can be such as a tablet computer, laptop computer, digital assistant, wearable device, etc. However, the implementation scenario of the training method described above is only an example scenario. The training method of the visual question answering model according to the exemplary embodiment of this disclosure can also be applied to other application scenarios. For example, the user can request the model to be trained from the server via the network on the user terminal (e.g., mobile phone, desktop computer, tablet computer, etc.). The server can complete the question answering by executing the visual question answering method according to the exemplary embodiment of this disclosure. Here, the server can be an independent server, a server cluster, a cloud computing platform, or a virtualization center.

[0183] The multimodal information detection method according to the exemplary embodiments of the present disclosure can more accurately detect the authenticity of multimodal information based on a fully trained multimodal information detection model.

[0184] like Figure 10 As shown, the multimodal information detection method may include: step S1010, acquiring target multimodal information, the target multimodal information including image information and text information; step S1020, inputting the target multimodal information into a multimodal information detection model to obtain a detection result for the authenticity of the multimodal information, wherein the multimodal information detection model may be trained by a training method for a multimodal information detection model according to an exemplary embodiment of the present disclosure.

[0185] The multimodal information detection method according to the exemplary embodiments of this disclosure can be implemented in a similar manner to the training method of the multimodal information detection model described above with reference to the first aspect. For example, in the training method, various sample data and training data are involved, while in the multimodal information detection method, it can be the actual task data corresponding to the various sample data and training data. Therefore, it is understood that the details of this multimodal information detection method can be the same as the details in the training method of the multimodal information detection model described above with reference to the training method. Accordingly, this multimodal information detection method has the same or similar beneficial effects as the training method of the multimodal information detection model described above, which will not be repeated here.

[0186] Based on the above training and detection methods, embodiments of this disclosure propose a unimodal event-agnostic knowledge distillation (UEKD) method. The model architecture of UEKD is as follows: Figure 3 As shown in the figure. Here, taking news as an example, we conducted relevant experiments on the authenticity detection of information using this method to verify its effectiveness.

[0187] In the experimental setup, the Chinese dataset Weibo21 and the English dataset GossipCop were used for evaluation. The significant imbalance in the number of positive and negative samples in GossipCop (over 80% of the samples are real news) could severely overestimate model performance, masking the modality imbalance problem. Therefore, all fake news was retained, and real news was downsampled to achieve a balanced distribution. Thus, the GossipCop dataset contained 2036 real news items and 2036 fake news items for training, and 545 real news items and 545 fake news items for testing. The Weibo21 dataset contained a total of 4640 real news items and 4487 fake news items, which were divided into training and testing sets in an 8:2 ratio. During the experiment, it was ensured that the posts in the training and testing sets came from different events to accurately evaluate the model's generalization ability to future events.

[0188] To evaluate the effectiveness of the method proposed in this disclosure, it can be embedded into four existing powerful multimodal fake news detection backbone models.

[0189] SpotFake+, a multimodal fake news detection method based on transfer learning, can be regarded as a simple backbone model. Textual and visual features extracted from the pre-trained model are concatenated after being projected onto the same dimension, and then MLP is used to fuse multimodal features and produce the final prediction result.

[0190] Multimodal Fusion with Co-Attention Networks (MCAN) takes into account both spatial and frequency domain features of an image as visual features, and then stacks multiple co-attention layers to learn fine-grained correlations across modalities.

[0191] The Hierarchical Multi-modal Contextual Attention Network (HMCAN) is one of the state-of-the-art methods. It leverages fine-grained features from pre-trained models to enhance unimodal representations and further captures intermodal and intramodal relationships through a contextual attention network.

[0192] The Cross-Modal Ambiguity-aware Fake News Detection Method (CAFE) is one of the most advanced methods. It adaptively relies on unimodal features and cross-modal correlations to make judgments based on the intensity of cross-modal ambiguity.

[0193] To verify the superior effectiveness of the algorithm proposed in this disclosure, experiments were also conducted here for comparison with several existing algorithms aimed at achieving balanced multimodal learning.

[0194] Gradient Blending (GB) aims to address inconsistent convergence between different modal encoders by training each encoder with an additional classifier. It simultaneously optimizes both single-modal and multimodal classifiers and reweights the loss based on their overfitting.

[0195] On-the-fly Gradient Modulation (OGM) incorporates dynamic adjustment of gradient magnitudes during training to control the convergence speed of different modal encoders. Before updating network parameters, the algorithm adaptively multiplies the gradients of different modalities by different coefficients based on their relative performance to balance their optimization speed.

[0196] Adaptive Gradient Modulation (AGM) is an extension of OGM, which proposes to approximate the overall multimodal output into single-modal outputs for each mode. During backpropagation, it partitions the overall gradient across modes and further modulates them to balance the learning speed.

[0197] Based on the aforementioned backbone model and baseline method, in order to fairly compare and alleviate the limitations of outdated unimodal pre-trained feature extractors on the upper limit of model capabilities, the same pre-trained model can be used uniformly to extract pre-processed text and visual features, and the feature dimensions can be adaptively adjusted to meet the original requirements of each backbone model.

[0198] Specifically, for the text modality, the bert-base-chinese model (a version of BERT pre-trained for Chinese text) was used to process the Weibo21 dataset, and the bert-base-uncased model (a version of BERT pre-trained for case-insensitive English text) was used to process the GossipCop dataset. The input text length for Weibo2 was set to 120 characters, and for GossipCop, it was set to 200 characters. For the visual modality, a ResNet50 model pre-trained with CLIP was used. Both pre-trained models were kept frozen during training. All backbone methods are open source, and publicly released code was used for experiments. The algorithm can be implemented in PyTorch and trained on an NVIDIA RTX3090 GPU. A batch size of 32 was used in the experiments, and the Adam optimizer was used for training with a learning rate of 0.0002. The model was trained for 150 epochs, and an early stopping strategy was used when the test accuracy did not improve within 30 epochs.

[0199] Figure 11A The overall experimental results are shown, and the evaluation metrics include classification accuracy (Acc), precision (P), recall (R), and F1 score.

[0200] like Figure 11A As shown in Table 1, the overall performance of the proposed UEKD method and several existing baseline methods is presented in this disclosure. From this table, the following observations can be drawn:

[0201] (1) Compared with baseline methods, UEKD shows a significant improvement. On the Weibo21 dataset and the GossipCop dataset, the method achieves an average classification accuracy improvement of 2.2% and 5.3%, respectively. In contrast, existing baseline methods show little improvement and even lead to performance degradation because they do not focus on how to solve the modality imbalance problem in the modality fusion stage.

[0202] (2) UEKD focuses on the input and output of the backbone model, thus it is applicable to various fusion strategies. However, the GB and OGM algorithms are only applicable to late-stage fusion methods. Although AGM is applicable to early-stage fusion models, it still faces two limitations that render it ineffective for backbone models. First, it still relies on a single multimodal training objective, which is insufficient when all modalities are not optimized. Second, gradient modulation becomes ineffective when faced with modal coupling during the modal fusion stage.

[0203] (3) A significant improvement can be observed for SpotFake+, which is due to its relatively weak dominant modality capability. Therefore, supplementing knowledge from weak modalities leads to a significant improvement. However, SpotFake+ using UEKD achieves performance comparable to the state-of-the-art method HMCAN, highlighting the importance of fully utilizing single-modal features. Similarly, UEKD achieves a more significant improvement on the GossipCop dataset, as relying on a single modality on the GossipCop dataset is often insufficient to achieve good results.

[0204] Figure 11B The single-modal performance analysis of the multimodal information detection model is shown, and the evaluation index includes classification accuracy, where Δimprov represents the percentage improvement in performance relative to the original model.

[0205] The UEKD proposed in this disclosure was investigated in experiments to determine whether it could alleviate the modal imbalance problem. To evaluate the unimodal performance in the multimodal model, a masking operation was used to obtain unimodal predictions during the testing phase. For comparison, the unimodal performance in several model variants is reported: (1) the original backbone model obtained through multimodal joint training; and (2) the backbone model with UEKD or other baselines inserted. Experimental results are as follows: Figure 11B As shown, according to Figure 11B The following observations can be obtained:

[0206] (1) The UEKD proposed in this disclosure significantly improves the performance of weak modes in multimodal models. This shows that UEKD can completely alleviate the modal imbalance problem, enabling the backbone model to make full use of information from all modes.

[0207] (2) GB improves the performance of weak modes to some extent. However, it is not enough to simply train the single-modal encoder, because single-modal knowledge may be destroyed after fusion.

[0208] (3) OGM and AGM still encounter difficulties in solving the modal optimization problem. Although gradient modulation improves the performance of weak modes, this comes at the cost of compressing the optimization space of the dominant mode, which in turn leads to a decrease in the performance of the dominant mode. In addition, AGM may exacerbate the modal coupling problem, as the HMCAN algorithm with AGM performs poorly with single-modal input.

[0209] (4) Different backbone models on the same dataset have different weak modes, which indicates that dataset characteristics and model architecture can affect the convergence speed of a certain mode. Therefore, it is necessary to design an adaptive learning speed to balance various modes.

[0210] Based on the above experiments, it can be seen that the method proposed in this disclosure innovatively solves the modality imbalance problem that was not considered in previous multimodal fake news detection algorithms. Furthermore, it significantly improves the modality imbalance problem in previous methods through a simple and easy-to-use training framework, thereby further enhancing model performance. When different multimodal fake news detection methods are incorporated, the method proposed in this disclosure can improve the performance of existing methods, and can also improve the optimization of weak modalities and achieve balanced multimodal learning.

[0211] Figure 12 This is a schematic block diagram of a training system for a multimodal information detection model according to exemplary embodiments of the present disclosure. Figure 12 As shown, the training system of the multimodal information detection model may include a training acquisition unit 1210, a masking unit 1220, a single-modal prediction unit 1230, a loss determination unit 1240, and a training unit 1250.

[0212] The training acquisition unit 1210 can be configured to acquire a multimodal training sample set and sample annotation information, wherein each training sample in the multimodal training sample set includes multimodal sample information, which includes image information and text information, and the modality includes image modality and text modality. The sample annotation information characterizes the authenticity of the information of each training sample.

[0213] The shielding unit 1220 can be configured to obtain a single-modal sample set corresponding to each modality in the multimodal sample information by shielding the information of other modalities outside the current modality in the multimodal sample information.

[0214] The single-modal prediction unit 1230 can be configured to input a multimodal training sample set and a single-modal sample set corresponding to each modality into a multimodal information detection model, respectively, so as to obtain multimodal predictions for the multimodal training sample set and single-modal predictions for each single-modal sample set.

[0215] The loss determination unit 1240 can be configured to determine the prediction loss based on multimodal prediction, each single-modal prediction, and sample labeling information.

[0216] The training unit 1250 can be configured to use the prediction loss to adjust the parameters of the multimodal information detection model in order to train the multimodal information detection model.

[0217] As an example, for each modality in the multimodal sample information, the masking unit 1220 can be configured to obtain the single-modal sample set corresponding to that modality by: determining the feature distribution of other modal information in the multimodal training sample set besides that modality; constructing a masking function based on the feature distribution, wherein the masking function can characterize the information-free noise data of the feature distribution; determining padding data to replace other modal information based on other modal information and the masking function; and replacing other modal information in the multimodal training sample set with the padding data to obtain the single-modal sample set.

[0218] As an example, the loss determination unit 1240 can be configured to determine the prediction loss by: determining the single-modal loss corresponding to each single-modal prediction based on each single-modal prediction; determining the multimodal loss corresponding to the multimodal prediction based on the multimodal prediction; and determining the prediction loss based on each single-modal loss and the multimodal loss, wherein, for each single-modal prediction, the single-modal loss is determined by: obtaining the single-modal teacher prediction of the pre-trained teacher model corresponding to that modality, wherein the single-modal teacher prediction is obtained by inputting the single-modal sample set corresponding to that modality into the pre-trained teacher model, and the value of the single-modal teacher prediction is a real number in the range [0,1]; and determining the single-modal loss based on the single-modal teacher prediction and the single-modal prediction.

[0219] As an example, the loss determination unit 1240 can also be configured to: determine a first loss based on the unimodal teacher prediction and the unimodal prediction; determine a second loss based on the unimodal prediction and sample labeling information; and determine a unimodal loss based on the first loss and the second loss.

[0220] As an example, the single-modal training samples in the single-modal sample set are divided into multiple sample clusters, each corresponding to a specific information topic. Different sample clusters correspond to different information topics, and there are no overlapping samples between different sample clusters. The single-modal teacher prediction is obtained as follows: for each sample cluster, the teacher model is trained using other sample clusters besides that sample cluster to obtain a trained teacher model; the sample cluster is then input into the trained teacher model to obtain the single-modal teacher prediction for that sample cluster.

[0221] As an example, the loss determination unit 1240 can also be configured to: for each modality, determine the contribution of that modality to the multimodal information detection model, wherein the contribution is determined based on the improvement of the overall training accuracy of introducing that modality into the multimodal information detection model under all possible modal sets other than that modality; determine the underfit of each modality based on the contribution of each modality and the performance of the teacher model corresponding to each modality; determine the loss weight corresponding to each single-modal loss based on the underfit of each modality; weight the corresponding single-modal losses based on the loss weights of each single-modal loss to determine the weighted single-modal loss; and determine the prediction loss based on the weighted single-modal loss and the multimodal loss.

[0222] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0223] Figure 13 This is a schematic block diagram of a multimodal information detection system according to exemplary embodiments of the present disclosure. Figure 13 As shown, the multimodal information detection system may include an information acquisition unit 1310 and an information detection unit 1320.

[0224] The information acquisition unit 1310 can be configured to acquire target multimodal information, which includes image information and text information.

[0225] The information detection unit 1320 can be configured to input target multimodal information into a multimodal information detection model to obtain a detection result for the authenticity of the multimodal information. Here, the multimodal information detection model is trained according to the training method of the multimodal information detection model of the first aspect of this disclosure.

[0226] The information acquisition unit 1310 and the information detection unit 1320 can, as described above Figure 10 The multimodal information detection method in the method embodiment shown executes the corresponding steps in the method. The specific implementation of the information acquisition unit 1310 and the information detection unit 1320 can be found in the method embodiment described above, and will not be repeated here.

[0227] Figure 14 This is a block diagram of an electronic device according to exemplary embodiments of the present disclosure. Figure 14 As shown, the electronic device 10 includes a processor 101 and a memory 102 for storing processor-executable instructions. Here, when executed by the processor, the processor-executable instructions cause the processor to execute a training method for a multimodal information detection model or a multimodal information detection method as described in the exemplary embodiments above.

[0228] As an example, electronic device 10 is not necessarily a single device, but can be a collection of any means or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 10 can also be part of an integrated control system or system manager, or can be configured to interconnect with a server locally or remotely (e.g., via wireless transmission).

[0229] In electronic device 10, processor 101 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor 101 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0230] The processor 101 can execute instructions or code stored in the memory 102, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transmission protocol.

[0231] The memory 102 may be integrated with the processor 101, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 102 may include a separate device, such as an external disk drive, a storage array, or other storage device that can be used by any database system. The memory 102 and the processor 101 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 101 to read files stored in the memory 102.

[0232] In addition, the electronic device 10 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 10 may be interconnected via a bus and / or network.

[0233] In an exemplary embodiment, a computer-readable storage medium may also be provided, which, when executed by a processor of an electronic device, enables the electronic device to perform a training method for a multimodal information detection model or a multimodal information detection method as described in the exemplary embodiments above. The computer-readable storage medium may be, for example, a memory including instructions. Optionally, the computer-readable storage medium may be: a read-only memory (ROM), a random access memory (RAM), a random access programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), flash memory, non-volatile memory, a CD-ROM, a CD-R, a CD+R, a CD-RW, a CD+RW, a DVD-ROM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a DVD-RAM, a BD-ROM, or a B-ROM. D-R, BD-RLTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0234] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, the computer program product including computer-executable instructions, which, when executed by at least one processor, implement a training method or a multimodal information detection method for a multimodal information detection model according to exemplary embodiments of the present disclosure.

[0235] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0236] Furthermore, it should be noted that although several examples of each step have been described above with reference to the specific accompanying drawings, it should be understood that the embodiments of this disclosure are not limited to the combinations given in the examples, and the steps appearing in different drawings can be combined, which will not be exhaustive here.

[0237] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for a multimodal information detection model, characterized in that, The training method includes: Obtain a multimodal training sample set and sample annotation information, wherein each training sample in the multimodal training sample set includes multimodal sample information, the multimodal sample information includes image information and text information, the modality includes image modality and text modality, and the sample annotation information characterizes the information authenticity of each training sample; For each mode in the multimodal sample information, by masking out information of other modes outside of that mode in the multimodal sample information, a single-modal sample set corresponding to that mode is obtained; The multimodal training sample set and the single-modal sample set corresponding to each modality are respectively input into the multimodal information detection model to obtain multimodal predictions for the multimodal training sample set and single-modal predictions for each single-modal sample set. Based on the multimodal prediction, each single-modal prediction, and the sample labeling information, the prediction loss is determined; The parameters of the multimodal information detection model are adjusted using the predicted loss to train the multimodal information detection model. The prediction loss is determined as follows: based on each unimodal prediction, a unimodal loss corresponding to each unimodal prediction is determined; based on the multimodal prediction, a multimodal loss corresponding to the multimodal prediction is determined; based on each unimodal loss and the multimodal loss, the prediction loss is determined, wherein, for each unimodal prediction, the unimodal loss is determined as follows: obtaining the unimodal teacher prediction of a pre-trained teacher model corresponding to that modality, wherein the unimodal teacher prediction is obtained by inputting the unimodal sample set corresponding to that modality into the pre-trained teacher model, and the value of the unimodal teacher prediction is a real number in the range [0,1]; based on the unimodal teacher prediction and the unimodal prediction, the unimodal loss is determined. The single-modal training samples in the single-modal sample set are divided into multiple sample clusters, each corresponding to a specific information topic. Different sample clusters correspond to different information topics, and there are no overlapping samples between different sample clusters. The single-modal teacher prediction is obtained as follows: for each sample cluster, the teacher model is trained using other sample clusters besides that sample cluster to obtain a trained teacher model; the sample cluster is then input into the trained teacher model to obtain the single-modal teacher prediction for that sample cluster.

2. The training method according to claim 1, characterized in that, For each mode in the multimodal sample information, the corresponding single-modal sample set is obtained in the following way: Determine the feature distribution of modal information other than the given modality in the multimodal training sample set; Based on the feature distribution, a masking function is constructed, wherein the masking function can characterize the information-free noise data of the feature distribution; Based on the other modal information and the masking function, determine the padding data to replace the other modal information; The other modality information in the multimodal training sample set is replaced with the padding data to obtain the single-modal sample set.

3. The training method according to claim 1, characterized in that, The step of determining the unimodal loss based on the unimodal teacher prediction and the unimodal prediction includes: Based on the unimodal teacher prediction and the unimodal prediction, a first loss is determined; Based on the single-modal prediction and the sample annotation information, a second loss is determined; The single-mode loss is determined based on the first loss and the second loss.

4. The training method according to claim 1, characterized in that, The step of determining the prediction loss based on each single-modal loss and the multimodal loss includes: For each modality, the contribution of that modality to the multimodal information detection model is determined, wherein the contribution is determined based on the degree to which the introduction of that modality improves the overall training accuracy of the multimodal information detection model in all possible modal sets other than that modality; Based on the contribution level of each modality and the performance of the teacher model corresponding to each modality, the underfit degree of each modality is determined; Based on the degree of underfitting for each mode, a loss weight corresponding to each single-mode loss is determined; Based on the loss weights of each single-mode loss, the corresponding single-mode losses are weighted to determine the weighted single-mode loss; The prediction loss is determined based on the weighted single-modal loss and the multimodal loss.

5. A method for detecting multimodal information, characterized in that, The multimodal information detection method includes: Acquire target multimodal information, which includes image information and text information; The target multimodal information is input into a multimodal information detection model to obtain a detection result of the information authenticity of the multimodal information, wherein the multimodal information detection model is trained by the training method of the multimodal information detection model according to any one of claims 1 to 4.

6. A training system for a multimodal information detection model, characterized in that, The training system includes: The training acquisition unit is configured to acquire a multimodal training sample set and sample annotation information, wherein each training sample in the multimodal training sample set includes multimodal sample information, the multimodal sample information includes image information and text information, the modality includes image modality and text modality, and the sample annotation information characterizes the information authenticity of each training sample; The masking unit is configured to, for each modality in the multimodal sample information, obtain a single-modal sample set corresponding to that modality by masking information of other modalities outside that modality in the multimodal sample information; The single-modal prediction unit is configured to input the multimodal training sample set and the single-modal sample set corresponding to each modality into the multimodal information detection model, respectively, so as to obtain multimodal prediction for the multimodal training sample set and single-modal prediction for each single-modal sample set. The loss determination unit is configured to determine the prediction loss based on the multimodal prediction, each single-modal prediction, and the sample labeling information; The training unit is configured to use the prediction loss to adjust the parameters of the multimodal information detection model in order to train the multimodal information detection model. The loss determination unit is configured to determine the prediction loss in the following ways: based on each unimodal prediction, determine the unimodal loss corresponding to each unimodal prediction; based on the multimodal prediction, determine the multimodal loss corresponding to the multimodal prediction; and based on each unimodal loss and the multimodal loss, determine the prediction loss, wherein, for each unimodal prediction, the unimodal loss is determined in the following way: obtaining the unimodal teacher prediction of a pre-trained teacher model corresponding to that modality, wherein the unimodal teacher prediction is obtained by inputting the unimodal sample set corresponding to that modality into the pre-trained teacher model, and the value of the unimodal teacher prediction is a real number in the range [0,1]; and based on the unimodal teacher prediction and the unimodal prediction, determine the unimodal loss. The single-modal training samples in the single-modal sample set are divided into multiple sample clusters, each corresponding to a specific information topic. Different sample clusters correspond to different information topics, and there are no overlapping samples between different sample clusters. The single-modal teacher prediction is obtained as follows: for each sample cluster, the teacher model is trained using other sample clusters besides that sample cluster to obtain a trained teacher model; the sample cluster is then input into the trained teacher model to obtain the single-modal teacher prediction for that sample cluster.

7. A multimodal information detection system, characterized in that, The multimodal information detection system includes: The information acquisition unit is configured to acquire target multimodal information, which includes image information and text information; The information detection unit is configured to input the target multimodal information into a multimodal information detection model to obtain a detection result for the authenticity of the multimodal information. The multimodal information detection model is trained using the training method for the multimodal information detection model according to any one of claims 1 to 4.

8. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions. Wherein, when the processor executes the processor, it causes the processor to execute the training method of the multimodal information detection model according to any one of claims 1 to 4 or the multimodal information detection method according to claim 5.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the training method of the multimodal information detection model according to any one of claims 1 to 4 or the multimodal information detection method according to claim 5.

10. A computer program product comprising computer-executable instructions, characterized in that, When the computer-executable instructions are executed by at least one processor, they implement the training method for the multimodal information detection model according to any one of claims 1 to 4 or the multimodal information detection method according to claim 5.

Images