Video false information cross-domain detection method and system under incomplete modal condition

Abstract

The application discloses a video false information cross-domain detection method and system under a modal incomplete condition, and relates to the technical field of artificial intelligence, comprising: acquiring available multi-modal data in video news samples from different sources; processing the available multi-modal data to generate a modal missing mask; projecting the available multi-modal data to a shared semantic space based on linear mapping to acquire shared semantics, and performing feature reconstruction on the shared semantics according to a preset residual autoencoder model and the modal missing mask to acquire completed modal features; and processing the completed modal features based on a hybrid expert mechanism to output a prediction probability of a false video. The application can significantly improve the accuracy and cross-domain adaptability of video false information detection in complex real scenes with serious modal loss or significant domain differences, and has good robustness, scalability and wide practical application prospects.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method and system for cross-domain detection of video misinformation under modal incompleteness conditions. Background Technology

[0002] Existing methods for detecting fake videos largely rely on multimodal data fusion, which comprehensively utilizes video frame images, audio signals, and text descriptions for joint analysis to improve detection accuracy and robustness. While multimodal fusion can uncover complementary information between different modalities, in practical applications, multimodal data often suffers from incompleteness or low quality. For example, some videos may lack audio tracks, have missing text descriptions, or exhibit degraded video quality, limiting the performance of traditional multimodal models. Furthermore, fake videos exhibit significant distribution differences across different platforms, topics, and regions, resulting in insufficient model generalization ability and difficulty adapting to complex cross-domain environments.

[0003] Therefore, there is an urgent need to provide a solution to improve the above problems. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for cross-domain detection of video misinformation under modal incomplete conditions, which can solve the problem of low accuracy and robustness of existing technologies in detecting video misinformation.

[0005] To solve the above-mentioned technical problems, the present invention is implemented as follows:

[0006] In a first aspect, the present invention provides a method for cross-domain detection of video misinformation under modal incompleteness conditions, comprising:

[0007] Obtain available multimodal data from video news samples from different sources, including video frames, audio, text descriptions, and related images;

[0008] The available multimodal data is processed to generate a modality missing mask;

[0009] Based on linear mapping, available multimodal data is projected onto a shared semantic space to obtain shared semantics, and the shared semantics are reconstructed according to a preset residual autoencoder model and a modality missing mask to obtain the completed modal features.

[0010] The completed modal features are processed using a hybrid expert mechanism to output the predicted probability of fake videos.

[0011] Optionally, before obtaining the predicted probability of fake videos, end-to-end training is required, including: constructing a retrieval enhancement contrastive loss function based on the modal fusion representation of video news samples in a shared semantic space; obtaining a joint optimization objective function based on the retrieval enhancement contrastive loss function, a pre-defined modal consistency reconstruction loss function, and a pre-defined classification loss function; establishing a multi-source domain joint loss function based on the joint optimization objective function; and performing end-to-end training based on the joint optimization objective function and the multi-source domain joint loss function to obtain the predicted probability of fake videos.

[0012] Optionally, the process of processing available multimodal data to generate a modality missing mask includes: performing format unification and feature extraction processing on the multimodal data to obtain an initial modality feature representation; performing modality missing detection on the initial modality feature representation, recording the missing modalities through a flag variable, and generating a judgment result; generating a modality missing mask based on the judgment result, wherein the modality missing mask represents the presence or absence of each modality in the current sample in binary vector form, and is used for reference and control in the subsequent modality completion and feature fusion process.

[0013] Optionally, the specific steps for constructing the retrieval enhancement contrast loss function include: calculating the semantic similarity between the current video news sample and other video news samples in the same batch based on the modal fusion representation of the current video news sample in the shared semantic space; in each training batch, retrieving positive samples with the same label and the closest semantics to the current video news sample, and negative samples with different labels but similar semantics, based on semantic similarity, to construct positive and negative contrast sample pairs; and establishing the retrieval enhancement contrast loss function based on a variant of temperature-regulated InfoNCE loss to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs, and introducing a memory replay mechanism.

[0014] Optionally, the memory replay mechanism is used to store the semantic representations and labels of historical training samples, and in each training step, it combines the current batch samples and the samples in the memory queue to perform positive and negative sample retrieval and comparative learning, expanding the diversity of sample pairs. The mathematical expression is:

[0015] ;

[0016] in, This indicates the index of the sample currently being trained. Indicates the current candidate sample, This represents the current set of candidate samples, which can come from the current training samples, or from the memory or cache. Indicates sample The true category label, Indicates sample The unified modal semantic representation vector, Indicates sample Compared with the current candidate samples semantic similarity function, Indicates sample The original set of positive samples, i.e., with A collection of samples with the same labels Indicates sample The original negative sample set, i.e., with A collection of samples with different labels This represents the enhanced set of positive samples. This represents the enhanced set of negative samples. For the set threshold, Represents the union, Indicates sample The true category label.

[0017] Optionally, the process of processing the completed modal features based on a hybrid expert mechanism to output the predicted probability of fake videos includes: constructing an expert set consisting of multiple expert sub-networks, where each expert sub-network performs specialized modeling for different modal combinations or semantic features to capture cross-modal differences and complementarities; processing the multimodal fusion representation after modal completion, the cross-sample transfer features generated by retrieval-enhanced contrastive learning, and residual information based on the expert set to obtain intermediate prediction results and feature embeddings corresponding to each expert sub-network; calculating the output weights of each expert sub-network based on the intermediate prediction results, feature embeddings, and preset weights; weighting and fusing the output results of each expert sub-network based on the output weights to obtain fused features; and performing binary classification on the fused features based on a classifier to output the predicted probability of fake videos.

[0018] Optionally, the mathematical expression for the preset weights is:

[0019] ;

[0020] in, Indicates the first Experts on the sample The weight, It integrates contextual information. It is a multilayer perceptron. It is the encoded number Hidden layer features It is the encoded number Hidden layer features Indicates the number of experts.

[0021] Secondly, the present invention provides a cross-domain detection system for video misinformation under modal incompleteness conditions, comprising:

[0022] The multimodal data acquisition module is used to acquire available multimodal data from video news samples from different sources. The available multimodal data includes video frames, audio, text descriptions, and related images.

[0023] The modality missing mask generation module is used to process available multimodal data to generate a modality missing mask;

[0024] The missing modality feature reconstruction module is used to project available multimodal data onto a shared semantic space based on a linear mapping to obtain shared semantics, and to reconstruct the features of the shared semantics according to a preset residual autoencoder model and a modality missing mask to obtain the completed modality features.

[0025] The fake video result generation module is used to process the completed modal features based on a hybrid expert mechanism to output the predicted probability of fake videos.

[0026] Thirdly, embodiments of the present invention provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method as described in the first aspect.

[0027] Fourthly, embodiments of the present invention provide a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, the steps of the method as described in the first aspect are implemented.

[0028] Compared with existing technologies, the beneficial effects of the cross-domain detection method for video misinformation under modal incompleteness conditions provided by this invention are as follows:

[0029] First, it enhances the ability to handle missing modalities: by accurately identifying missing modalities through the missing modality perception module, and combining modal consistency learning and residual autoencoder to effectively reconstruct the features of missing modalities, it significantly improves the robustness of the model under the condition of incomplete modalities and avoids a significant performance drop caused by missing modalities.

[0030] Second, enhance cross-sample discrimination and transfer capabilities: adopt a retrieval-enhanced contrastive learning mechanism to dynamically construct positive and negative sample pairs, strengthen the model's ability to learn cross-sample semantic consistency and differences, effectively improve the model's transfer reasoning ability across different samples and different domains, and enhance the stability and accuracy of detection.

[0031] Third, achieve dynamic fusion of multimodal information: introduce a fusion hybrid expert mechanism, through the collaborative work of multiple expert sub-networks and a weighted fusion strategy based on attention gating, to achieve flexible processing of multimodal and residual information, thereby improving the adaptability and discriminative ability of cross-modal feature fusion.

[0032] Fourth, improve cross-domain generalization performance: By combining the joint loss function with the multi-source domain multi-task training strategy, the generalization ability of the model is improved in complex environments with different platforms, different topics and different distributions, overcoming the performance degradation problem that traditional methods are prone to in cross-domain scenarios.

[0033] Fifth, it has good practicality and scalability: This method adopts an end-to-end training framework with a modular design, which is easy to integrate and deploy, adapts to the diverse and dynamically changing needs of video fake information detection in practical applications, and has broad application value. Attached Figure Description

[0034] Figure 1 This is a flowchart of the cross-domain detection method for video misinformation under modal incomplete conditions provided in the first embodiment of the present invention;

[0035] Figure 2 This is a model diagram of the cross-domain detection method for video misinformation under modal incomplete conditions provided in the first embodiment of the present invention;

[0036] Figure 3 This is an internal structure diagram of the video misinformation cross-domain detection system under modal incomplete conditions provided in the second embodiment of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0039] The following description, in conjunction with the accompanying drawings, details the cross-domain detection method and system for video misinformation under modal incomplete conditions provided by the present invention through specific embodiments and application scenarios.

[0040] Example 1

[0041] Please see Figure 1This is a flowchart illustrating the cross-domain detection method for video misinformation under modal incomplete conditions provided by the present invention, including steps S1 to S4.

[0042] S1. Obtain available multimodal data from video news samples from different sources. Available multimodal data includes video frames, audio, text descriptions, and related images.

[0043] It is worth noting that this step, involving multimodal input representation and modality missing detection, aims to achieve a structured representation of multimodal information in video news samples and accurate identification of modality missing conditions. This is the prerequisite for addressing the modality incompleteness problem in this invention. By designing a modality sensing module, the system can effectively identify the missing types of modal information such as image frames, audio, and text in video samples and construct corresponding modality missing masks, providing guidance for subsequent modality completion and fusion mechanisms. The innovation lies in adopting an explicit missing modeling mechanism, which enhances the system's adaptability to incomplete input. This step specifically includes four key stages: multimodal data acquisition, modality feature extraction, modality integrity detection and missing mask construction, and unified multimodal data representation.

[0044] Specifically, the process of acquiring and extracting modal information from video samples to construct a modality missing mask includes:

[0045] S1-1. Obtain multimodal data, assuming the video news sample set is... ,in, Indicates the first A news sample This represents the total number of video news samples.

[0046] Each sample contains several modal information, and typical modalities include: (1) image frame sequence: ,in Indicates the first The number of frames in a video. For the first Frame image data, (2) Audio signal: ,in Indicates the audio sample length. For the first One audio sampling point, (3) Text description: This includes text sequences such as video titles, subtitles, and comments. Indicates the length of the text sequence. Indicates the first One word, Represents a text sequence.

[0047] S1-2, Modal Feature Extraction: For different modalities, corresponding encoders are used for feature extraction. Specifically, this invention uses visual feature encoding for each frame in the video frame sequence. Feature extraction is performed to obtain visual feature vectors, where the visual feature vectors are... The mathematical expression is:

[0048] ;

[0049] in, Represents a visual encoder. Represents the dimensions of visual features. Representing the feature space, we obtain the overall visual modality features by taking the temporal average pooling or temporal modeling of the video frame sequence features. :

[0050] ;

[0051] Furthermore, this invention uses audio feature coding for audio signals. Perform feature extraction to obtain audio features The expression is:

[0052] ;

[0053] in, Indicates an audio encoder. For audio feature dimensions.

[0054] Furthermore, for text sequences This invention utilizes the pre-trained language model RoBERTa to analyze the text sequence. Encode to obtain text modal features :

[0055] ;

[0056] in, Indicates a text encoder. This indicates the dimension of text features.

[0057] S1-3, Modal Integrity Detection and Missing Mask Construction. This invention defines a modal integrity indicator vector to identify the presence or absence of modalities in each sample:

[0058] ;

[0059] ;

[0060] in, This indicates whether a visual modality is missing. This indicates whether a speech modality is missing. This indicates whether a text modality is missing. Indicates a modality missing mask. This indicates the missing state of the corresponding mode.

[0061] Among them, the modality missing mask This is used to indicate the existence status of each modality in the current sample. Modality validity can be determined through preprocessing and detection of modality data, such as determining whether the audio length exceeds a threshold, whether the text content is empty, and whether the video frame is complete.

[0062] S1-4. Unified multimodal data representation. Existing modal features are concatenated or fused to form multimodal input features.

[0063] Specifically, multimodal input features The mathematical expression is:

[0064] ;

[0065] in, This indicates a vector concatenation operation where there is no zeroed feature vector or other placeholder for the modality. This represents the actual concatenated feature dimensions. This multimodal representation is compared with the modality missing mask. Together, modal completion and fusion are performed to guide the reconstruction of missing modalities and cross-modal consistency learning. Representing the overall characteristics of the visual modality Indicates audio features, Represents text modal features.

[0066] S2. Process the available multimodal data to generate a modality missing mask.

[0067] It is worth noting that this step involves modality consistency learning and missing modality reconstruction. Specifically, this step addresses the problem of missing modality completion by proposing a method that integrates cross-modal semantic sharing and residual autoencoder reconstruction mechanisms to achieve feature recovery and unified representation generation for unavailable modalities. This module is based on the idea of ​​"cross-sensory reasoning" in brain cognition theory, mapping available modality features to a shared semantic space and using a residual autoencoder in conjunction with multimodal contextual information for feature reconstruction. The innovation lies in integrating cross-modal information to assist modeling and introducing a residual mechanism, making modality completion more stable and efficient. This step mainly includes five key sub-processes: shared semantic space mapping, residual autoencoder design, missing modality feature reconstruction, unified representation of modality completion, and loss function design, as detailed below:

[0068] S2-1, Shared semantic space mapping. This invention targets multimodal data. We design independent linear mapping networks to project the features of each modality into a shared semantic space:

[0069]

[0070] in, , It is the first Sample mode eigenvectors; It is modal The projection matrix, This is the feature dimension of the modality; It is a dimension of shared semantic space. It is a bias vector. It is a semantic representation projected onto the shared space, if modality Missing (i.e.) Then the mode corresponding to It is generated by the subsequent missing modality reconstruction module.

[0071] S2-2, Residual Autoencoder Design. To achieve effective reconstruction of missing modes, this invention constructs a residual autoencoder model, the structure of which includes an encoder. and decoder Meanwhile, cross-modal auxiliary information is introduced in each decoding layer to enhance stability.

[0072] Specifically, this invention utilizes the encoder within the residual autoencoder to compress and encode the semantic representation of the existing mode, as expressed in the following expression:

[0073] ;

[0074] in, This represents a concatenation or fusion operation on all existing modal semantic representations. These are the encoded hidden layer features. Indicates encoder, This indicates the missing state of the corresponding mode.

[0075] Furthermore, this invention utilizes a decoder to decode and reconstruct the missing modal features layer by layer, the first... The layer decoder output is Introducing cross-modal auxiliary information Perform residual learning:

[0076]

[0077] in, , Indicates the first Layer decoder output, auxiliary information Obtained through linear transformation of semantic features from other existing modalities, it is used to guide the decoding process. Indicates the first The encoder of the layer.

[0078] S2-3, Reconstruction of missing modal features, specifically for each missing modality. x The decoder outputs the corresponding reconstructed features, expressed as follows:

[0079] ;

[0080] in, This represents the reconstructed semantic representation of the missing modality.

[0081] S2-4. Unified Modal Completion Representation. The actual semantic representations and reconstructed representations of all modalities are merged to obtain a unified multimodal semantic representation after completion. The expression is:

[0082] ;

[0083] ;

[0084] in, Representing various multimodal semantics, where the completed unified multimodal semantic representation... It contains both semantic information of real-existing modalities and features of missing modalities, serving as input for subsequent retrieval, enhancement, contrastive learning, and classification.

[0085] S2-5. Loss Function Design. To ensure reconstruction quality, this invention designs a modal consistency reconstruction loss function. The mathematical expression is:

[0086] ;

[0087] in, This represents the total number of video news samples.

[0088] In this approach, reconstruction error is calculated only for modes that exist to avoid invalid optimization.

[0089] S3. Based on linear mapping, the available multimodal data is projected onto the shared semantic space to obtain shared semantics, and the shared semantics are reconstructed according to the preset residual autoencoder model and modality missing mask to obtain the completed modal features.

[0090] It is worth noting that this step involves retrieval-enhanced contrastive learning and cross-sample transfer modeling. Specifically, to improve the model's generalization ability among training samples, this step designs a retrieval-enhanced contrastive learning mechanism. By constructing semantically similar positive and negative sample pairs, it enhances the model's consistent modeling of cross-sample discriminative features. A memory replay strategy is employed to further expand the semantic contrast dimension, improving the breadth and robustness of representation transfer. The innovation lies in combining semantic retrieval and contrastive learning objectives, significantly improving the model's robust learning ability among samples with different distributions. This step mainly includes five key sub-processes: semantic retrieval mechanism, positive and negative sample retrieval, contrastive learning objective construction, memory replay mechanism, and overall contrastive loss, as detailed below:

[0091] S3-1, Semantic Retrieval Mechanism. (Targeting samples) Unified modal semantic representation vector In each training batch, the semantic similarity matrix between samples is calculated, and the mathematical expression is:

[0092] ;

[0093] in, Represents the semantic similarity matrix between samples. Indicates the number of training batches. Represents the elements within the semantic similarity matrix. The cosine similarity is calculated using the following expression:

[0094] ;

[0095] in, Indicates sample The unified modal semantic representation vector, Indicates sample The unified modal semantic representation vector.

[0096] S3-2, Positive and Negative Sample Retrieval. Samples are classified as positive or negative using label information. Specifically, a positive sample represents the set of samples with the same label as the current sample and the highest semantic similarity; a negative sample represents the set of samples with the same label as the current sample and the highest semantic similarity; and a negative sample represents the set of samples with different labels than the current sample but high semantic similarity. The expressions for positive and negative samples are:

[0097] ;

[0098] ;

[0099] in, Represents the sample set, Represents the set of negative samples. and These are preset hyperparameters representing the number of positive and negative samples, respectively. This indicates the items with the highest similarity. Indicates sample The true category label, Indicates sample The true category label.

[0100] S3-3, Constructing the Contrastive Learning Objective. Using the current sample as the anchor point, construct positive and negative sample pairs for contrastive learning. Define the anchor point. Compared with positive samples negative samples The contrastive loss is based on InfoNCE to generate the first... Each training batch corresponds to Enhanced contrastive learning loss function The mathematical expression is:

[0101] ;

[0102] in, It's a temperature parameter. Used to adjust the smoothness of the similarity distribution. Indicates sample The unified modal semantic representation vector, Indicates sample The unified modal semantic representation vector, Indicates sample The unified modal semantic representation vector, Represents the set of positive samples. This represents the set of negative samples.

[0103] S3-4, Memory Replay Mechanism. To enhance training stability and semantic diversity, a memory queue is introduced to store the semantic representations and labels of historical training samples. In each training step, positive and negative sample retrieval and comparative learning are performed together with the samples in the current batch and the memory queue, expanding the diversity of sample pairs. The mathematical expression is:

[0104] ;

[0105] in, This indicates the index of the sample currently being trained. Indicates the current candidate sample, This represents the current set of candidate samples, which can come from the current training samples, or from the memory or cache. Indicates sample The true category label, such as real news or fake news, Indicates sample The unified modal semantic representation vector, Indicates sample Compared with the current candidate samples semantic similarity function, Indicates sample The original set of positive samples, i.e., with A collection of samples with the same labels Indicates sample The original negative sample set, i.e., with A collection of samples with different labels This represents the enhanced set of positive samples. This represents the enhanced set of negative samples. For the set threshold, Represents the union, Indicates sample The true category label.

[0106] S3-5, Overall Contrast Loss. The average loss across all samples within a batch is used to form the final retrieval enhancement contrastive learning loss function, mathematically expressed as:

[0107] ;

[0108] in, Indicates batch size, To enhance contrastive learning loss function, this function is used to guide the model in learning to distinguish fine-grained semantic differences across samples, thereby improving discriminative and transfer abilities. Indicates the first The retrieval enhancement contrastive learning loss function corresponding to each training batch.

[0109] S4. The completed modal features are processed based on a hybrid expert mechanism to output the predicted probability of fake videos.

[0110] It is worth noting that this step involves cross-domain classification using a hybrid expert mechanism. This step constructs a hybrid expert mechanism to achieve dynamic fusion of multi-source completed modality features and cross-domain classification prediction. Each expert sub-network focuses on different feature dimensions, and its weight contribution in the final decision is dynamically adjusted through an attention gating mechanism. This structure fully leverages the importance of various types of information and enhances the model's adaptability to different modality combinations and domain differences. The innovation lies in introducing a multi-expert structure and an attention gating fusion mechanism to achieve an adaptive and flexible modality fusion strategy. Through the expert mechanism and dynamic weight adjustment, this step allows the model to adaptively emphasize key modalities and features for different samples, effectively improving the discriminative ability and cross-domain adaptability when modality is incomplete.

[0111] This step mainly includes four key sub-processes: expert sub-network construction, attention gating mechanism, multi-expert weighted fusion, and classification judgment, as detailed below:

[0112] S4-1, Construction of Expert Subnetwork. (This is part of a larger discussion on the construction of an expert subnetwork.) A sub-network of experts Each expert performs specialized modeling for different modality combinations, residual information, or inter-sample correlation features. The input is the completed unified modality semantic representation. Cross-sample transfer features and residual information are denoted as follows: .in, Indicates the first The input feature vector of an expert network This is the expert input dimension. Each expert is processed through a multi-layer neural network, outputting an expert feature representation. The mathematical expression is:

[0113] ;

[0114] in, This indicates the output dimension of the expert.

[0115] S4-2, Attention Gating Mechanism. To dynamically adjust the contribution weights of each expert to the final decision, an attention gating mechanism is designed. The weights are defined. :

[0116] ;

[0117] in, Indicates the first Experts on the sample The weight, It integrates contextual information. It is a multilayer perceptron. It is the encoded number Hidden layer features It is the encoded number Hidden layer features Indicates the number of experts.

[0118] S4-3, Multi-expert weighted fusion. Based on weights... The output features of each expert are weighted and fused to obtain the fused features, where the mathematical expression of the fused features is:

[0119]

[0120] S4-4, Classification Determination. Input the fused features into the classifier. In the process, a binary classification judgment is performed, and the probability of the video sample being fake or real is output:

[0121]

[0122] in, and For classifier parameters, This represents the probability distribution of the false (label 1) and true (label 0) categories.

[0123] Furthermore, before obtaining the predicted probability of fake videos, end-to-end training is required to ensure that the model uniformly optimizes missing modality reconstruction, cross-sample discrimination, and fake video classification objectives during training, while also considering the generalization requirements of multi-domain samples, thereby improving the overall performance of the model. The specific process of end-to-end training includes the following steps:

[0124] This step mainly includes four key sub-processes: joint loss function construction, multi-source domain sample training, end-to-end backpropagation training, and cross-domain generalization capability enhancement, as detailed below:

[0125] Joint Loss Function Construction. To achieve end-to-end training and coordinate the objectives of each module, this invention establishes a joint optimization objective function based on the retrieval enhancement contrastive learning loss function, a pre-defined modality consistency reconstruction loss function, and a classification loss function, and then performs backpropagation training to obtain the prediction probability of fake videos. Specifically, the modality consistency reconstruction loss function measures the reconstruction error of missing modality features; the retrieval enhancement contrastive learning loss function optimizes the semantic distribution structure between samples, enhancing the model's discriminative ability; and the classification loss function uses the cross-entropy loss function to measure the difference between the model's final output and the true label.

[0126] Specifically, this invention constructs a joint loss function comprising the following three parts:

[0127] ;

[0128] in, The modality consistency reconstruction loss function measures the reconstruction error of missing modes. To enhance the contrastive learning loss function for retrieval, which is used to improve cross-sample discrimination ability, , and These are loss weight hyperparameters used to balance the contributions of different losses. The classification loss function is expressed mathematically as follows:

[0129] ;

[0130] in, For the sample The true label indicates, Predict probabilities for the model.

[0131] Furthermore, to enhance the model's cross-domain generalization ability, this invention also employs multi-source domain sample training by introducing video sample sets from multiple different domains (such as different platforms, topics, regions, etc.). Training utilizes a multi-task learning strategy, defining a joint loss function for the multi-source domains. :

[0132]

[0133] in, For the number of domains, For the joint loss function of multiple source domains, These are domain weights, ensuring balanced learning of samples from different domains.

[0134] Furthermore, this invention also performs end-to-end backpropagation training. The model as a whole adopts an end-to-end structure to jointly optimize the objective function. All module parameters are updated using a gradient descent algorithm (such as the Adam optimizer).

[0135] ;

[0136] in, Represents the set of model parameters. It's the learning rate. This represents the gradient.

[0137] In summary, this step involves joint optimization and cross-domain generalization training. This step aims to achieve end-to-end joint training of the overall model and enhance the system's cross-platform adaptability through the participation of multi-source domain samples. By designing a joint loss function that comprehensively considers modality reconstruction error, contrastive learning objectives, and classification performance, the parameters of each module are collaboratively optimized. Simultaneously, multi-task training and domain regularization techniques are introduced to enhance the model's generalization performance. The innovation lies in integrating multiple sub-objectives and employing a multi-source domain sample training strategy, enabling the model to possess stronger robustness and transferability in diverse and complex scenarios.

[0138] Compared with existing technologies, this invention preprocesses video news samples to extract available multimodal information (including image frames, audio, text descriptions, etc.) and identifies missing modal types, constructing a modality missing mask for subsequent modeling. Secondly, it constructs a modality consistency learning module, based on the cross-modal reasoning mechanism in brain cognition theory, mapping available modalities to a shared semantic space and reconstructing missing modal features using a residual autoencoder to obtain a unified representation of modality completion. A retrieval enhancement contrastive learning module is designed, combining semantic similarity to retrieve positive and negative samples in the training batch, enhancing the model's cross-sample transfer reasoning ability by constructing contrastive learning objectives. A hybrid expert mechanism is introduced, dynamically fusing the outputs of multiple expert networks based on modality importance and feature confidence, improving the flexibility and robustness of cross-modal fusion. Finally, a joint loss function is constructed to jointly optimize modality reconstruction error, contrastive loss, and classification error, achieving end-to-end training. This method can significantly improve the accuracy and cross-domain adaptability of video fake information detection in complex real-world scenarios with severe modality loss or significant domain differences. It has good robustness, scalability and broad practical application prospects.

[0139] Example 2

[0140] Please see Figure 3 , Figure 3 The present invention provides a cross-domain detection system for video misinformation under modal incompleteness conditions, comprising:

[0141] The multimodal data acquisition module 100 is used to acquire available multimodal data from video news samples from different sources. The available multimodal data includes video frames, audio, text descriptions, and related images.

[0142] The modality missing mask generation module 200 is used to process available multimodal data to generate a modality missing mask;

[0143] The missing modality feature reconstruction module 300 is used to project available multimodal data onto a shared semantic space based on a linear mapping to obtain shared semantics, and to reconstruct the features of the shared semantics according to a preset residual autoencoder model and a modality missing mask to obtain the completed modality features.

[0144] The fake video result generation module 400 is used to process the completed modal features based on a hybrid expert mechanism to output the predicted probability of fake videos.

[0145] This invention proposes, as follows Figure 2The model diagram shown illustrates a cross-domain detection method for video misinformation under incomplete modal conditions. This method addresses the problems of severe multimodal data loss, large differences in cross-domain sample distribution, and insufficient model generalization ability in existing video misinformation detection technologies. Based on a cross-modal reasoning mechanism derived from brain cognitive theory, this system combines modality consistency learning, retrieval-enhanced contrastive learning, and a hybrid expert mechanism to achieve the completion of missing modalities and effective fusion of multimodal data. This enhances the model's cross-sample transfer reasoning ability and cross-domain generalization performance, significantly improving the accuracy and robustness of video misinformation detection.

[0146] The cross-domain detection system for video misinformation under modal incompleteness conditions in this embodiment of the invention can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can refer to a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., while a non-mobile electronic device can refer to a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This embodiment of the invention does not impose specific limitations.

[0147] The cross-domain detection system for video misinformation under modal incompleteness conditions in this embodiment of the invention can represent a device with an operating system. This operating system can represent Android, iOS, or other possible operating systems; this embodiment of the invention does not specifically limit the scope of the system.

[0148] The cross-domain detection system for video misinformation under modal incompleteness conditions provided in this invention can achieve... Figures 1 to 2 The various processes implemented in the method embodiment under the condition of incomplete modality for cross-domain detection of video misinformation will not be repeated here to avoid repetition.

[0149] Optionally, embodiments of the present invention also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described embodiments of the method for cross-domain detection of video misinformation under modal incompleteness conditions and can achieve the same technical effect. To avoid repetition, further details are omitted here.

[0150] This invention also provides a readable storage medium storing a program or instructions. When executed by a processor, the program or instructions implement the various processes of the above-described method for cross-domain detection of video misinformation under incomplete modality conditions, and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0151] The processor refers to the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0152] It should be noted that, in this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of this invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0153] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0154] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A method for cross-domain detection of video misinformation under modal incompleteness conditions, characterized in that, include: Obtain available multimodal data from video news samples from different sources, including video frames, audio, text descriptions, and related images; The available multimodal data is processed to generate a modality missing mask; The available multimodal data is projected onto a shared semantic space based on a linear mapping to obtain shared semantics, and the shared semantics are reconstructed according to a preset residual autoencoder model and the modality missing mask to obtain the completed modal features. The completed modal features are processed based on a hybrid expert mechanism to output the predicted probability of fake videos, including: An expert set consisting of multiple expert subnetworks is constructed. Each expert subnetwork performs specialized modeling for different modal combinations or semantic features to capture cross-modal differences and complementarities. Based on the expert set, the modality-completed multimodal fusion representation, the cross-sample transfer features generated by retrieval-enhanced contrastive learning, and the residual information are processed to obtain the intermediate prediction results and feature embeddings corresponding to each expert sub-network; The output weights of each expert subnetwork are calculated based on the intermediate prediction results, feature embeddings, and preset weights. The output results of each expert subnetwork are weighted and fused based on the output weights to obtain the fusion features; The fused features are classified into two categories based on a classifier to output the predicted probability of fake videos. Before obtaining the predicted probability of the fake video, end-to-end training is required, including: Based on the modality fusion representation of the video news samples in the shared semantic space, a retrieval-enhanced contrastive loss function is constructed. Based on the aforementioned retrieval enhancement contrast loss function, the preset modality consistency reconstruction loss function, and the preset classification loss function, a joint optimization objective function is obtained; A multi-source domain joint loss function is established based on the aforementioned joint optimization objective function; End-to-end training is performed based on the joint optimization objective function and the multi-source domain joint loss function to obtain the prediction probability of fake videos.

2. The method for cross-domain detection of video misinformation under modal incompleteness conditions according to claim 1, characterized in that, The process of processing the available multimodal data to generate a modality missing mask includes: The multimodal data is subjected to format unification and feature extraction processing to obtain initial modal feature representations; Modality missing detection is performed on the initial modality feature representation, and missing modes are recorded through flag variables to generate a judgment result; Based on the judgment result, a modality missing mask is generated, wherein the modality missing mask represents the presence or absence of each modality in the current sample in the form of a binary vector, and is used for reference and control in the subsequent modality completion and feature fusion process.

3. The method for cross-domain detection of video misinformation under modal incompleteness conditions according to claim 1, characterized in that, The specific steps for constructing the retrieval-enhanced contrastive loss function include: Based on the modal fusion representation of the current video news sample in the shared semantic space, the semantic similarity between the current video news sample and other video news samples in the same batch is calculated. In each training batch, based on the semantic similarity, positive samples with the same label and the closest semantics to the current video news sample are retrieved, as well as negative samples with different labels but similar semantics, to construct positive and negative comparison sample pairs; A variant of the temperature-regulated InfoNCE loss is proposed to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs, and a memory replay mechanism is introduced to establish a retrieval-enhanced contrastive loss function.

4. The method for cross-domain detection of video misinformation under modal incompleteness conditions according to claim 3, characterized in that, The memory replay mechanism is used to store the semantic representations and labels of historical training samples. In each training step, it combines the current batch samples with the samples in the memory queue to perform positive and negative sample retrieval and comparative learning, expanding the diversity of sample pairs. The mathematical expression is: ； in, This indicates the index of the sample currently being trained. Indicates the current candidate sample, Represents the current set of candidate samples. Indicates sample The true category label, Indicates sample The unified modal semantic representation vector, Indicates sample Compared with the current candidate samples semantic similarity function, Indicates sample The original set of positive samples, i.e., with A collection of samples with the same labels Indicates sample The original negative sample set, i.e., with A collection of samples with different labels This represents the enhanced set of positive samples. This represents the enhanced set of negative samples. For the set threshold, Represents the union, Indicates sample The true category label.

5. The method for cross-domain detection of video misinformation under modal incompleteness conditions according to claim 1, characterized in that, The mathematical expression for the preset weights is: ； in, Indicates the first Experts on the sample The weight, It integrates contextual information. It is a multilayer perceptron. It is the encoded number Hidden layer features It is the encoded number Hidden layer features Indicates the number of experts.

6. A cross-domain detection system for video misinformation under modal incompleteness conditions, characterized in that, A system for implementing the cross-domain detection method for video misinformation under modal incompleteness conditions as described in any one of claims 1-5; the system comprises: The multimodal data acquisition module is used to acquire available multimodal data from video news samples from different sources. The available multimodal data includes video frames, audio, text descriptions, and related images. A modality missing mask generation module is used to process the available multimodal data to generate a modality missing mask; The missing modality feature reconstruction module is used to project the available multimodal data onto the shared semantic space based on a linear mapping to obtain shared semantics, and to perform feature reconstruction on the shared semantics according to a preset residual autoencoder model and the missing modality mask to obtain the completed modality features. The fake video result generation module is used to process the completed modal features based on a hybrid expert mechanism to output the predicted probability of fake videos.

7. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of the method for cross-domain detection of video misinformation under any of claims 1-5.

8. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method for cross-domain detection of video misinformation under modal incomplete conditions as described in any one of claims 1-5.

Images