A false short video information detection method based on multi-modal feature adaptive fusion
Patent Information
- Application Number
- CN202611034902.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-13
- Publication Date
- 2026-10-23
AI Technical Summary
然而，尽管取得了一定进展，当前研究仍然面临诸多挑战
本公开的实施例中，通过上述方法，一方面，在特征提取后引入信息效用加权模块，基于信息瓶颈理论通过KL散度压缩冗余信息并最大化特征与标签的互信息，获得纯化特征，提升了特征的判别性和泛化能力。另一方面，基于纯化特征生成自适应模态权重，使权重分配依据各模态的真实信息效用而非原始特征量级，相比传统加权拼接方法更准确地反映各模态对检测任务的贡献程度。再一方面，设置预测一致性校准模块，将各单模态独立预测分布作为参考锚点，通过最小化联合预测与各独立预测之间的KL散度，使融合表示在训练中持续向各单模态预测方向修正，确保各模态特异性在融合后得以积极保留。
Smart Images
Figure CN122964815A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of fake short video information detection technology, and in particular to a fake short video information detection method based on multimodal feature adaptive fusion. Background Technology
[0002] With the continuous development of the internet, online social platforms have experienced rapid expansion. While lowering the barriers to information access, this trend has also accelerated the spread of fake short video content. Leveraging the widespread dissemination of social media, fake short video information has gradually evolved into a global problem, not only misleading public perception and disrupting social order, but also damaging the online ecosystem and potentially having an extremely negative impact on public opinion. Therefore, exploring efficient methods for detecting fake short videos is of significant practical importance.
[0003] In recent years, with the explosive growth of social media and information dissemination, the technology for detecting fake short videos has also developed rapidly. This research field integrates various technologies such as natural language processing and machine learning, aiming to achieve automatic identification of fake short video content.
[0004] Effective detection of fake short videos is crucial to curbing their further spread. Existing technologies have shown promise in identifying and suppressing misinformation. Early research relied primarily on textual analysis, but with the increasing diversity of social media content, text-based detection methods are insufficient. Therefore, multimodal methods have emerged in recent years, integrating text, audio, and visual information to significantly improve detection performance. However, despite these advancements, current research still faces numerous challenges.
[0005] Current mainstream methods generally employ separate encoders to extract features from different modalities of a video, followed by simple concatenation or weighted concatenation of these modalities. This approach fails to remove redundant information. Furthermore, existing fusion methods dilute the unique information of each individual modality's features when fused. Therefore, removing redundant information before multimodal feature fusion and finding more efficient ways to fuse multimodal features are crucial. Therefore, it is necessary to improve one or more of the problems existing in the above-mentioned related technical solutions.
[0006] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention
[0007] The purpose of this disclosure is to provide a method for detecting fake short video information based on multimodal feature adaptive fusion, thereby overcoming, to at least to some extent, one or more problems caused by the limitations and defects of related technologies.
[0008] According to a first aspect of the present disclosure, a method for detecting fake short video information based on multimodal feature adaptive fusion is provided, comprising: Step S1: Use a pre-trained encoder to extract text modal features, visual modal features, and audio modal features from the short video as coarse-grained features for each modality; Step S2: Based on the information bottleneck constraint, the information utility weighting module performs redundant information compression and retains prediction utility on the coarse-grained features of each modality to obtain the purified features of each modality. Step S3: Generate adaptive weights for each modality based on the purification features of each modality, and perform weighted summation of the purification features of each modality according to the adaptive weights to construct the initial multimodal fusion features; Step S4: Use the first classifier corresponding to each modality to predict the purified features of each modality to obtain the independent prediction distribution of each modality; Step S5: Use the second classifier to predict the initial multimodal fusion features and obtain the joint prediction distribution; Step S6: The prediction consistency calibration module uses the independent prediction distribution as the anchor point and calibrates the initial multimodal fusion features into the final multimodal fusion features by minimizing the KL divergence between the independent prediction distribution and the joint prediction distribution. Step S7: Input the final multimodal fusion features into the third classifier to obtain the detection results, and jointly optimize the network parameters based on the total loss function.
[0009] Furthermore, step S1 specifically includes: The short video is analyzed and keyframes are extracted. These keyframes are then input into an encoder for feature extraction to obtain visual modality features. ; Text recognition is performed on keyframes, and the recognized text, along with the title of the short video, is input into the encoder for feature extraction to obtain text modality features. ; Speech recognition is performed on short videos, and the recognized speech is input into an encoder for feature extraction to obtain audio modal features. ; Text modality features Visual modal features and audio modal features Input into mapping function , Processing is performed to obtain coarse-grained features for each mode. ;in, , , These represent the text modality, visual modality, and audio modality, respectively.
[0010] Furthermore, step S2 specifically includes: The encoding process of coarse-grained features of each modality is modeled as a conditional probability distribution; Based on the KL divergence between the conditional probability distribution and the task-independent prior distribution, a compression loss is constructed to perform redundancy compression. Based on the fourth classifier, a utility loss is constructed to preserve the predicted utility; Based on the optimization of compression loss and utility loss, an information utility-weighted loss function is obtained; Based on the information utility weighted loss function, the coarse-grained features of each modality are redundantly compressed and the prediction utility is preserved to obtain the purified features of each modality.
[0011] Furthermore, the compression loss is:
[0012] in, For conditional probability distribution, For task-independent prior distributions. It is characterized by coarse-grained properties. Characteristics of purification; The utility loss is:
[0013] in, This is the fourth classifier introduced; The information utility weighted loss function is:
[0014] in, This is a hyperparameter used to adjust redundant compression strength.
[0015] Furthermore, step S3 specifically includes: Purification characteristics of each mode Perform pooling operations to obtain the global utility vectors for each modality. ; The global utility vector of each modality Send to shared gated network The adaptive weights for each mode are calculated using the Softmax function. ; Utilizing adaptive weights for each mode The corresponding purified features are weighted and summed to obtain the initial multimodal fusion features. .
[0016] Furthermore, the global utility vector for each modality is:
[0017] The adaptive weights for each mode are:
[0018] The initial multimodal fusion features are:
[0019] in, For a shared gated network, The weights for each mode.
[0020] Furthermore, the independent probability distribution is:
[0021] in, As the first classifier, For the number of categories, For the first The probability output by the first classifier; The joint prediction distribution is:
[0022] in, This is the second classifier.
[0023] Furthermore, step S6 specifically includes: Calculate the KL divergence between each independent prediction distribution and the joint prediction distribution; The sum of the KL divergences is used as the prediction consistency calibration loss function. Backpropagation is used to optimize the parameters of the second classifier and the generation parameters of the initial multimodal fusion features, correcting the joint prediction distribution towards the prediction directions of each modality, thereby calibrating the initial multimodal fusion features into the final multimodal fusion features. The prediction consistency calibration loss function is:
[0024]
[0025] In the formula, For the first The joint prediction distribution of the outputs of the second classifiers.
[0026] Furthermore, the test results are as follows:
[0027] in, For the final multimodal fusion features, For the third classifier; The classification loss of the third classifier is:
[0028] in, For real labels, This is the cross-entropy loss function.
[0029] Furthermore, the method also includes: The total loss function is:
[0030] in, The classification loss of the third classifier. The information utility-weighted loss function, The loss function is used to calibrate the consistency prediction.
[0031] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: In the embodiments of this disclosure, the above method, on the one hand, introduces an information utility weighting module after feature extraction. Based on the information bottleneck theory, it compresses redundant information through KL divergence and maximizes the mutual information between features and labels to obtain purified features, thereby improving the discriminative and generalization capabilities of the features. On the other hand, it generates adaptive modality weights based on the purified features, making the weight allocation based on the true information utility of each modality rather than the original feature magnitude, which more accurately reflects the contribution of each modality to the detection task compared to the traditional weighted concatenation method. Furthermore, a prediction consistency calibration module is set up, using the independent prediction distribution of each single modality as a reference anchor point. By minimizing the KL divergence between the joint prediction and each independent prediction, the fused representation is continuously corrected towards the prediction direction of each single modality during training, ensuring that the specificity of each modality is actively preserved after fusion. Attached Figure Description
[0032] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0033] Figure 1 The diagram illustrates the steps of a method for detecting fake short video information based on adaptive fusion of multimodal features in an exemplary embodiment of this disclosure. Figure 2 A complete illustration of the method for detecting fake short video information based on multimodal feature adaptive fusion in an exemplary embodiment of this disclosure is shown. Figure 3 Experimental results of various methods in exemplary embodiments of this disclosure on the FakeSV and FakeTT datasets are shown. Detailed Implementation
[0034] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0035] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.
[0036] This example implementation provides a method for detecting fake short video information based on adaptive fusion of multimodal features. (Reference) Figure 1 As shown, the method may include: Step S1: Use a pre-trained encoder to extract text modal features, visual modal features, and audio modal features from the short video as coarse-grained features for each modality; Step S2: Based on the information bottleneck constraint, the information utility weighting module performs redundant information compression and retains prediction utility on the coarse-grained features of each modality to obtain the purified features of each modality. Step S3: Generate adaptive weights for each modality based on the purification features of each modality, and perform weighted summation of the purification features of each modality according to the adaptive weights to construct the initial multimodal fusion features; Step S4: Use the first classifier corresponding to each modality to predict the purified features of each modality to obtain the independent prediction distribution of each modality; Step S5: Use the second classifier to predict the initial multimodal fusion features and obtain the joint prediction distribution; Step S6: The prediction consistency calibration module uses the independent prediction distribution as the anchor point and calibrates the initial multimodal fusion features into the final multimodal fusion features by minimizing the KL divergence between the independent prediction distribution and the joint prediction distribution. Step S7: Input the final multimodal fusion features into the third classifier to obtain the detection results, and jointly optimize the network parameters based on the total loss function.
[0037] The above methods achieve several advantages. First, an information utility weighting module is introduced after feature extraction. Based on the information bottleneck theory, redundant information is compressed using KL divergence, and the mutual information between features and labels is maximized to obtain purified features, thus improving their discriminative and generalization abilities. Second, adaptive modality weights are generated based on the purified features. This ensures that weight allocation is based on the true information utility of each modality rather than the magnitude of the original features, more accurately reflecting the contribution of each modality to the detection task compared to traditional weighted concatenation methods. Third, a prediction consistency calibration module is set up. Using the independent prediction distributions of each single modality as reference anchors, the KL divergence between the joint prediction and each independent prediction is minimized, allowing the fused representation to continuously correct towards the prediction directions of each single modality during training, ensuring that the specificity of each modality is actively preserved after fusion.
[0038] Below, we will refer to Figures 1 to 3 The steps of the method described above in this example embodiment will be explained in more detail.
[0039] In one embodiment, such as Figure 1 The diagram shows a flowchart of a method for detecting fake short video information based on multimodal feature adaptive fusion; the method specifically includes: In step S1, the pre-trained encoder is used to extract the text modality features, visual modality features and audio modality features of the short video as coarse-grained features for each modality.
[0040] Specifically, for any given short video, keyframes are extracted and then fed into a visual encoder to obtain visual features. For any given short video, after extracting keyframes, OCR technology is used to recognize the text in the keyframes, and then the text is fed into a text encoder along with the video title to obtain the features of the text modality. For any given short video, use the Whisper tool for speech recognition, and use an audio encoder (e.g., a Transformer sequence model) to extract features from the recognized speech to obtain the features of the audio modality; The audio features are obtained through the above steps. There are also text features Visual semantic features Then, each modal feature is fed into a mapping function. , To obtain coarse-grained features .
[0041] In step S2, the information utility weighting module performs redundant information compression and retains prediction utility on the coarse-grained features of each modality based on information bottleneck constraints, so as to obtain the purified features of each modality.
[0042] For any given short video, after step S1, coarse-grained features for each modality are obtained. , , To obtain purification characteristics , First, perform redundant feature compression.
[0043] Existing methods for removing feature redundancy mainly rely on attention mechanisms or contrastive learning: attention mechanisms softly suppress redundancy by learning importance scores at each position, but this process is passive; the model is only indirectly adjusted during backpropagation of the task loss and is not explicitly required to forget task-irrelevant components. Contrastive learning implicitly compresses irrelevant information by bringing positive sample representations closer together, but the compression constraint does not directly apply to the information flow of a single modality itself. The fundamental limitation of these two types of methods is that redundancy compression is "outcome-oriented" rather than "information-theoretic-constrained," and cannot actively sever the connection between irrelevant components and downstream predictions at the information flow level.
[0044] The underlying technology of the IUW module in this application is derived from the Information Bottleneck (IB) theory. IB theory models feature extraction as a constrained information transfer problem: for the original input features… Internally, it contains both discriminative components highly correlated with the classification label Y and a large amount of statistical redundancy irrelevant to the task. The core objective of IB theory is to learn a compressed representation. This maximizes the amount of predictive information for label Y. At the same time, minimize the dependence on the original input. This involves establishing a controllable trade-off between "redundancy reduction" and "utility preservation," with the optimization objective being:
[0045] in, It is a theoretical trade-off coefficient in the information bottleneck theory, which controls the relative importance between the compression term and the utility term. It is part of the IB objective function itself and belongs to the theoretical derivation level.
[0046] because and These are all difficult to calculate directly. This application transforms the above objectives into an optimizable loss function through variational lower bound approximation. Specifically, the encoding process is modeled as a conditional probability distribution. The encoder is subject to explicit gradient constraints in both directions: "redundancy compression" and "preservation of prediction utility." This forces the model to perform information filtering in each forward propagation, enabling proactive extraction of useful signals rather than passively adjusting weights based on gradients. To allow the model to forget task-irrelevant redundant information, purified features are obtained by minimizing the KL divergence.
[0047] in, It is a prior distribution independent of the task, usually taking the standard normal distribution. KL divergence, as an information constraint, penalizes the posterior distribution. It functions by deviating from the prior distribution. This is because the prior distribution does not contain features about the input. Minimizing the KL divergence allows the encoder to compress the representation, ensuring that purified features are preserved in the final representation. .
[0048] Simply compressing redundancy is not enough; the characteristics of the purified material must also be ensured. Retain sufficient task-determining information. Therefore, for utility items... We maximize the variational lower bound and introduce an auxiliary classifier (i.e., a fourth classifier). With an approximate true posterior distribution Based on the variational lower bound derivation, maximizing This is equivalent to minimizing the following cross-entropy loss:
[0049] in, Indicates the distribution from purification The feature representation obtained from sampling. The objective function directly constrains the purified representation's contribution to the classification task through the supervision signal, ensuring... During training, the most useful discriminative features for the task are always retained to prevent excessive discarding of useful information during the compression step.
[0050] After the above steps, the purified characteristics are obtained. The total loss function of the information utility weighted module integrates the compression term and the utility term as follows:
[0051] in, It is a hyperparameter used to adjust the intensity of redundant compression, controlling the tightness of the information bottleneck constraint. When When the signal is large, the encoder is forced to compress more information, which helps to eliminate redundancy but may lose some useful signals; when... When the pressure is small, the compression constraint tends to be loose, and the purification effect weakens. Therefore, The selection of [a specific parameter] needs to strike a balance between sufficient compression and information integrity; the optimal value was determined through parameter sensitivity analysis in the experiment. The resulting purification characteristics... As a reliable foundation for subsequent adaptive weight generation and multimodal fusion, it ensures that the allocation of modal weights is based on the real information utility of each modality, rather than the magnitude of the original features.
[0052] In step S3, adaptive weights for each modality are generated based on the purification features of each modality. The purification features of each modality are then weighted and summed according to the adaptive weights to construct the initial multimodal fusion features.
[0053] Specifically, the purified features generated in step S2 provide an indispensable foundation for generating adaptive modality weights, as shown in the following formula:
[0054] in This represents the global utility vector for mode m. The final adaptive weights for each mode are then... It is generated by a scoring network, and the calculation process is as follows:
[0055] in This represents a shared gating network that is responsible for mapping utility vectors to weights for each modality.
[0056] After generating the weights for each modality, we begin constructing the initial multimodal fusion features, as follows:
[0057] in, This represents the characteristics after purification.
[0058] In steps S4 to S6, the purified features of each modality are predicted using the first classifier corresponding to each modality to obtain the independent prediction distribution of each modality; the initial multimodal fusion features are predicted using the second classifier to obtain the joint prediction distribution; the prediction consistency calibration module uses the independent prediction distribution as the anchor point and calibrates the initial multimodal fusion features into the final multimodal fusion features by minimizing the KL divergence between the independent prediction distribution and the joint prediction distribution.
[0059] Specifically, to preserve the unique properties of each modality, the prediction consistency calibration module performs independent classification operations on the purified features. Existing multimodal fusion methods typically employ cross-attention or feature concatenation followed by unified mapping via a fully connected network when handling cross-modal interactions. The core problem with these approaches is that the generation process of the multimodal joint representation is a unidirectional operation aimed at "merging information," and the unique features of each modality are inevitably diluted by the signals of other modalities during deep interactions. Once a modality has noise or semantic ambiguity, its negative impact will spread to the entire joint representation, causing the discriminative cues of individual modalities to be masked. The underlying technical principle of the PCC module in this application is to use the independent classification predictions of each individual modality as "reference anchors" in an information-theoretic sense, and to impose continuous calibration constraints on the joint representation by minimizing the KL divergence between the joint prediction distribution and each independent prediction distribution at the loss function level. The fundamental difference between this module and existing methods lies in the following: existing methods implicitly protect modality specificity, passively relying on network structure design; while this module transforms modality specificity protection into an explicitly optimizable objective. The joint representation must maintain consistency with the judgments of each individual modality at the prediction level. Any fusion operation that causes the disappearance of unique cues of a particular modality will lead to increased consistency loss, thus being actively corrected during training. This mechanism is equivalent to setting up an independent "prediction supervisor" for each modality, ensuring that the unique prediction contributions of each modality are not neutralized or canceled out while integrating multimodal information during the fusion process. Based on the above principles, each purified feature... Each is determined by its corresponding first classifier. This process generates independent probability distributions:
[0060] Where K represents the number of categories, and in this task, K is set to 2. These independent predictions It captures salient features of each modality stream, namely discriminative cues that may be obscured or neutralized in deep interactions within the fused feature space. By maintaining these parallel classification results, the framework establishes a set of reference distributions representing independent predictive perspectives for each modality, providing anchors for subsequent consistent alignment.
[0061] After obtaining individual prediction results for each modality, the prediction consistency calibration module is designed with a loss-based approach. A mechanism that pulls joint predictions toward independent branches. Global joint prediction is defined as... ,in This serves as the second classifier. To ensure that the joint representation does not lose key insights from each modality, the framework minimizes the divergence between independent predictions and the joint output, calculated as follows:
[0062] in, The KL divergence is defined as follows:
[0063] By minimizing This enables joint forecasting It aligns with the most informative independent insights from each branch. This alignment mechanism acts as a corrective guide, reinforcing mode-specific features within a unified feature space and effectively preventing the dilution of unique signals. From an information theory perspective, this relates to the direction selection of the KL divergence. Instead of reverse propagation, the goal is to bring the joint prediction closer to the individual modality predictions, rather than compressing the distribution range of individual modality predictions, thus preserving the unique discriminative contribution of each modality to the maximum extent. The second classifier is continuously optimized through backpropagation. and network parameters in the process of fusion feature generation This allows the joint representation to be gradually corrected towards the prediction directions of each single modality during training, thereby... Calibration This serves as the input to the third classifier. Therefore, the intrinsic importance of each modality is fully realized, resulting in a robust multimodal representation where the modality specificity is actively preserved in the final detection task.
[0064] In step S7, the final multimodal fusion features are input into the third classifier to obtain the detection results, and the network parameters are jointly optimized based on the total loss function.
[0065] The final multimodal representation obtained This comprehensive feature incorporates highly useful features from text, vision, and audio, while an alignment mechanism ensures that individual features of each modality remain significant. Finally, it is output to the third classifier, as follows:
[0066] in This represents the predicted result.
[0067] To ensure that all independent branches of a single mode and the joint representation of multiple modes can learn the correct representation, a loss function is used for optimization, as follows:
[0068] in, Represents the true label, Represents the prediction result. This represents the cross-entropy loss function.
[0069] The total loss function is as follows:
[0070] in, The classification loss of the third classifier. The information utility-weighted loss function, The loss function is used to calibrate the consistency prediction.
[0071] In one specific embodiment, Table 1 shows the statistical information of the FakeSV and FakeTT datasets under different positive sample ratios. The FakeSV and FakeTT datasets are a Chinese dataset and an English dataset, respectively. Both datasets collect short video information from a certain app over the past few years.
[0072] Table 1: Statistical information for the FakeSV and FakeTT datasets
[0073] Figure 2 The diagram shown is a complete illustration of this application.
[0074] like Figure 3The diagram shows the experimental results of this application compared with other existing methods. Acc. represents the accuracy metric, Prec. represents the precision metric, Rec. represents the recall metric, F1 represents the harmonic mean, and Macro F1 represents the macro-harmonic mean. The results show that this application achieves improved performance on the FakeSV and FakeTT datasets compared to existing methods, maintaining stable and optimal accuracy. (Note: The last sentence appears to be a separate, unrelated section about GPT-4 and GPT-4 technical report.)RoBERTa(Roberta:A robustlyoptimized bert pretraining approach),ResNet(Deep residual learning for imagerecognition),HuBert(Hubert: Self-supervised speech representation learning bymasked prediction of hidden units),CLIP(Learning transferable visual modelsfrom natural language supervision),Timesformer(Is space-time attention allyou need for video understanding?),FANVM(Using topic modeling and adversarialneural networks for fake news video detection),TikTec(A multimodalmisinformation detector for COVID-19 short videos on TikTok),SVFEND(Fakesv: Amultimodal benchmark with rich social context for fake news detection onshort video platforms),FakingRecipe(FakingRecipe: Detecting Fake News onShort Video Platforms from the Perspective of Creative Process),KDSGAT-FNVD(Knowledge-Enhanced Dynamic Scene Graph Attention Network for Fake News VideoDetection),MFSVFEND(MFSVFND: Multimodal Fusion Network for Detecting FakeNews on Short Video Platforms),MFAE (Multimodal Feature Adaptive Enhancement for Fake News Video Detection) is an authoritative and widely recognized method in the field of news detection containing fake short videos, while MFAF is the method proposed in this application.
[0075] The above methods achieve several advantages. First, an information utility weighting module is introduced after feature extraction. Based on the information bottleneck theory, redundant information is compressed using KL divergence, and the mutual information between features and labels is maximized to obtain purified features, thus improving their discriminative and generalization abilities. Second, adaptive modality weights are generated based on the purified features. This ensures that weight allocation is based on the true information utility of each modality rather than the magnitude of the original features, more accurately reflecting the contribution of each modality to the detection task compared to traditional weighted concatenation methods. Third, a prediction consistency calibration module is set up. Using the independent prediction distributions of each single modality as reference anchors, the KL divergence between the joint prediction and each independent prediction is minimized, allowing the fused representation to continuously correct towards the prediction directions of each single modality during training, ensuring that the specificity of each modality is actively preserved after fusion.
[0076] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.
[0077] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.
Claims
1. A method for detecting fake short video information based on multimodal feature adaptive fusion, characterized in that, include: Step S1: Use a pre-trained encoder to extract text modal features, visual modal features, and audio modal features from the short video as coarse-grained features for each modality; Step S2: Based on the information bottleneck constraint, the information utility weighting module performs redundant information compression and retains prediction utility on the coarse-grained features of each modality to obtain the purified features of each modality. Step S3: Generate adaptive weights for each modality based on the purification features of each modality, and perform weighted summation of the purification features of each modality according to the adaptive weights to construct the initial multimodal fusion features; Step S4: Use the first classifier corresponding to each modality to predict the purified features of each modality to obtain the independent prediction distribution of each modality; Step S5: Use the second classifier to predict the initial multimodal fusion features and obtain the joint prediction distribution; Step S6: The prediction consistency calibration module uses the independent prediction distribution as the anchor point and calibrates the initial multimodal fusion features into the final multimodal fusion features by minimizing the KL divergence between the independent prediction distribution and the joint prediction distribution. Step S7: Input the final multimodal fusion features into the third classifier to obtain the detection results, and jointly optimize the network parameters based on the total loss function.
2. The method for detecting fake short video information based on multimodal feature adaptive fusion according to claim 1, characterized in that, Step S1 specifically includes: The short video is analyzed and keyframes are extracted. These keyframes are then input into an encoder for feature extraction to obtain visual modality features. ; Text recognition is performed on keyframes, and the recognized text, along with the title of the short video, is input into the encoder for feature extraction to obtain text modality features. ; Speech recognition is performed on short videos, and the recognized speech is input into an encoder for feature extraction to obtain audio modal features. ; Text modality features Visual modal features and audio modal features Input into mapping function , Processing is performed to obtain coarse-grained features for each mode. ;in, , , These represent the text modality, visual modality, and audio modality, respectively.
3. The method for detecting fake short video information based on multimodal feature adaptive fusion according to claim 2, characterized in that, Step S2 specifically includes: The encoding process of coarse-grained features of each modality is modeled as a conditional probability distribution; Based on the KL divergence between the conditional probability distribution and the task-independent prior distribution, a compression loss is constructed to perform redundancy compression. Based on the fourth classifier, a utility loss is constructed to preserve the predicted utility; Based on the optimization of compression loss and utility loss, an information utility-weighted loss function is obtained; Based on the information utility weighted loss function, the coarse-grained features of each modality are redundantly compressed and the prediction utility is preserved to obtain the purified features of each modality.
4. The method for detecting fake short video information based on multimodal feature adaptive fusion according to claim 3, characterized in that, The compression loss is: in, For conditional probability distribution, For task-independent prior distributions. It is characterized by coarse-grained properties. Characteristics of purification; The utility loss is: in, This is the fourth classifier introduced; The information utility weighted loss function is: in, This is a hyperparameter used to adjust redundant compression strength.
5. The method for detecting fake short video information based on multimodal feature adaptive fusion according to claim 4, characterized in that, Step S3 specifically includes: Purification characteristics of each mode Perform pooling operations to obtain the global utility vectors for each modality. ; The global utility vector of each modality Send to shared gated network The adaptive weights for each mode are calculated using the Softmax function. ; Utilizing adaptive weights for each mode The corresponding purified features are weighted and summed to obtain the initial multimodal fusion features. .
6. The method for detecting fake short video information based on multimodal feature adaptive fusion according to claim 5, characterized in that, The global utility vector for each modality is: The adaptive weights for each mode are: The initial multimodal fusion features are: in, For a shared gated network, The weights for each mode.
7. The method for detecting fake short video information based on multimodal feature adaptive fusion according to claim 6, characterized in that, The independent probability distribution is: in, As the first classifier, For the number of categories, For the first The probability output by the first classifier; The joint prediction distribution is: in, This is the second classifier.
8. The method for detecting fake short video information based on multimodal feature adaptive fusion according to claim 7, characterized in that, Step S6 specifically includes: Calculate the KL divergence between each independent prediction distribution and the joint prediction distribution; The sum of the KL divergences is used as the prediction consistency calibration loss function. Backpropagation is used to optimize the parameters of the second classifier and the generation parameters of the initial multimodal fusion features, correcting the joint prediction distribution towards the prediction directions of each modality, thereby calibrating the initial multimodal fusion features into the final multimodal fusion features. The prediction consistency calibration loss function is: In the formula, For the first The joint prediction distribution of the outputs of the second classifiers.
9. The method for detecting fake short video information based on multimodal feature adaptive fusion according to claim 8, characterized in that, The test results are as follows: in, For the final multimodal fusion features, For the third classifier; The classification loss of the third classifier is: in, For real labels, This is the cross-entropy loss function.
10. The method for detecting fake short video information based on multimodal feature adaptive fusion according to claim 9, characterized in that, The method also includes: The total loss function is: in, The classification loss of the third classifier. The information utility-weighted loss function, The loss function is used to calibrate the consistency prediction.