Multi-modal video authenticity detection method based on policy-driven hybrid gating

Abstract

The application discloses a kind of multi-modal video authenticity detection methods based on strategy driven hybrid gate, obtain the text semantics, text emotion, visual semantics, audio emotion, OCR editing and segment-level editing of the video to be detected Multi-modal features;The above-mentioned features are divided into content-type evidence and editing-type evidence, and are respectively input into content-type evidence encoder and editing-type evidence encoder, to obtain corresponding evidence representation;Strategy state is constructed based on content-type evidence representation, editing-type evidence representation and the difference between them;Output action probability and fusion weight through hybrid strategy gate network, adaptively select content-type evidence, editing-type evidence, interactive fusion or weighted collaborative fusion mode, generate prediction result;According to prediction result, output video authenticity detection result.The present application solves the problem that the existing method uses fixed fusion structure and is difficult to adapt to different sample evidence dependence, realizes sample-level adaptive evidence fusion, and improves detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of video detection technology, and specifically to a multimodal video authenticity detection method based on strategy-driven hybrid gating. Background Technology

[0002] With the rapid development of social media and short video platforms, multimodal content combining text, images, audio, and video has become an important carrier of online information dissemination. Compared with misinformation in the form of single text, multimodal video misinformation can comprehensively utilize exaggerated text, misleading visuals, emotional audio, abnormal subtitles, and video editing to influence users' judgment, and is characterized by rapid spread, strong concealment, and high difficulty in identification. Therefore, accurate and stable authenticity detection of multimodal video content has become an urgent technical problem to be solved in the fields of online content governance and cyberspace security.

[0003] Existing multimodal video authenticity detection methods mainly extract features from different modalities through text encoders, visual encoders, and audio encoders, and then fuse these features using methods such as feature concatenation, attention weighting, or cross-modal interaction. While these methods can utilize complementary information between different modalities to some extent, they typically aggregate the extracted information as general features, lacking explicit differentiation of the authenticity evidence represented by different features.

[0004] From the perspective of the mechanisms underlying the formation of misinformation, different video samples rely on different pieces of evidence for authenticity. Some fake videos primarily exhibit semantic inconsistencies between text and visuals, emotional conflicts between text and audio, or exaggerated content; others mainly show editing traces such as abnormal OCR text layout, abnormal subtitle presentation, abnormal video clip duration, or abnormal editing order. Therefore, for different video samples, authenticity detection models should be able to select content-based evidence, editing-based evidence, or a combination of both for discrimination.

[0005] Existing feature fusion methods typically employ a fixed fusion structure. Even if the attention mechanism can generate different weights for different samples, the underlying fusion structure remains unchanged, with each evidence branch participating in the final prediction. When a certain evidence branch is irrelevant to the judgment of the authenticity of the current sample, forcibly including that branch in the fusion process can easily introduce noise. Existing methods based on dynamic weights can continuously adjust the contribution ratio of different evidence, but it is difficult to explicitly close invalid evidence branches; existing methods based on dynamic routing can select specific branches, but it is difficult to continuously and finely adjust the collaborative contributions between multiple branches.

[0006] Furthermore, evidence branch selection is a discrete decision, while evidence contribution ratio is a continuous decision; together, they constitute a hybrid decision space combining discrete and continuous approaches. Traditional supervised learning methods struggle to jointly optimize fusion structure selection and branch contribution adjustment in the absence of action-level labels, resulting in the detection model's inability to adaptively determine the optimal fusion method based on the specific evidence state of different samples.

[0007] In summary, the existing technical methods have the following problems and shortcomings: First, existing methods typically aggregate multimodal information as general features, lacking explicit differentiation and targeted modeling of content-based and editorial evidence. Second, the existing fixed fusion structure is difficult to adapt to the different dependencies of different video samples on authentic evidence, and is prone to introducing irrelevant evidence noise; Third, existing continuous weighting methods cannot explicitly skip invalid evidence branches, while existing discrete routing methods have difficulty in achieving fine-grained adjustment of the contribution ratio of multiple evidence branches. Fourth, existing technologies lack a mechanism for jointly optimizing discrete fusion structures and continuous fusion weights, making it difficult to form a sample-level adaptive evidence fusion strategy.

[0008] Based on this, the present invention proposes a multimodal video authenticity detection method based on strategy-driven hybrid gating to solve the above problems. Summary of the Invention

[0009] This invention provides a strategy-driven hybrid gating-based multimodal video authenticity detection method to solve the problem that existing multimodal video authenticity detection methods use a fixed fusion structure, making it difficult to dynamically select authenticity evidence and adjust the evidence contribution ratio according to different video samples.

[0010] This invention provides a multimodal video authenticity detection method based on policy-driven hybrid gating, the method comprising: Multimodal feature extraction is performed on the video to be detected to obtain a multimodal feature set; The features in the multimodal feature set are divided into content-based evidence and editing-based evidence, and the content-based evidence and editing-based evidence are encoded to obtain content-based evidence encoding vectors and editing-based evidence encoding vectors; Based on the differences between the content-based evidence encoding vector and the edit-based evidence encoding vector, a dual-evidence strategy state is constructed; The dual-evidence strategy state is input into a pre-trained hybrid strategy gating network, which outputs the prediction probabilities of different strategy actions. The strategy actions include using only the individual prediction results corresponding to content-type evidence, using only the individual prediction results corresponding to editing-type evidence, performing interactive fusion on the individual prediction results of the two types of evidence, and performing weighted fusion on the individual prediction results of the two types of evidence. The strategy action corresponding to the maximum prediction probability is selected and executed to obtain the unnormalized prediction result. The unnormalized prediction results are input into a linear classification layer, and after softmax normalization, a binary classification probability of whether the video to be detected is real or fake is obtained. The category corresponding to the larger probability is selected as the final video authenticity detection result.

[0011] Furthermore, multimodal feature extraction is performed on the video to be detected to obtain a multimodal feature set, specifically including: Multimodal feature extraction is performed on the video to be tested, including text semantic features, text emotion features, visual semantic features, audio emotion features, OCR-related editing features, and video segment-level editing features; Among them, the text semantic features are used to characterize the meaning of the text expressed in the video, including subtitle text, video title, video description and speech-to-text text; the text emotion features are used to characterize the emotional state contained in the text; the visual semantic features are used to characterize the semantic information of the picture in the video key frame or sample frame, including people, objects, scenes, people's postures and their spatial relationships; and the audio emotion features are used to characterize the voice emotion state conveyed by the audio acoustic signal. The OCR-related editing features include OCR phrase features and OCR visual pattern features; OCR phrase features are used to represent the text content in the video frame obtained based on OCR recognition; OCR visual pattern features are used to represent the text layout, color, position, and display format in the video frame obtained based on OCR recognition. The video segment-level editing features include segment-level editing content features, segment duration features, and segment temporal position features. The segment-level editing content features are used to characterize the spatiotemporal content within a video segment and the editing information between adjacent segments, including image changes, motion changes, and content jumps. The segment duration features and segment temporal position features are used to represent the segment length and the segment's position on the video timeline, respectively.

[0012] Furthermore, the features in the multimodal feature set are divided into content-based evidence and editing-based evidence, specifically including: Textual semantic features, textual emotional features, visual semantic features, and audio emotional features are classified as content-based evidence; OCR-related editing features, including OCR phrase features and OCR visual pattern features, as well as video segment-level editing features, including segment-level edited content features, segment duration features, and segment temporal location features, are classified as editing evidence. The text sentiment features and audio sentiment features in the content-type evidence are fed into the Transformer-based sentiment evidence encoding module. By performing modal alignment, Transformer encoding and mean pooling on the text sentiment features and audio sentiment features, a unified sentiment evidence vector is output. The textual and visual semantic features in the content-based evidence are fed into the Transformer-based semantic evidence encoding module. By performing modal alignment, Transformer encoding, cross-modal attention operation, and mean pooling on the textual and visual semantic features, a unified semantic evidence vector is output. The OCR phrase features and OCR visual pattern features in the edited evidence are fed into the Transformer-based OCR evidence encoding module. The OCR phrase features and OCR visual pattern features are processed by modal alignment, Transformer encoding and mean pooling to output a unified OCR evidence vector. The fragment-level edited content features, fragment duration features, and fragment time position features in the edited evidence are fed into the Transformer-based OCR evidence encoding module. The OCR phrase features and OCR visual pattern features are processed by modal alignment, Transformer encoding, and mean pooling to output a unified fragment edited evidence vector. The obtained unified sentiment evidence vector and unified semantic evidence vector are concatenated to obtain the content-based evidence encoding vector; The obtained unified OCR evidence vector and unified fragment editing evidence vector are concatenated to obtain the editing evidence encoding vector.

[0013] Furthermore, based on the differences between the content-based evidence encoding vector and the edit-based evidence encoding vector, a dual-evidence strategy state is constructed, specifically including:

[0014] in, Indicates the state of the dual-evidence strategy. Represents the encoding vector of content-based evidence and editable evidence encoding vector The absolute differences between elements.

[0015] Furthermore, the hybrid policy gating network includes a policy encoder, an action mapping branch, and a weight mapping branch; the output of the policy encoder is connected to the action mapping branch and the weight mapping branch, respectively. The policy encoder comprises a first fully connected layer, a first ReLU activation layer, a second fully connected layer, and a second ReLU activation layer connected in sequence, wherein the first fully connected layer operates in a dual-evidence policy state. As input, intermediate features are obtained after linear transformation and ReLU activation. Then the intermediate features The hidden decision representation is obtained by inputting it into the second fully connected layer and then activating it with ReLU. :

[0016]

[0017] in, and These represent the weight matrices of the first and second fully connected layers, respectively. and These represent the corresponding bias vectors; Indicates the policy encoder; The obtained hidden decision representation Input the action mapping branch and the weight mapping branch respectively; The action mapping branch includes a third fully connected layer with an output dimension of 4 and a Softmax layer; the third fully connected layer hides the decision representation. Mapped to action scores corresponding to the four policy actions:

[0018] Then, the probability distribution of policy actions is obtained through the Softmax layer:

[0019] in, This represents the weight matrix of the fourth fully connected layer. This represents the corresponding bias vector; Represents a discrete action mapping function; This indicates that, given a dual-evidence strategy state Under the given conditions, the probability of selecting strategy action a; The weight mapping branch includes a fourth fully connected layer with an output dimension of 1 and a Sigmoid layer; the fourth fully connected layer hides the decision representation. Mapped to fusion weight score:

[0020] in, Represents the weight mapping function; Represents the Sigmoid function; This represents the fusion weights used for weighted fusion. .

[0021] Furthermore, the strategy actions include using only the individual prediction results corresponding to content-type evidence, using only the individual prediction results corresponding to editing-type evidence, performing interactive fusion of the individual prediction results of the two types of evidence, and performing weighted fusion of the individual prediction results of the two types of evidence, specifically including: Establish an action space A = {0, 1, 2, 3}, where action 0: performs interactive fusion of the individual prediction results of the two types of evidence; action 1: uses only the individual prediction results corresponding to content-type evidence; action 2: uses only the individual prediction results corresponding to edit-type evidence; and action 3: performs weighted fusion of the individual prediction results of the two types of evidence. Let... , , , These are the prediction results for executing the corresponding actions 0, 1, 2, and 3, respectively.

[0022] in, To use only the individual prediction results corresponding to content-based evidence, the content-based evidence encoding vector is... The video authenticity prediction result is output after inputting the content branch prediction network used for binary classification task.

[0023] in, To use only the individual prediction results corresponding to edited evidence, the edited evidence encoding vector is... The video authenticity prediction result is output after inputting the edit branch prediction network used for binary classification task;

[0024] in, represents the hyperbolic tangent activation function, and ⊙ represents element-wise multiplication;

[0025] in, This represents the fusion weights, output by the weight mapping branch.

[0026] Furthermore, the method also includes: During the training process, a two-stage training strategy, including a warm-up training stage and a strategy optimization stage, is adopted to train the network model. In the pre-training phase, the probabilities of each action output by the hybrid policy gating network are used as weighting coefficients to sum the execution results of each action in a weighted manner, thereby obtaining the prediction results of the video samples. This avoids the model prematurely selecting a certain action in the early stages of training.

[0027] in, Indicates the policy state Next choice The probability of a strategy action. Indicates the first The execution corresponding to the video sample of the _th video sample The prediction result for each action; This indicates the first stage of the warm-up training. Prediction results for each video sample; During the policy optimization phase, the network model parameters obtained from the pre-training are used, and actions are sampled based on the action probabilities output by the hybrid policy gating network. The sampled actions are then executed to obtain the corresponding prediction results. The reward function is used to determine whether the sampled actions are suitable for the current sample, thereby optimizing the action probability distribution.

[0028] in, This indicates the first stage of the strategy optimization phase. The prediction results for each video sample This indicates the action to be performed corresponding to the i-th video sample. The prediction results.

[0029] Furthermore, in the strategy optimization phase, for the first... Construct a reward function from video samples:

[0030] in, Indicates the first The total reward obtained for the sampled action corresponding to each video sample; This represents the detection reward, used to evaluate the predictive effectiveness of the prediction results on the true labels; This represents the branch importance reward, used to evaluate whether the sampled action is biased towards evidence branches that are more reliable in judging the current sample; This represents the regularization term for the fusion weights, used to constrain the fusion weights; and This represents the weighting coefficient of the corresponding reward item; The reward for testing is represented as follows:

[0031]

[0032] in, Indicates the first The prediction result obtained after performing the sampled action on a video sample; is the true category label for the i-th video sample, where a value of 0 indicates a real video and a value of 1 indicates a fake video; Indicates to The class probability vector obtained after executing the Softmax function; Represents the true class label extracted from the class probability vector. The corresponding predicted probability; This represents the cross-entropy loss between the predicted result and the true classification label; Branch importance rewards are represented as follows:

[0033] in:

[0034]

[0035] in, Indicates the first Branch importance reward for each video sample; and They represent the first Individual prediction results for each video sample using only content-based evidence and individual prediction results using only editing-based evidence. and They are respectively for and The class probability vector obtained after executing the Softmax function; Indicates the first The true classification labels of each video sample; and From the class probability vector and The real category extracted from The corresponding predicted probability; Indicates the first Branch preference indicator for each video sample; This represents the hyperbolic tangent activation function, used to limit the branch importance reward to a certain value. Within the range; Among them, the branch preference indicator Represented as:

[0036] in, Indicates that for the first Actions obtained from sampling video samples; when At this time, the prediction results of the two types of evidence are interactively fused, without bias towards either the content branch or the editing branch, thus making ;when When only the prediction result corresponding to content-based evidence is used, it indicates a bias towards the content-based evidence branch, therefore let ;when When only the individual prediction result corresponding to edited evidence is used, it indicates a bias towards the edited evidence branch, therefore let ;when At that time, a weighted fusion is performed on the individual prediction results of the two types of evidence, making According to the fusion weight Determine the direction and degree of branch preference; The fusion weight regularization term is expressed as:

[0037] in, Indicates the first The fusion weight regularization term for each video sample; Indicates the first The fusion weights of each video sample; This indicates the degree of deviation of the fusion weight from the balance weight of 0.5; This represents the fusion weight regularization control parameter; when When, constraints are imposed on the fusion weights that deviate from the equilibrium state to improve the stability of the fusion process; when In this case, the fusion weights are encouraged to tilt towards one of the evidence branches to improve the clarity of the fusion decision.

[0038] Furthermore, during the strategy optimization phase, the moving reward baseline is updated based on the average reward of the batch of samples:

[0039]

[0040] in, This represents the average reward for all video samples in the current training batch; This indicates the number of video samples included in the current training batch; Indicates the first The total reward for each video sample; This represents the moving reward baseline, used to characterize the average reward level of the model during training; The momentum coefficient representing the moving reward baseline; symbol " "" indicates that the moving reward baseline on the left is updated using the calculation result on the right side of the symbol; the moving reward baseline is set when the model starts training. Initialize to 0; after the reward calculation is completed in each training batch, use the average reward of the current batch. Update the mobile rewards baseline.

[0041] Furthermore, the method also includes: The total loss during training is expressed as:

[0042] in, This represents the total loss during the model training process; Indicates supervised classification loss; Indicates the strategy loss; The weighting coefficients representing the policy loss are used to adjust the contribution ratio between the supervised classification objective and the policy optimization objective. The supervised classification loss is expressed as:

[0043] in, Indicates supervised classification loss; Indicates the first Prediction results for one video sample; and They represent the first Individual prediction results for each video sample using only content-based evidence and individual prediction results using only editing-based evidence; Indicates the first The true classification labels of each video sample; Represents the cross-entropy loss function; and These represent the corresponding weighting coefficients; The strategy loss is expressed as:

[0044] in, Indicates the strategy loss; This indicates the number of video samples in the current training batch; Indicates the first in the current training batch One video sample; Indicates the first The actions sampled from each video sample The advantage value; Indicates the first The dual-evidence strategy state corresponding to each video sample; Indicating a dual-evidence strategy state Select the sampled action below The probability of; Represents the entropy regularity coefficient; This represents the entropy function.

[0045] This invention provides a multimodal video authenticity detection method based on policy-driven hybrid gating, which has the following beneficial effects: 1. This invention explicitly divides multimodal features into content-based evidence and editing-based evidence, and combines this with a strategy-driven fusion mechanism to adaptively select the most discriminative source of genuine evidence for different fake video samples. When a certain type of evidence has weak correlation with the current sample, its participation in the fusion can be effectively reduced or even skipped directly, thereby reducing the interference of irrelevant or noisy information on the detection results and improving the overall accuracy and robustness of the discrimination.

[0046] Second, this invention constructs a hybrid decision-making mechanism that combines fusion structure selection and fusion weight adjustment. It achieves dynamic routing of evidence branches through discrete actions and fine adjustment of evidence contribution ratio through continuous weights. This enables the model to flexibly switch between different evidence branches and achieve fine-grained fusion when multiple pieces of evidence are coordinated, thereby significantly improving the expressive and adaptive capabilities of the fusion mechanism.

[0047] Third, this invention adopts a policy learning approach that combines preheating training with reward-driven optimization. In the early stage of training, a soft execution mechanism is used to ensure that each fusion primitive obtains sufficient gradient signals. In the later stage, reward-based policy optimization is used to gradually strengthen the effective fusion path, thereby effectively improving the stability of policy learning, avoiding premature convergence of the gating network to a single fusion mode, and enhancing the model's adaptability to complex sample distributions.

[0048] Fourth, this invention achieves high detection accuracy and stability on both Chinese and English short video datasets, verifying the generalization ability of the proposed method under different language environments and data distributions on different platforms. This demonstrates that the method has good cross-language and cross-scenario applicability and can adapt to diverse practical application needs.

[0049] Fifth, this invention can be widely applied to scenarios such as content review on short video platforms, identification of online rumors, verification of the authenticity of media content, monitoring of public opinion risks, and governance of cyberspace security. By improving the accuracy and reliability of fake video detection, it helps to reduce the social risks and economic losses caused by the spread of false information, and has significant engineering application value and promotion prospects. Attached Figure Description

[0050] Figure 1 A flowchart illustrating a strategy-driven hybrid gating-based multimodal video authenticity detection method according to an embodiment of the present invention; Figure 2 This is an overall architecture diagram of a strategy-driven hybrid gating multimodal video authenticity detection method provided in one embodiment of the present invention. Detailed Implementation

[0051] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of the invention. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to the present invention are not shown or described in the specification. This is to avoid obscuring the core parts of the invention with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0052] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0053] The first embodiment of this invention provides a multimodal video authenticity detection method based on policy-driven hybrid gating. The following is a combination of... Figure 1 and Figure 2 Please provide a detailed explanation.

[0054] Step S100: Perform multimodal feature extraction on the video to be detected to obtain a multimodal feature set.

[0055] The above steps specifically include: In this implementation, multimodal feature extraction is performed on the video to be detected, including text semantic features, text emotion features, visual semantic features, audio emotion features, OCR-related editing features, and video segment-level editing features; Among them, text semantic features are used to characterize the meaning of the text expressed in the video, including subtitle text, video title, video description and speech-to-text text; text emotion features are used to characterize the emotional state contained in the text; visual semantic features are used to characterize the semantic information of the picture such as people, objects, scenes, people's postures and their spatial relationships in the video key frames or sampled frames; and audio emotion features are used to characterize the voice emotion state conveyed by the audio acoustic signal. OCR-related editing features include OCR phrase features and OCR visual pattern features; OCR phrase features are used to represent the text content in video frames obtained based on OCR recognition; OCR visual pattern features are used to represent the layout, color, position, and display format of text in video frames obtained based on OCR recognition. Video clip-level editing features include clip-level editing content features, clip duration features, and clip temporal position features. Clip-level editing content features are used to characterize the spatiotemporal content within a video clip, as well as editing information such as image changes, motion changes, and content jumps between adjacent clips. Clip duration features and clip temporal position features are used to represent the clip length and the clip's position on the video timeline, respectively.

[0056] In this embodiment, the multimodal feature set is represented as follows:

[0057] in, Representing the semantic features of the text. Indicates the emotional characteristics of the text. Representing visual semantic features, Indicates the emotional characteristics of audio. Indicates OCR-related editing features, This indicates video clip-level editing features.

[0058] The following examples illustrate the methods for extracting various features: In this embodiment, text semantic features Automatic speech recognition methods can be used to convert the audio content in the video into transcribed text. The title text and the transcribed text can then be concatenated in the correct order to obtain the combined text. The combined text is segmented into words and then input into a pre-trained BERT language model. The BERT language model typically includes word embedding layers, positional embedding layers, and multiple Transformer encoding layers, outputting a text feature sequence: .in, Indicates the first The number of lexical units in a combined video text. Indicates the first The context feature vector corresponding to each word element. The hidden vector corresponding to the classification label in the last layer of BERT is taken as the sentence-level text semantic feature: .

[0059] Text sentiment features The emotion recognition data can be obtained by setting a text emotion recognition head after the BERT language model. The text emotion recognition head consists of a fully connected layer 1, a ReLU activation layer, a Dropout layer, a fully connected classification layer 2, and a Softmax layer. The classification label hidden vector output by BERT is input into the fully connected layer 1:

[0060] in, and These represent the weight matrix and bias vector of the first fully connected layer, respectively. This represents the hidden sentiment representation of the text. The weight matrix mentioned above... and bias vector All are trainable parameters, and their dimensions are determined by the input feature dimension and the output hidden dimension. In this embodiment, the BERT classification label hidden vector is 768-dimensional, and the text sentiment hidden representation is 128-dimensional. It has 128 rows and 768 columns. The length is 128. At the start of training, Initialize using a Kaiming uniform distribution based on the input dimension. The parameters are initialized using a uniform distribution with the same upper and lower bounds; when the input dimension is 768, the initial values ​​are approximately -0.0361 and 0.0361. These parameters are then updated via backpropagation, and the trained values ​​are not limited by this initialization range.

[0061] During the training of the text emotion recognition network, the hidden representation of the text emotion is input into the fully connected classification layer 2 and the Softmax layer to obtain the text emotion category probability:

[0062] The hidden representation of text sentiment prior to the classification layer is used as the text sentiment feature: .

[0063] Visual semantic features Video frames can be sampled from the video to be detected at fixed time intervals, and frame-level static visual features can be extracted using a pre-trained VGG19 network. ,in, Indicates the first The number of sampled frames in a video. Indicates the first The static visual features of each sampled frame are analyzed. Simultaneously, 16 consecutive frames are selected centered on each sampling time point to form a video segment, and a pre-trained C3D network is used to extract segment-level dynamic visual features. Average pooling is performed on frame-level static visual features and fragment-level dynamic visual features respectively, and the pooling results are concatenated and linearly mapped to obtain visual semantic features:

[0064] in, and These represent the weight matrix and bias vector of the linear mapping layer, respectively. The weight matrix above... Both the bias vector and the concatenated visual feature dimension are trainable parameters, and their size is determined by the concatenated visual feature dimension and the mapped visual semantic feature dimension. In this embodiment, the visual semantic features are uniformly mapped to 128 dimensions, therefore... The number of rows is 128, and the number of columns is equal to the input visual feature dimension. The length is 128. At the start of training, Initialize using a Kaiming uniform distribution based on the input dimension. The parameters are initialized using a uniform distribution with the same upper and lower bounds. The absolute values ​​of the initial upper and lower bounds are the reciprocals of the square root of the input dimension. The parameters are then updated through backpropagation, and the values ​​after training are not restricted by this initialization interval.

[0065] Audio Emotional Characteristics Audio signals can be separated from the video to be detected and input into a pre-trained VGGish audio coding network to extract audio feature sequences corresponding to speech, ambient sounds, and background music. ,in, Indicates the first The audio signal of the video. Indicates the number of audio segments. Indicates the first The feature vectors of each audio segment are used. Average pooling is applied to the audio feature sequence, and then the audio emotion features are obtained through a fully connected layer and an activation function.

[0066] in, and These represent the weight matrix and bias vector of the fully connected layer, respectively. The weight matrix above... and bias vector All are trainable parameters, and their dimensions are determined by the dimensions of the pooled audio features and the audio emotion features. In this embodiment, the pooled audio features are 768-dimensional, and the audio emotion features are 128-dimensional. It has 128 rows and 768 columns. The length is 128. At the start of training, Initialize using a Kaiming uniform distribution based on the input dimension. The parameters are initialized using a uniform distribution with the same upper and lower bounds; when the input dimension is 768, the initial values ​​are approximately -0.0361 and 0.0361. These parameters are then updated via backpropagation, and the trained values ​​are not limited by this initialization range.

[0067] Editing evidence includes OCR-related editing features. and video clip-level editing features It is used to characterize the text presentation pattern, clip editing pattern, and time organization features of the video to be detected.

[0068] OCR-related editing features are obtained by detecting and recognizing text regions in video sample frames. First, the DBNet text detection network is used to detect text regions in the video frames, obtaining bounding boxes for each text region. Then, a CRNN text recognition network is used to recognize the text within the bounding boxes, obtaining OCR phrases. The recognized OCR phrases are then input into a pre-trained BERT language model for encoding, yielding OCR phrase features. OCR visual pattern features are constructed based on the position, width, height, area, aspect ratio of the text detection boxes, and the arrangement of the text within video frames. It is used to characterize the position, size, layout, and display format of OCR text in a video frame. For example, for a line of subtitles appearing at the bottom of a video frame, DBNet first determines the text detection box corresponding to the subtitle, CRNN recognizes the text content such as "an event has occurred", and uses BERT to generate the corresponding OCR phrase features; at the same time, it forms OCR visual pattern features based on information such as the text box being located at the bottom of the screen, having a large horizontal width, and a small text area height.

[0069] OCR-related editing features It can be represented as:

[0070] in, Indicates OCR phrase features, This represents the visual pattern features of OCR.

[0071] Video clips are obtained through shot boundary detection. First, the TransNet V2 shot boundary detection network is used to calculate the shot transition probability between adjacent frames. Based on the detected shot transition positions, the complete video is divided into several consecutive video clips. For each video clip, segment-level spatiotemporal content features (including spatial and temporal information; spatial information refers to the scene content such as people, objects, scenes, and positional relationships in each frame; temporal information refers to the changes in scene content over time) are extracted based on the video frames it contains. When a clip contains a large number of frames, 16 frames are uniformly sampled from within the clip and input into the C3D network to obtain the segment-level editable content features of that clip. For example, when a camera switch is detected at the 3-second and 8-second marks in the video, the corresponding time interval can be divided into different video segments, and then the spatiotemporal content, duration, and time position in the complete video can be extracted for each segment.

[0072] Video clip-level editing features It can be represented as:

[0073] in, Indicates the characteristics of fragment-level edited content. Indicates the duration feature of the segment. This indicates the temporal location characteristics of a segment.

[0074] It should be noted that the text semantic features, text emotion features, visual semantic features, audio emotion features, OCR-related editing features, and video segment-level editing features described in this invention can be pre-extracted using existing text encoders, visual encoders, audio encoders, OCR recognition models, and video segment analysis models. This invention does not limit the specific network structure of the above feature extraction models; any multimodal features of the corresponding type that can be obtained can be used to implement this invention.

[0075] Step S200: Divide the features in the multimodal feature set into content-based evidence and editing-based evidence, and encode the content-based evidence and editing-based evidence to obtain content-based evidence encoding vector and editing-based evidence encoding vector.

[0076] The above steps specifically include: S210 classifies text semantic features, text emotion features, visual semantic features, and audio emotion features as content-based evidence; and classifies OCR-related editing features, including OCR phrase features and OCR visual pattern features, as well as video segment-level editing features, including segment-level edited content features, segment duration features, and segment time position features, as editing-based evidence.

[0077] In this embodiment, the content-based evidence and the editable evidence are then input into the content-based evidence encoder and the editable evidence encoder, respectively, to obtain the content-based evidence representation. and edited evidence indicates The content-based evidence encoder includes a sentiment evidence encoding module and a semantic evidence encoding module. The editing-based evidence encoder includes an OCR evidence encoding module and a fragment evidence encoding module. Each evidence encoding module includes a modality alignment layer, a Transformer encoding layer, and a mean pooling layer. The semantic evidence encoding module also includes a cross-modal attention layer.

[0078] S220: The text sentiment features and audio sentiment features in the content-type evidence are fed into the Transformer-based sentiment evidence encoding module. By performing modal alignment, Transformer encoding and mean pooling on the text sentiment features and audio sentiment features, a unified sentiment evidence vector is output.

[0079] Specifically, the emotion evidence encoding module encodes the emotion features of the text. and audio emotional characteristics As input, text sentiment features and audio sentiment features are processed through corresponding modality alignment layers and Transformer encoding layers to obtain text sentiment hidden features. and audio emotion concealment features The text sentiment latent features and audio sentiment latent features are concatenated and then aggregated using a mean pooling layer to obtain the sentiment evidence representation.

[0080] in, This indicates a feature concatenation operation. This indicates that the input features are subjected to mean pooling. Emotional evidence representation is used to characterize the consistency or conflict between the textual and audio emotions in a video.

[0081] S230: The textual semantic features and visual semantic features in the content-type evidence are fed into the Transformer-based semantic evidence encoding module. By performing modal alignment, Transformer encoding, cross-modal attention operation and mean pooling on the textual semantic features and visual semantic features, a unified semantic evidence vector is output.

[0082] Specifically, the semantic evidence encoding module encodes text semantic features and visual semantic features As input, textual semantic features and visual semantic features are respectively processed through corresponding modality alignment layers and Transformer encoding layers to obtain textual semantic hidden features. Visual semantic hidden features Furthermore, the semantic features after interaction are obtained by calculating the correlation between textual and visual features through cross-modal attention operations.

[0083] The semantic features after interaction are averaged and pooled to obtain the semantic evidence representation:

[0084] S240 feeds the OCR phrase features and OCR visual pattern features in the edited evidence into the Transformer-based OCR evidence encoding module. The OCR phrase features and OCR visual pattern features are processed by modal alignment, Transformer encoding and mean pooling to output a unified OCR evidence vector.

[0085] Specifically, the OCR evidence encoding module encodes OCR phrase features. and OCR visual pattern features As input, OCR phrase features are used to characterize the text content identified in the video frame; OCR visual pattern features are used to characterize the position, arrangement and presentation of the text in the video frame.

[0086] OCR phrase features and OCR visual pattern features are respectively processed through corresponding modality alignment layers and Transformer encoding layers to obtain OCR phrase hidden features. and hidden features in OCR visual modes And the OCR evidence is obtained through mean pooling:

[0087] S250 inputs the fragment-level edited content features, fragment duration features, and fragment time position features from the edited evidence into the Transformer-based OCR evidence encoding module. The OCR phrase features and OCR visual pattern features are processed by modal alignment, Transformer encoding, and mean pooling to output a unified fragment edited evidence vector.

[0088] Specifically, the fragment evidence encoding module will encode fragment-level edited content features. Fragment duration characteristics and fragment time location features Input, fragment-level editing content features Fragment duration characteristics and fragment time location features The fragment content hidden features are obtained by passing the corresponding modality alignment layer and Transformer encoding layer respectively. Duration hidden features and time location hidden features When the fragment duration feature or fragment temporal location feature is a one-dimensional or low-dimensional value, it is converted into a higher-dimensional value through the corresponding fully connected modality alignment layer. The vector is then input into the Transformer encoding layer. The fragment content hidden features, duration hidden features, and temporal location hidden features are concatenated and passed through an average pooling layer to obtain the fragment editing evidence representation.

[0089] S260, the obtained unified sentiment evidence vector and unified semantic evidence vector are concatenated to obtain the content-based evidence encoding vector.

[0090] Specifically, the emotional evidence representation and the semantic evidence representation are concatenated to obtain the content-based evidence representation:

[0091] S270, the obtained unified OCR evidence vector and unified fragment editing evidence vector are concatenated to obtain the editing evidence encoding vector.

[0092] Specifically, the OCR evidence representation and the fragment editing evidence representation are concatenated to obtain the edited evidence representation:

[0093] in, This indicates the mean pooling operation. This indicates a cross-modal attention operation. This indicates a feature splicing operation.

[0094] Step S300: Construct a dual-evidence strategy state based on the difference between the content-based evidence encoding vector and the editing-based evidence encoding vector.

[0095] Specifically, based on content-based evidence representation Edited evidence And the difference between the two, constructing the dual-evidence strategy state:

[0096] in, It represents the element-by-element absolute difference between content-based and editorial evidence representations, and is used to characterize the degree of consistency and conflict between the two types of evidence.

[0097] Step S400: Input the dual-evidence policy state into the pre-trained hybrid policy gating network, and output the prediction probabilities of different policy actions. The policy actions include using only the individual prediction results corresponding to content-type evidence, using only the individual prediction results corresponding to editing-type evidence, performing interactive fusion on the individual prediction results of the two types of evidence, and performing weighted fusion on the individual prediction results of the two types of evidence. Then, select the policy action corresponding to the maximum prediction probability to execute and obtain the unnormalized prediction result.

[0098] The above steps specifically include: S410, change the dual-evidence strategy state The input is a hybrid policy gating network, which includes a policy encoder, a discrete action mapping branch, and a continuous weight mapping branch; the output of the policy encoder is connected to the discrete action mapping branch and the continuous weight mapping branch, respectively.

[0099] The policy encoder consists of a first fully connected layer, a first ReLU activation layer, a second fully connected layer, and a second ReLU activation layer connected in sequence. The first fully connected layer takes the two-evidence policy state s as input, and after linear transformation and ReLU activation, it obtains intermediate features:

[0100] intermediate features The hidden decision representation is obtained by inputting the second fully connected layer and performing ReLU activation. :

[0101] in, and These represent the weight matrices of the first and second fully connected layers, respectively. and These represent the corresponding bias vectors; This represents the policy encoder consisting of the two fully connected layers described above. The weight matrix above... , and bias vector , All parameters are trainable, and their size is determined by the dimension of the dual-evidence policy state and the policy hiding dimension. In this embodiment, the policy hiding dimension is 256. At the start of training, the weights of each fully connected layer are initialized using a Kaiming uniform distribution based on the corresponding input dimension, and the corresponding biases are initialized using a uniform distribution with the same upper and lower bounds. The absolute values ​​of the initial upper and lower bounds are the reciprocals of the square root of the input dimension of that layer. The above parameters are updated through backpropagation based on the total loss, and their values ​​after training are not limited by the initialization interval.

[0102] Based on the hidden decision representation (u), the discrete action mapping branch and the continuous weight mapping branch are input respectively.

[0103] The discrete action mapping branch consists of a third fully connected layer with an output dimension of 4 and a softmax layer. The third fully connected layer hides the decision representation. Mapped to action scores corresponding to the four policy actions:

[0104] Then, the discrete policy action distribution is obtained through the Softmax layer:

[0105] The continuous weight mapping branch includes a fourth fully connected layer with an output dimension of 1 and a sigmoid layer. The fully connected layer hides the decision representation. Mapped to fusion weight score:

[0106] in, The policy encoder, a component of the hybrid policy gating network, hides the decision representation. It also serves as the input for both the discrete action mapping layer and the continuous weight mapping layer. This represents the discrete action mapping function. The discrete action mapping layer uses a third fully connected layer with an output dimension of 4 to hide the decision representation. The mapping is to the action scores corresponding to the four strategy actions. This represents the continuous weight mapping function, implemented by a fully connected layer with an output dimension of 1, used to linearly map the hidden decision representation u to the fused weight score. This represents the Sigmoid function. .

[0107] S420, Content Branch Prediction and Edit Branch Prediction: The content branch prediction head uses a fully connected classification layer to represent content-based evidence. Given the input, the output is the branch prediction result:

[0108] The edit branch prediction head uses a fully connected classification layer with the same structure as the content branch prediction head but with independent parameters, representing edit-type evidence. As input, output the edit branch prediction result:

[0109] in, , These represent the weight matrices for the content branch prediction head and the edit branch prediction head, respectively. , This represents the corresponding bias vector. This indicates the content branch prediction head. This represents the edit branch prediction head. The above weight matrix... , and bias vector , All parameters are trainable. The number of rows in the weight matrix equals the number of true categories, the number of columns equals the dimension of the corresponding evidence representation, and the length of the bias vector equals the number of true categories; in this embodiment, the number of true categories is 2. At the start of training, the weights are initialized using a Kaiming uniform distribution based on the corresponding input dimensions, and the biases are initialized using a uniform distribution with the same upper and lower bounds. The absolute values ​​of the initial upper and lower bounds are the reciprocals of the square root of the input dimension. The above parameters are updated through backpropagation based on the supervised classification loss, and their values ​​after training are not limited by the initialization interval.

[0110] S430 determines the policy action from the preset action space based on the discrete policy action distribution, executes the corresponding fusion primitive, and generates the prediction result.

[0111] The preset motion space is represented as follows: A={0,1,2,3} Among them, actions , , and Corresponding to fusion primitives , , and Specifically:

[0112] This indicates an interactive fusion of content branch prediction results and edit branch prediction results:

[0113] This indicates that only content-based evidence branches are used for prediction;

[0114] This indicates that only editable evidence branches are used for prediction;

[0115] This indicates that content-based evidence and editorial evidence are weighted and collaboratively fused based on continuous fusion weights.

[0116] Where w represents the content-based evidence fusion weight, 1 w represents the editable evidence fusion weight; (⊙) represents element-wise multiplication.

[0117] S440, Model Training: In the warm-up training phase, all four fusion primitives participate in the calculation of the fusion result, thus enabling them to receive gradients and update parameters based on the classification error. This prevents the model from prematurely fixing its choice of a particular policy action in the early stages of training. The model parameters, evidence representation, and preliminary action probability distribution obtained during warm-up training serve as the initialization results for the policy optimization phase. The paper also employs the probability weighting results of the four fusion primitives in the warm-up phase, ensuring that all fusion primitives receive gradient signals. The prediction results are expressed as follows:

[0118] in, Indicates the policy state Next choice The probability of each policy action, the policy state Indicates the first The dual-evidence state corresponding to each video sample is constituted by the content-based evidence representation, the editing-based evidence representation, and the differences between the two. ,in, This represents the content-based evidence representation of the i-th video sample. Indicates the first Edited evidence from a single video sample indicates that, This indicates the element-by-element absolute difference between two types of evidence, reflecting the degree of agreement or conflict between them. (Subscript) This represents the i-th video sample in the training batch or dataset. This indicates the i-th sample corresponding to the i-th sample. A fusion primitive.

[0119] During the policy optimization phase, the model parameters obtained from the pre-training are used, and a policy action is selected for each video sample based on the discrete policy action distribution:

[0120] The sampling is not a random selection from four actions with equal probability, but rather sampling based on the action probabilities output by the hybrid policy gating network. The higher the action probability, the higher the likelihood of it being selected. Then, the fusion primitive corresponding to the sampled action is executed.

[0121] in, Indicates the first In the video samples, the action The corresponding fusion primitives, This indicates the prediction result obtained after executing the fusion primitive.

[0122] The policy optimization phase employs probabilistic sampling to allow the model to explore different policy actions during training and to determine whether the sampled actions are suitable for the current sample based on subsequent rewards, thereby optimizing the action probability distribution. During the model inference phase, probabilistic sampling is no longer performed; instead, the policy action with the highest probability is selected. In this embodiment, the policy optimization process also samples actions from the discrete action distribution and executes the corresponding fusion primitive.

[0123] The relationship between the warm-up training phase and the policy optimization phase is as follows: the warm-up training phase uses a soft-weighting method to initially train the evidence encoder, branch prediction head, and hybrid policy gating network; the policy optimization phase, based on the results of the warm-up training, further optimizes the policy action selection strategy corresponding to different video samples through discrete action sampling and reward feedback. The two phases are executed sequentially and together complete the model training.

[0124] Regarding the first Construct a reward function for each sample:

[0125] in, Indicates the first The total reward obtained by the policy action sampled for each video sample; This represents the detection reward, used to evaluate the predictive effectiveness of the prediction results on the true labels; This represents the branch importance reward, used to evaluate whether the sampled action is biased towards evidence branches that are more reliable in judging the current sample; This represents the regularization term for the fusion weights, used to constrain continuous fusion weights; and This indicates the weighting coefficient of the corresponding reward item; and All values ​​are non-negative and determined based on the magnitude of the branch importance reward and fusion weight regularization term relative to the detection reward, the detection performance on the validation set, and the stability of policy training. In this embodiment, γ and β can be adjusted within the range of 0 to 1, so that the auxiliary reward can guide policy selection while preventing its contribution from exceeding that of the detection reward, which is centered on detection accuracy.

[0126] The reward for testing is represented as follows:

[0127] in:

[0128]

[0129] in, Indicates the first The prediction result obtained after executing the sampled strategy action on a video sample; Indicates the category in the prediction results The corresponding unnormalized prediction score; Indicates to The class probability vector obtained after executing the Softmax function; This indicates that the prediction result is assigned to the first... True category of each sample The predicted probability; The cross-entropy loss, representing the difference between the predicted result and the true label, is calculated as follows:

[0130] Therefore, the detection reward is the negative of the cross-entropy loss. The higher the probability the model assigns to the true class, the lower the cross-entropy loss and the higher the detection reward; conversely, the lower the probability the model assigns to the true class, the lower the detection reward.

[0131] Branch importance rewards are represented as follows:

[0132] in:

[0133]

[0134] in, Indicates the first Branch importance reward for each video sample; superscript Indicates the importance of branches; and They represent the first Content branch prediction results and edit branch prediction results for each video sample; superscript Indicates a branch of content-based evidence, indicated by a superscript. Indicates editable evidence branches; and These represent the category probability vectors obtained after applying the Softmax function to the content branch prediction results and the edit branch prediction results, respectively. Indicates the first The authenticity label of each video sample; and These represent the content branch and the edit branch in relation to the actual category, respectively. Assigned predicted probabilities; square brackets This represents the probability of extracting the true class from the class probability vector; Indicates the first Branch preference indicator for each video sample; This represents the hyperbolic tangent function, used to limit the reward based on branch importance. Within the range.

[0135] Branch importance reward is calculated based on the difference in the predicted probability of the true category between the content branch and the editing branch, as well as the branch preference reflected in the current sampling action. When the sampling action is biased towards the evidence branch that predicts more accurately, the branch importance reward is positive; when the sampling action is biased towards the evidence branch that predicts less accurately, the branch importance reward is negative.

[0136] Branch preference indicator Represented as:

[0137] in, Indicates that for the first The strategy action is obtained from sampling video samples; when At that time, execute the interactive fusion primitive. This action does not favor either content branches or editing branches, therefore... ;when When executing the content branch selection primitive. This indicates a bias towards content-based evidence, therefore... ;when When executing the edit branch, select the primitive. This indicates a bias towards editable evidence, therefore... ;when At that time, the weighted collaborative fusion primitives are executed. ,make The branch preference direction and preference degree are represented by the continuous fusion weight.

[0138] The fusion weight regularization term is expressed as:

[0139] in, Indicates the first The fusion weight regularization term for each video sample; superscript The regularization term is used to constrain the continuous fusion weights; Indicates the first Content-based evidence fusion weights for each video sample; This indicates the degree of deviation of the content-based evidence fusion weights from the balanced weight of 0.5; This represents the fusion weight regularization control parameter. When... When, constraints are imposed on the fusion weights that deviate from the equilibrium state to improve the stability of the fusion process; when In this case, the fusion weights are encouraged to tilt towards one of the evidence branches to improve the clarity of the fusion decision.

[0140] Update the moving reward baseline based on the average reward of the batch samples:

[0141]

[0142] in, This represents the average reward for all video samples in the current training batch; This indicates the number of video samples included in the current training batch; Indicates the first The total reward for each video sample; This represents the moving reward baseline, used to characterize the average reward level of the model during recent training. The momentum coefficient representing the moving reward baseline; symbol " This indicates that the moving reward baseline on the left is updated using the result calculated on the right side of the formula. The moving reward baseline is used when the model begins training. Initialize to 0; after the reward calculation is completed in each training batch, use the average reward of the current batch. The moving reward baseline is updated. μ is determined based on the volatility of batch rewards and the required smoothness of the moving baseline, balancing responsiveness to recent reward changes with the stability of the baseline estimate; a larger μ is used. Makes the mobile reward baseline change smoother and smaller. This allows it to reflect changes in rewards for the current batch more quickly.

[0143] And calculate the advantage value of the policy action corresponding to the (i)th sample:

[0144] in, Indicates the first The advantage value of the strategy action sampled for each video sample; This represents the total reward obtained by the policy action sampled in this sample; This indicates the mobile reward baseline.

[0145] The strategy loss is expressed as:

[0146] in, Indicates the strategy loss; This indicates the number of video samples in the current training batch; Indicates the first in the current training batch One video sample; Indicates the first The advantage value of the strategy action sampled for each video sample; Indicates that for the first The strategy action is obtained by sampling a video sample; Indicates the first The dual-evidence strategy state corresponding to each video sample; Indicates the policy state Select the sampled action below The probability of; Represents the entropy regularity coefficient; This represents the entropy function.

[0147] The supervised classification loss is expressed as:

[0148] in, Indicates supervised classification loss; subscript This indicates supervised learning; Indicates the first Prediction results for one video sample; Indicates the first Content branch prediction results for each video sample; Indicates the first Edit branch prediction results for one video sample; Indicates the first The authenticity label of each video sample; Represents the cross-entropy loss function; and These represent the weight coefficients of the content branch auxiliary classification loss and the editing branch auxiliary classification loss, respectively. and All values ​​are non-negative and determined based on the magnitudes of the main classification loss and the two auxiliary classification losses, as well as the detection performance on the validation set. In this embodiment, and The weight can be adjusted within the range of 0 to 1, so that the two evidence branches can obtain auxiliary supervision without weakening the role of the fusion prediction result in the main classification target; when either weight is 0, it means that the auxiliary classification loss of the corresponding branch is not included.

[0149] The total loss during training is expressed as:

[0150] in, This represents the total loss during the model training process; Indicates supervised classification loss; Indicates the strategy loss; subscript This indicates optimization of reinforcement learning strategies; The weighting coefficients representing the policy loss are used to adjust the contribution ratio between the supervised classification objective and the policy optimization objective. The value is taken to be non-negative and determined based on the magnitudes of the supervised classification loss and policy loss, the detection performance on the validation set, and the training stability. In this embodiment, The parameter can be adjusted within the range of 0 to 1, allowing both the strategy optimization objective and the supervised classification objective to participate in parameter updates; when When the value is 0, it means that the strategy loss is not included.

[0151] S450, during the inference phase, selects the action with the highest probability of the discrete policy action as the target action:

[0152] Execute the fusion primitive corresponding to the target action to obtain the final prediction result:

[0153] Step S500: Input the unnormalized prediction result into the linear classification layer, and obtain the binary classification probability of whether the video to be detected is real or fake by softmax normalization. Select the category corresponding to the larger probability as the final video authenticity detection result.

[0154] Specifically, the Softmax function is used to obtain the predicted probability of the video to be detected belonging to each category, and the category with the highest predicted probability is taken as the final realism detection result:

[0155] in, and All indicate the first The final authenticity test result of the video to be tested. Category 0 represents real videos, and Category 1 represents fake videos.

[0156] The embodiments of this invention have achieved positive results during the research and development and testing process. The technical effects of this invention are explained below with reference to experimental data.

[0157] This experiment uses two video authenticity detection datasets: FakeSV and FakeTT. FakeSV is a Chinese short video fake information dataset, sourced from Douyin (TikTok), containing 3624 videos, of which 1810 are fake and 1814 are real. FakeTT is an English short video fake information dataset, sourced from TikTok, containing 1991 videos, of which 1172 are fake and 819 are real.

[0158] Table 1 shows the size of the test dataset; Table 1 also shows the source information of the image dataset used in the test. Table 1 Test Dataset Size

[0159] This experiment uses pre-extracted text, visual, audio, OCR, and video clip-level features, without end-to-end fine-tuning of the original feature extraction model. The model is trained using the Adam optimizer, with the auxiliary classification loss weights for both the content branch and the edit branch set to 0.01. The hidden dimension of the policy gating network is set to 256, and the optimal model parameters are selected based on the validation set accuracy.

[0160] This invention uses a dual-branch encoding of content-based and edit-based evidence to capture different types of authenticity cues, such as semantics, sentiment, OCR, and clip editing. By introducing evidence differences into the policy state, it can identify consistency or conflict relationships between different pieces of evidence. Through a hybrid policy gating network that jointly outputs discrete actions and continuous weights, it achieves synergistic regulation of the fusion structure and evidence contribution. Through reward-driven policy optimization, the model can learn a sample-level fusion strategy based on the final detection results.

[0161] The above specific examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the ideas of this invention. It is impossible to exhaustively list all possible embodiments here; all obvious variations or modifications derived from the technical solutions of this invention are still within the protection scope of this invention.

Claims

1. A multimodal video authenticity detection method based on policy-driven hybrid gating, characterized in that, The method includes: Multimodal feature extraction is performed on the video to be detected to obtain a multimodal feature set; The features in the multimodal feature set are divided into content-based evidence and editing-based evidence, and the content-based evidence and editing-based evidence are encoded to obtain content-based evidence encoding vectors and editing-based evidence encoding vectors; Based on the differences between the content-based evidence encoding vector and the edit-based evidence encoding vector, a dual-evidence strategy state is constructed; The dual-evidence strategy state is input into a pre-trained hybrid strategy gating network, which outputs the prediction probabilities of different strategy actions. The strategy actions include using only the individual prediction results corresponding to content-type evidence, using only the individual prediction results corresponding to editing-type evidence, performing interactive fusion on the individual prediction results of the two types of evidence, and performing weighted fusion on the individual prediction results of the two types of evidence. The strategy action corresponding to the maximum prediction probability is selected and executed to obtain the unnormalized prediction result. The unnormalized prediction results are input into a linear classification layer, and after softmax normalization, a binary classification probability of whether the video to be detected is real or fake is obtained. The category corresponding to the larger probability is selected as the final video authenticity detection result.

2. The multimodal video authenticity detection method based on policy-driven hybrid gating as described in claim 1, characterized in that, Multimodal feature extraction is performed on the video to be detected to obtain a multimodal feature set, specifically including: Multimodal feature extraction is performed on the video to be tested, including text semantic features, text emotion features, visual semantic features, audio emotion features, OCR-related editing features, and video segment-level editing features; Among them, the text semantic features are used to characterize the meaning of the text expressed in the video, including subtitle text, video title, video description and speech-to-text text; the text emotion features are used to characterize the emotional state contained in the text; the visual semantic features are used to characterize the semantic information of the picture in the video key frame or sample frame, including people, objects, scenes, people's postures and their spatial relationships; and the audio emotion features are used to characterize the voice emotion state conveyed by the audio acoustic signal. The OCR-related editing features include OCR phrase features and OCR visual pattern features; OCR phrase features are used to represent the text content in the video frame obtained based on OCR recognition; OCR visual pattern features are used to represent the text layout, color, position, and display format in the video frame obtained based on OCR recognition. The video segment-level editing features include segment-level editing content features, segment duration features, and segment temporal position features. The segment-level editing content features are used to characterize the spatiotemporal content within a video segment and the editing information between adjacent segments, including image changes, motion changes, and content jumps. The segment duration features and segment temporal position features are used to represent the segment length and the segment's position on the video timeline, respectively.

3. The multimodal video authenticity detection method based on policy-driven hybrid gating as described in claim 2, characterized in that, The features in the multimodal feature set are divided into content-based evidence and editing-based evidence, specifically including: Textual semantic features, textual emotional features, visual semantic features, and audio emotional features are classified as content-based evidence; OCR-related editing features, including OCR phrase features and OCR visual pattern features, as well as video segment-level editing features, including segment-level edited content features, segment duration features, and segment temporal location features, are classified as editing evidence. The text sentiment features and audio sentiment features in the content-type evidence are fed into the Transformer-based sentiment evidence encoding module. By performing modal alignment, Transformer encoding and mean pooling on the text sentiment features and audio sentiment features, a unified sentiment evidence vector is output. The textual and visual semantic features in the content-based evidence are fed into the Transformer-based semantic evidence encoding module. By performing modal alignment, Transformer encoding, cross-modal attention operation, and mean pooling on the textual and visual semantic features, a unified semantic evidence vector is output. The OCR phrase features and OCR visual pattern features in the edited evidence are fed into the Transformer-based OCR evidence encoding module. The OCR phrase features and OCR visual pattern features are processed by modal alignment, Transformer encoding and mean pooling to output a unified OCR evidence vector. The fragment-level edited content features, fragment duration features, and fragment time position features in the edited evidence are fed into the Transformer-based OCR evidence encoding module. The OCR phrase features and OCR visual pattern features are processed by modal alignment, Transformer encoding, and mean pooling to output a unified fragment edited evidence vector. The obtained unified sentiment evidence vector and unified semantic evidence vector are concatenated to obtain the content-based evidence encoding vector; The obtained unified OCR evidence vector and unified fragment editing evidence vector are concatenated to obtain the editing evidence encoding vector.

4. The multimodal video authenticity detection method based on policy-driven hybrid gating as described in claim 1, characterized in that, Based on the differences between the content-based evidence encoding vector and the edit-based evidence encoding vector, a dual-evidence strategy state is constructed, specifically including: in, Indicates the state of the dual-evidence strategy. Represents the encoding vector of content-based evidence and editable evidence encoding vector The absolute differences between elements.

5. The multimodal video authenticity detection method based on policy-driven hybrid gating as described in claim 1, characterized in that, The hybrid policy gating network includes a policy encoder, an action mapping branch, and a weight mapping branch; the output of the policy encoder is connected to the action mapping branch and the weight mapping branch, respectively. The policy encoder comprises a first fully connected layer, a first ReLU activation layer, a second fully connected layer, and a second ReLU activation layer connected in sequence, wherein the first fully connected layer operates in a dual-evidence policy state. As input, intermediate features are obtained after linear transformation and ReLU activation. Then the intermediate features The hidden decision representation is obtained by inputting it into the second fully connected layer and then activating it with ReLU. : in, and These represent the weight matrices of the first and second fully connected layers, respectively. and These represent the corresponding bias vectors; Indicates the policy encoder; The obtained hidden decision representation Input the action mapping branch and the weight mapping branch respectively; The action mapping branch includes a third fully connected layer with an output dimension of 4 and a Softmax layer; the third fully connected layer hides the decision representation. Mapped to action scores corresponding to the four policy actions: Then, the probability distribution of policy actions is obtained through the Softmax layer: in, This represents the weight matrix of the fourth fully connected layer. This represents the corresponding bias vector; Represents a discrete action mapping function; This indicates that, given a dual-evidence strategy state Under the given conditions, the probability of selecting strategy action a; The weight mapping branch includes a fourth fully connected layer with an output dimension of 1 and a Sigmoid layer; the fourth fully connected layer hides the decision representation. Mapped to fusion weight score: in, Represents the weight mapping function; Represents the Sigmoid function; This represents the fusion weights used for weighted fusion. .

6. The multimodal video authenticity detection method based on policy-driven hybrid gating as described in claim 5, characterized in that, The strategy actions include using only the individual prediction results corresponding to content-based evidence, using only the individual prediction results corresponding to editorial evidence, performing interactive fusion of the individual prediction results of the two types of evidence, and performing weighted fusion of the individual prediction results of the two types of evidence, specifically including: Establish an action space A = {0, 1, 2, 3}, where action 0: performs interactive fusion of the individual prediction results of the two types of evidence; action 1: uses only the individual prediction results corresponding to content-type evidence; action 2: uses only the individual prediction results corresponding to edit-type evidence; and action 3: performs weighted fusion of the individual prediction results of the two types of evidence. Let... , , , These are the prediction results for executing the corresponding actions 0, 1, 2, and 3, respectively. in, To use only the individual prediction results corresponding to content-based evidence, the content-based evidence encoding vector is... The video authenticity prediction result is output after inputting the content branch prediction network used for binary classification task. in, To use only the individual prediction results corresponding to edited evidence, the edited evidence encoding vector is... The video authenticity prediction result is output after inputting the edit branch prediction network used for binary classification task; in, represents the hyperbolic tangent activation function, and ⊙ represents element-wise multiplication; in, This represents the fusion weights, output by the weight mapping branch.

7. The multimodal video authenticity detection method based on policy-driven hybrid gating as described in claim 1, characterized in that, The method further includes: During the training process, a two-stage training strategy, including a warm-up training stage and a strategy optimization stage, is adopted to train the network model. In the pre-training phase, the probabilities of each action output by the hybrid policy gating network are used as weighting coefficients to sum the execution results of each action in a weighted manner, thereby obtaining the prediction results of the video samples. This avoids the model prematurely selecting a certain action in the early stages of training. in, Indicates the policy state Next choice The probability of a strategy action. Indicates the first The execution corresponding to the video sample of the _th video sample The prediction result for each action; This indicates the first stage of the warm-up training. Prediction results for each video sample; During the policy optimization phase, the network model parameters obtained from the pre-training are used, and actions are sampled based on the action probabilities output by the hybrid policy gating network. The sampled actions are then executed to obtain the corresponding prediction results. The reward function is used to determine whether the sampled actions are suitable for the current sample, thereby optimizing the action probability distribution. in, This indicates the first stage of the strategy optimization phase. The prediction results for each video sample This indicates the action to be performed corresponding to the i-th video sample. The prediction results.

8. The multimodal video authenticity detection method based on policy-driven hybrid gating as described in claim 7, characterized in that, During the strategy optimization phase, for the first... Construct a reward function from video samples: in, Indicates the first The total reward obtained for the sampled action corresponding to each video sample; This represents the detection reward, used to evaluate the predictive effectiveness of the prediction results on the true labels; This represents the branch importance reward, used to evaluate whether the sampled action is biased towards evidence branches that are more reliable in judging the current sample; This represents the regularization term for the fusion weights, used to constrain the fusion weights; and This represents the weighting coefficient of the corresponding reward item; The reward for testing is represented as follows: in, Indicates the first The prediction result obtained after performing the sampled action on a video sample; is the true category label for the i-th video sample, where a value of 0 indicates a real video and a value of 1 indicates a fake video; Indicates to The class probability vector obtained after executing the Softmax function; Represents the true class label extracted from the class probability vector. The corresponding predicted probability; This represents the cross-entropy loss between the predicted result and the true classification label; Branch importance rewards are represented as follows: in: in, Indicates the first Branch importance reward for each video sample; and They represent the first Individual prediction results for each video sample using only content-based evidence and individual prediction results using only editing-based evidence. and They are respectively for and The class probability vector obtained after executing the Softmax function; Indicates the first The true classification labels of each video sample; and From the class probability vector and The real category extracted from The corresponding predicted probability; Indicates the first Branch preference indicator for each video sample; This represents the hyperbolic tangent activation function, used to limit the branch importance reward to a certain value. Within the range; Among them, the branch preference indicator Represented as: in, Indicates that for the first Actions obtained from sampling video samples; when At this time, the prediction results of the two types of evidence are interactively fused, without bias towards either the content branch or the editing branch, thus making ;when When only the prediction result corresponding to content-based evidence is used, it indicates a bias towards the content-based evidence branch, therefore let ;when When only the individual prediction result corresponding to edited evidence is used, it indicates a bias towards the edited evidence branch, therefore let ;when At that time, a weighted fusion is performed on the individual prediction results of the two types of evidence, making According to the fusion weight Determine the direction and degree of branch preference; The fusion weight regularization term is expressed as: in, Indicates the first The fusion weight regularization term for each video sample; Indicates the first The fusion weights of each video sample; This indicates the degree of deviation of the fusion weight from the balance weight of 0.5; This represents the fusion weight regularization control parameter; when When, constraints are imposed on the fusion weights that deviate from the equilibrium state to improve the stability of the fusion process; when In this case, the fusion weights are encouraged to tilt towards one of the evidence branches to improve the clarity of the fusion decision.

9. The multimodal video authenticity detection method based on policy-driven hybrid gating as described in claim 8, characterized in that, During the strategy optimization phase, the moving reward baseline is updated based on the average reward of the batch of samples: in, This represents the average reward for all video samples in the current training batch; This indicates the number of video samples included in the current training batch; Indicates the first The total reward for each video sample; This represents the moving reward baseline, used to characterize the average reward level of the model during training; The momentum coefficient representing the moving reward baseline; symbol " "" indicates that the moving reward baseline on the left is updated using the calculation result on the right side of the symbol; the moving reward baseline is set when the model starts training. Initialize to 0; after the reward calculation is completed in each training batch, use the average reward of the current batch. Update the mobile rewards baseline.

10. The multimodal video authenticity detection method based on policy-driven hybrid gating as described in claim 7, characterized in that, The method further includes: The total loss during training is expressed as: in, This represents the total loss during the model training process; Indicates supervised classification loss; Indicates the strategy loss; The weighting coefficients representing the policy loss are used to adjust the contribution ratio between the supervised classification objective and the policy optimization objective. The supervised classification loss is expressed as: in, Indicates supervised classification loss; Indicates the first Prediction results for one video sample; and They represent the first Individual prediction results for each video sample using only content-based evidence and individual prediction results using only editing-based evidence; Indicates the first The true classification labels of each video sample; Represents the cross-entropy loss function; and These represent the corresponding weighting coefficients; The strategy loss is expressed as: in, Indicates the strategy loss; This indicates the number of video samples in the current training batch; Indicates the first in the current training batch One video sample; Indicates the first The actions sampled from each video sample The advantage value; Indicates the first The dual-evidence strategy state corresponding to each video sample; Indicating a dual-evidence strategy state Select the sampled action below The probability of; Represents the entropy regularity coefficient; This represents the entropy function.

Images