Video detection method, apparatus, device, storage medium, and program product

Abstract

This specification provides a video detection method, apparatus, device, storage medium, and program product. Specifically, multiple video frames are acquired from the video to be detected, and multimodal features of the multiple video frames are extracted; based on the multimodal features of the multiple video frames, a multimodal feature sequence is generated, and word embedding processing is performed on the speech content text of the multiple video frames to obtain a word embedding representation sequence; the multimodal feature sequence is enhanced using a spatiotemporal fusion network to obtain an enhanced multimodal feature sequence; based on the enhanced multimodal feature sequence, the word embedding representation sequence, and video detection task prompts, a multimodal large language model is triggered to determine the video detection result of the video to be detected. This improves the accuracy of video authenticity detection.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a video detection method, apparatus, device, storage medium, and program product. Background Technology

[0002] In high-security applications such as remote account opening, online identity verification, and video interviews in the financial sector, it is necessary to verify the authenticity of user-submitted videos. Traditional video verification methods typically rely on the analysis of individual video frames, mainly judging the authenticity of videos by identifying static visual features such as local texture anomalies, boundary discontinuities, and inconsistent lighting. However, this approach suffers from low accuracy. Summary of the Invention

[0003] This specification provides a video detection method, apparatus, device, storage medium, and program product to improve the accuracy of video authenticity detection.

[0004] This specification provides a video detection method, comprising: acquiring multiple video frames from a video to be detected and extracting multimodal features from the multiple video frames; generating a multimodal feature sequence based on the multimodal features of the multiple video frames, and performing word embedding processing on the speech content text of the multiple video frames to obtain a word embedding representation sequence; using a spatiotemporal fusion network to enhance the features of the multimodal feature sequence to obtain an enhanced multimodal feature sequence; and triggering a multimodal large language model to determine the video detection result of the video to be detected based on the enhanced multimodal feature sequence, the word embedding representation sequence, and video detection task prompt words.

[0005] This specification also provides a video detection device, comprising: an acquisition module, configured to acquire multiple video frames from a video to be detected and extract multimodal features from the multiple video frames; generate a multimodal feature sequence based on the multimodal features of the multiple video frames, and perform word embedding processing on the speech content text of the multiple video frames to obtain a word embedding representation sequence; a feature enhancement module, configured to enhance the multimodal feature sequence using a spatiotemporal fusion network to obtain an enhanced multimodal feature sequence; and a video detection module, configured to trigger a multimodal large language model to determine the video detection result of the video to be detected based on the enhanced multimodal feature sequence, the word embedding representation sequence, and video detection task prompt words.

[0006] This specification also provides an electronic device, including: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute one or more computer instructions to perform the steps in the method provided in this specification.

[0007] This specification also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps of the method provided in this specification.

[0008] This specification also provides a computer program product, including: a computer program / instructions, which, when executed by a processor, can implement the steps of the method provided in this specification.

[0009] In this embodiment, multiple video frames are acquired from the video to be detected, and multimodal features of these frames are extracted. A multimodal feature sequence is generated based on these features, and word embedding processing is performed on the speech content text of the multiple video frames to obtain a word embedding representation sequence. A spatiotemporal fusion network is used to enhance the multimodal feature sequence, resulting in an enhanced multimodal feature sequence. Based on the enhanced multimodal feature sequence, the word embedding representation sequence, and video detection task prompts, a multimodal large language model is triggered to determine the video detection result of the video to be detected. Therefore, a video detection process of "extracting multimodal feature sequences from multiple video frames, enhancing multimodal feature sequences based on a spatiotemporal fusion network, and inferring based on a multimodal large language model" is provided, achieving efficient and accurate video authenticity detection and providing reliable technical support for high-security scenarios such as finance and government affairs. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings: Figure 1 A flowchart illustrating a video detection method provided as an exemplary embodiment of this specification; Figure 2 A diagram illustrating a video detection process provided as an exemplary embodiment of this specification; Figure 3 A schematic diagram of the structure of a video detection device provided for an exemplary embodiment of this specification; Figure 4 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this specification. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0012] It should be noted that, in the case of user information involved in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0013] The various models involved in this application (including but not limited to language models or large language models) comply with relevant laws and standards.

[0014] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0015] Figure 1 This is a flowchart illustrating a video detection method provided as an exemplary embodiment of this specification. Figure 1 As shown, the method may include the following steps: 101. Obtain multiple video frames from the video to be detected, and extract multimodal features from the multiple video frames.

[0016] Specifically, the goal of video authenticity detection is to determine whether a video was filmed in a real-world setting. In practice, some videos are not actually filmed in real-world locations, but are fake videos (also known as fraudulent videos) generated through artificial intelligence (AI) synthesis, screen replay, or photo manipulation.

[0017] For ease of understanding, the video whose authenticity needs to be verified is referred to as the "video to be tested." First, the video to be tested is sampled frame by frame at a set sampling rate, resulting in P video frames, where P is a positive integer. For example, a sampling rate of 20 frames per second is set, meaning 20 frames are uniformly extracted from the video to be tested every second. Finally, the total number of sampled frames P is determined based on the video's duration.

[0018] Next, multimodal features are extracted from each of the P video frames to obtain the multimodal features of each video frame. The multimodal features of a video frame can include feature data from multiple modalities, and there is no limit to the number of modalities.

[0019] In real-world scenarios, real people typically speak with coordinated mouth movements, facial expressions, and speech output. These different modalities are highly aligned temporally and exhibit semantic consistency. However, forged videos often struggle to fully simulate this complex cross-modal coordination during generation, especially in non-frontal poses, large-angle movements, or complex contexts, where inconsistencies are more easily revealed. Therefore, to improve the accuracy of video authenticity detection, one approach to extracting multimodal features from multiple video frames is as follows: For any given video frame, extract its frame ID, facial visual features, facial state features, facial motion features, and speech-lip-sync information; extract the speech segment from the video stream and extract its acoustic features; perform speech-to-text processing on the speech segment to obtain the speech content text; and unify the format of the frame ID, facial visual features, facial state features, facial motion features, speech content text, acoustic features, and speech-lip-sync information to obtain the multimodal features of the video frame.

[0020] Specifically, a video frame ID (identifier) ​​is a unique identifier assigned to each video frame during video processing, typically indicating the frame's sequential position within the video to be detected. As a temporal anchor, the frame ID aligns various modal information at the same moment, ensuring consistency of cross-modal information over time. This effectively supports the modeling and analysis of inter-frame temporal evolution patterns (such as facial expression changes, lip movements, and the synchronization of speech and lip movements).

[0021] For any given video frame, face detection can be performed to identify the facial regions within the frame. Visual features can then be extracted from these regions to obtain the corresponding facial visual features. These features include, but are not limited to, texture features (such as skin details and pore distribution), color features, and geometric features (such as the relative positions of facial features and facial contours). Facial visual features can reflect the authenticity of a real face. In video authenticity detection, facial visual features help identify common anomalies in fake videos, such as blurred boundaries, unnatural lighting transitions, distorted geometric structures, or repetitive textures.

[0022] For any given video frame, facial pose recognition, facial size recognition, or facial orientation recognition are performed on the facial region within the frame to obtain facial state features. These features include, for example, facial pose information, facial size information, and facial orientation information. Facial state features not only reflect the natural behavioral patterns of real people in videos (such as slight head turns, moving closer to or away from the camera), but also effectively identify common spatial inconsistencies in forged videos, such as abrupt pose changes, inconsistent viewpoint transitions, and abrupt changes in facial scale.

[0023] For any given video frame, facial motion features are extracted from the facial region within the frame to obtain the facial motion features of the video frame. These facial motion features include, but are not limited to, mouth motion information (such as the degree of lip opening and closing, mouth opening or closing) and facial expression motion information (such as smiling, crying, frowning, surprise, and anger). These facial motion features can finely depict the natural temporal dynamics of the face during speech or facial expression, providing crucial visual evidence for speech-lip-sync analysis and effectively identifying common anomalies in forged videos, such as stiff movements, incoherent expressions, and discrepancies between lip movements and speech semantics.

[0024] For any given video frame, the audio segment of the video frame is extracted from the video to be detected, and the acoustic features of the audio segment are extracted. The audio segment is then processed by Automatic Speech Recognition (ASR) to obtain the audio content text.

[0025] Specifically, extracting a speech segment that is time-aligned with the video frame from the video to be detected can be done by extracting a short speech segment from the audio track of the video to be detected based on the timestamp of the video frame. For example, a speech segment within a time window centered on the timestamp of the video frame and extending 100 to 200 milliseconds before and after it, to ensure that the extracted speech segment is synchronized with the visual content such as mouth movements and facial expressions of the current video frame in terms of semantics and temporal sequence.

[0026] For any given video frame, the synchronization information between speech and lip movements is also extracted. This information measures the degree of synchronization between mouth movements and speech content within a video frame, reflecting whether lip movements and audio signals match naturally during speech. In real videos, mouth movements are usually highly synchronized with speech, while in fake videos, abnormal phenomena such as misalignment, delay, or mismatch between lip movements and speech often occur. Real videos can refer to video content directly recorded in the real world using physical imaging devices (such as mobile phones, cameras, webcams, etc.), where the scenes, people, actions, and sounds in the footage all originate from real events. Fake or virtual videos can refer to videos that are entirely or partially computer-generated or deepfake, relying not on (or only partially on) direct filming of the real world, but synthesized through algorithms, models, or graphics engines.

[0027] Optionally, in order to accurately extract the audio-lip-sync information of the video frame, a audio-lip-sync detection model can be trained. Based on this, the method for extracting the audio-lip-sync information of the video frame is as follows: obtain the mouth video, including the video frame, and the corresponding audio stream from the video to be detected; input the mouth video and its audio stream into the audio-lip-sync detection model to obtain the audio-lip-sync information of the video frame.

[0028] Specifically, several consecutive video frames, including the target video frame, can be extracted from the video to be detected. The mouth region images within these consecutive video frames are then used as mouth video segments, along with the temporally aligned audio stream corresponding to each mouth video segment. The mouth video and its audio stream are input into a speech and lip-sync detection model for processing, obtaining the speech and lip-sync information of the video frames output by the model. This speech and lip-sync information can be a confidence score between 0 and 1. A higher confidence score indicates that the mouth movements and speech content in the current video frame are more consistent and conform to the natural laws of human speech; conversely, a lower confidence score indicates that the mouth movements and speech content in the current video frame are less consistent and do not conform to the natural laws of human speech.

[0029] In practical applications, speech-lip-sync detection models can be various machine learning models trained using training samples. For example, first, a large number of sample videos are prepared, including both real and fake videos. These videos are then labeled; real videos are labeled to indicate speech-lip-sync synchronization, while fake videos are labeled to indicate asynchrony. Next, the visual encoder in the speech-lip-sync detection model extracts mouth movement features from the mouth segments in the sample videos; the audio encoder extracts acoustic representations from the corresponding audio segments. Then, the degree of speech-lip-sync synchronization in the sample videos output by the model is obtained. Finally, a loss function (such as cross-entropy loss) is calculated based on the degree of speech-lip-sync synchronization and the corresponding standard labels. This loss function is then used to adjust the speech-lip-sync detection model until the model's iterative training terminates.

[0030] Optionally, the speech and lip-sync detection model is obtained as follows: positive and negative samples are acquired. Positive samples include lip region image sequences extracted from real videos and their time-synchronized audio segments, while negative samples include lip region image sequences and their time-unsynchronized audio segments. An initial speech and lip-sync detection model containing a visual encoder and an audio encoder is constructed. The visual encoder is used to extract mouth movement features from the lip region image sequences, and the audio encoder is used to extract acoustic representations from the audio segments. Based on the mouth movement features and acoustic representations corresponding to the positive and negative samples, the initial speech and lip-sync detection model is trained through comparative learning to obtain a trained speech and lip-sync detection model.

[0031] Specifically, through comparative learning, the speech and lip-sync detection model achieves better performance in speech and lip-sync detection. Positive samples enable the speech and lip-sync detection model to learn the temporal and semantic synchronicity between mouth movements and speech signals during natural human speech; negative samples enable the speech and lip-sync detection model to prioritize the identification of common anomalies such as "lip-sync misalignment".

[0032] During the contrastive learning training process, the mouth movement features and acoustic representations corresponding to positive samples are input into the contrastive learning loss function to obtain the contrastive loss value for positive samples; similarly, the mouth movement features and acoustic representations corresponding to negative samples are input into the contrastive learning loss function to obtain the contrastive loss value for negative samples. Based on the contrastive loss values, the model parameters of the speech and lip-sync detection model are adjusted so that the model can narrow the feature distance between the mouth movement features and acoustic representations of positive samples and widen the feature distance between the mouth movement features and acoustic representations of negative samples in the feature space. The speech and lip-sync detection model trained in this way has stronger generalization ability and robustness, and can accurately evaluate the audio-visual consistency of each video frame in complex scenes (such as side profiles, lighting changes, background noise, etc.), providing a highly reliable cross-modal verification basis for video authenticity detection.

[0033] Contrastive loss functions include, but are not limited to, the contrastive loss function, the triplet loss function, and the N-Pair loss function. These functions distinguish between similar and dissimilar samples by adjusting the distance between samples in the embedding space.

[0034] For any given video frame, after extracting various modal features such as frame ID, facial visual features, facial state features, facial action features, speech content text, acoustic features, and information on the synchronization between speech and lip movements, these modal features can be formatted uniformly and organized into the multimodal features of the video frame.

[0035] Specifically, format unification can map various modal features to a vector space of the same dimension (such as 256, 512 or 768 dimensions), that is, heterogeneous modal features can be format unified in the same semantic space.

[0036] 102. Based on the multimodal features of multiple video frames, generate a multimodal feature sequence, and perform word embedding processing on the speech content text of multiple video frames to obtain a word embedding representation sequence.

[0037] Specifically, the multimodal features of multiple video frames are organized according to their acquisition time sequence to obtain a multimodal feature sequence. This multimodal feature sequence not only contains local information from each video frame but also preserves the temporal relationships between different frames.

[0038] For any given video frame, extract the speech segment that is time-aligned with the video frame from the video to be detected, perform speech-to-text processing on the speech segment to obtain the speech content text of the video frame, and perform word embedding processing on the speech content text to obtain the word embedding representation corresponding to the speech content text. The word embedding representation corresponding to the speech content text can include the word embedding representation of each word segment in the speech content text.

[0039] 103. Use a spatiotemporal fusion network to enhance the features of multimodal feature sequences to obtain enhanced multimodal feature sequences.

[0040] Specifically, spatiotemporal fusion networks are a class of deep learning architectures specifically designed for jointly modeling spatial structure dependencies and temporal dynamic evolution. They integrate multi-scale, multi-modal, or multi-level feature information through effective fusion strategies. Their core objective is to enhance the modeling and prediction capabilities of complex spatiotemporal processes by collaboratively modeling the spatial and temporal dimensions. Preferably, spatiotemporal fusion networks can be based on the Transformer architecture. The Transformer architecture is a neural network architecture based on a self-attention mechanism, processing sequential data through this mechanism. It consists of an encoder and a decoder, employing multi-head attention and positional encoding to capture long-range dependencies.

[0041] Specifically, a multimodal feature sequence is input into a spatiotemporal fusion network. This network uses a self-attention mechanism to dynamically interact and weightedly fuse features across time steps and between different modalities, generating a context-aware enhanced multimodal feature sequence. This enhanced multimodal feature sequence includes multiple enhanced multimodal features. For any given enhanced multimodal feature, it is obtained by fusing various modal features and contextual information from the multimodal features, thus enhancing the features relative to the corresponding multimodal features. The spatiotemporal fusion network performs temporal evolution analysis based on the multimodal feature sequence to obtain contextual information. Each enhanced multimodal feature incorporates cross-modal and cross-temporal contextual information. This enhanced multimodal feature sequence can effectively capture unnatural movements, lip-sync asynchrony, or abrupt facial expressions in forged videos. The spatiotemporal fusion network can perform various temporal evolution analyses, including but not limited to: speech-lip-sync detection analysis, mouth movement abrupt change detection analysis, and naturalness detection analysis of facial expression changes.

[0042] During model training, multimodal feature sequences and their labels are collected from sample videos. The labels indicate whether the sample video is real or fake. The multimodal feature sequences of the sample videos are input into a spatiotemporal fusion network to obtain an enhanced multimodal feature sequence output by the spatiotemporal fusion network. A prediction network, which may include global average pooling layers and fully connected layers, outputs a predicted probability that the sample video is real based on the enhanced multimodal feature sequence. The model parameters of the spatiotemporal fusion network are adjusted using a backpropagation algorithm based on the loss value between the label and the predicted probability of the sample video. Model training is iteratively executed until the termination condition for model training iteration is met. When calculating the loss value, if the label indicates that the sample video is real, the probability of realism corresponding to the label is 100%; if the label indicates that the sample video is fake, the probability of realism corresponding to the label is 0. The probability of realism corresponding to the label and the predicted probability are input into the cross-entropy loss function to obtain the loss value corresponding to the sample video.

[0043] Optionally, the spatiotemporal fusion network includes a first spatiotemporal fusion module, a second spatiotemporal fusion module, a third spatiotemporal fusion module, and a feature fusion module. Correspondingly, the spatiotemporal fusion network is used to enhance the features of the multimodal features, resulting in an enhanced multimodal feature sequence. This includes: using the first spatiotemporal fusion module to perform a speech and lip-sync detection task on the multimodal feature sequence to obtain speech and lip-sync detection results; using the second spatiotemporal fusion module to perform a mouth movement abrupt change detection task on the multimodal feature sequence to obtain mouth movement abrupt change detection results; using the third spatiotemporal fusion module to perform a facial expression change naturalness detection task on the multimodal feature sequence to obtain facial expression change naturalness detection results; for any multimodal feature in the multimodal feature sequence, the feature fusion module fuses the various modal features, the speech and lip-sync detection results, the mouth movement abrupt change detection results, and the facial expression change naturalness detection results to obtain enhanced multimodal features, and an enhanced multimodal feature sequence is obtained based on multiple enhanced multimodal features.

[0044] Specifically, the first, second, and third spatiotemporal fusion modules can all include a Transformer encoder. These modules possess temporal evolution analysis capabilities and temporal data analysis functions. The feature fusion module can be a network module with feature fusion functionality. The spatiotemporal fusion network can perform refined modeling of speech-lip-sync, abrupt changes in mouth movements, and naturalness of facial expression changes, respectively. It deeply fuses these high-order semantic contextual information with multimodal features at the time-step granularity, thereby generating enhanced multimodal feature sequences with strong discriminative power. This improves the perception and recognition of subtle spatiotemporal inconsistencies (such as lip-sync and speech misalignment, stiff expressions, or abrupt movements) in forged videos.

[0045] 104. Based on the enhanced multimodal feature sequence, word embedding representation sequence, and video detection task prompt words, trigger the multimodal large language model to determine the video detection result of the video to be detected.

[0046] In practical applications, multimodal large language models refer to large-scale pre-trained models with the ability to jointly understand images, text, and speech. They can uniformly model multimodal information such as images, text, and audio, and complete cross-modal semantic reasoning tasks.

[0047] Specifically, the video detection task prompts are used to trigger the multimodal large language model to detect the authenticity of the video based on the enhanced multimodal feature sequence and word embedding representation sequence.

[0048] Optionally, to guide the multimodal large language model to perform video authenticity detection efficiently and accurately, the video detection task prompts can include input requirement information and output requirement information. The input requirement information constrains the input information to the multimodal large language model, requiring that the enhanced multimodal feature sequence, word embedding representation sequence, and video detection task prompts be organized into a mixed text-image sequence. The output requirement information constrains the output result of the multimodal large language model, requiring that the video detection result output by the multimodal large language model include video authenticity probability information, natural language explanation information, and speech-visual synchronization information for the video to be detected. The video authenticity probability information characterizes the probability that the video to be detected is a real video; the natural language explanation information is used to explain the reasons for determining the video authenticity probability information using natural language; the speech-visual synchronization information for the video to be detected reflects the overall speech-lip-sync information of the video to be detected. The speech-lip-sync information for the video to be detected can be obtained by weighted averaging of the speech-lip-sync information of multiple video frames corresponding to the video to be detected.

[0049] In practical applications, the enhanced multimodal feature sequence can be feature-mapped to obtain image embedding information that is adapted to the image embedding format of the multimodal large language model. The specific implementation of "organizing the enhanced multimodal feature sequence, word embedding representation sequence and video detection task prompt words into a mixed image and text sequence" is "organizing the image embedding information, word embedding representation sequence and video detection task prompt words corresponding to the enhanced multimodal feature sequence into a mixed image and text sequence".

[0050] Optionally, to better adapt the enhanced multimodal feature sequence to the multimodal large language model, several representative keyframes can be selected based on the video duration and computational resource limitations of the video to be detected. For example, several representative keyframes can be uniformly sampled from multiple video frames corresponding to the enhanced multimodal feature sequence according to temporal relationships. A linear transformation module is then used to map the enhanced multimodal features of each keyframe to image embedding information adapted to the image embedding format of the multimodal large language model, making it compatible with the image encoder output format of the multimodal large language model.

[0051] Based on the above, the implementation method for triggering the multimodal large language model to determine the video detection result of the video to be detected, according to the enhanced multimodal feature sequence, word embedding representation sequence, and video detection task prompts, is as follows: The enhanced multimodal feature sequence, word embedding representation sequence, and video detection task prompts are organized into a mixed image-text sequence according to the input requirements information in the video detection task prompts; the multimodal large language model is triggered to perform video detection operations based on the mixed image-text sequence according to the output requirements information in the video detection task prompts, and outputs the video detection result of the video to be detected; wherein, the video detection result includes: video authenticity probability information, natural language explanation information, and information on the degree of speech and lip-sync for the video to be detected; the natural language explanation information is used to explain the reasons for determining the video authenticity probability information of the video to be detected.

[0052] Specifically, an input-output control mechanism for structured video detection task prompts is introduced, enabling the multimodal large language model to stably output quantifiable, interpretable, and verifiable video detection results.

[0053] For example, the prompt for a video detection task might be: "Please analyze whether the video to be detected is a fake video. The input sequence is a mixed image and text sequence, which is obtained by sequentially organizing the enhanced multimodal feature sequence, word embedding representation sequence, and video detection task prompt. The mixed image and text sequence is then input into a multimodal large language model for joint inference. The output of the multimodal large language model includes: video authenticity probability information, natural language explanation information (such as "speech content does not match lip movements," "speaker did not open their mouth," etc.), and information on the degree of synchronization between speech and lip movements in the video to be detected."

[0054] In practical applications, multimodal large language models used to perform video detection tasks can be pre-trained multimodal large language models. Pre-trained multimodal large language models are obtained by pre-training on massive amounts of multimodal data (such as image-text pairs, video-speech-text triples, etc.) on various general tasks (such as image-text matching, visual question answering, cross-modal retrieval, etc.) using self-supervised or unsupervised methods, and possess strong cross-modal understanding and generalization capabilities.

[0055] Optionally, the multimodal large language model used to perform the video detection task is obtained by fine-tuning the pre-trained multimodal large language model using a labeled dataset related to the video authenticity detection task, so as to adapt to the specific needs of video authenticity detection and thus improve its inference accuracy in the video authenticity detection scenario.

[0056] Specifically, during fine-tuning training, a labeled dataset related to the video authenticity detection task can be obtained. The labeled dataset includes multiple training samples, each of which includes an enhanced multimodal feature sequence, word embedding representation sequence, and labeling results for the sample video. The labeling results include labeled labels and fake type labels, with the labeled labels indicating whether the sample video is a real video or a fake video. The pre-trained multimodal large language model is then fine-tuned using the labeled dataset to obtain the multimodal large language model.

[0057] Specifically, if the label indicates that the sample video is real, the probability of authenticity corresponding to the label is 100%; if the label indicates that the sample video is fake, the probability of authenticity corresponding to the label is 0. Forgery type labels include, but are not limited to: "lip-syncing", "stiff facial expression" or "unnatural facial expression".

[0058] Optionally, fine-tuning the pre-trained multimodal large language model using an labeled dataset to obtain the multimodal large language model includes: for any training sample, inputting the enhanced multimodal feature sequence, word embedding representation sequence, and video detection task prompts of the sample video into the pre-trained multimodal large language model to obtain the video detection results of the sample video output by the pre-trained multimodal large language model; calculating the authenticity classification loss value based on the video authenticity probability information and corresponding annotation labels in the video detection results of the sample video; extracting keywords representing the forgery type from the natural language interpretation in the video detection results of the sample video; determining the forgery type classification loss value based on the keywords representing the forgery type and their corresponding forgery type labels; adjusting the model parameters of the pre-trained multimodal large language model based on the authenticity classification loss value and the forgery type classification loss value, and repeating the above operations until the model iteration termination condition is met to obtain the multimodal large language model.

[0059] Specifically, to further improve the inference accuracy of multimodal large language models in video authenticity detection scenarios, a fine-tuning method using multiple loss functions can be employed. These multiple loss functions include an authenticity classification loss function and a forgery type classification loss function, which can be cross-entropy loss functions.

[0060] For any training sample, the probability information corresponding to the video authenticity probability information and the probability information corresponding to the labeled information are input into the cross-entropy loss function to obtain the authenticity classification loss value. Keywords are extracted from the natural language interpretation in the video detection results of the sample video to extract keywords representing the forgery type; these keywords and their corresponding forgery type labels are input into the cross-entropy loss function to obtain the forgery type classification loss value; the authenticity classification loss value and the forgery type classification loss value corresponding to the training sample are weighted and summed to obtain the total loss value of the training sample. The model parameters of the pre-trained multimodal large language model are adjusted based on the total loss value of the training sample. This completes one model training process. The multimodal large language model trained in the previous iteration is used as the pre-trained multimodal large language model for the next iteration. Multiple training samples are used to perform multiple model training operations until the model iteration termination condition is met, resulting in the final multimodal large language model.

[0061] The technical solution provided in this application involves acquiring multiple video frames from a video to be detected and extracting multimodal features from these frames. Based on these multimodal features, a multimodal feature sequence is generated, and word embedding processing is performed on the speech content text of the multiple video frames to obtain a word embedding representation sequence. A spatiotemporal fusion network is used to enhance the multimodal feature sequence, resulting in an enhanced multimodal feature sequence. Based on the enhanced multimodal feature sequence, the word embedding representation sequence, and video detection task prompts, a multimodal large language model is triggered to determine the video detection result of the video to be detected. Therefore, a video detection process of "extracting multimodal feature sequences from multiple video frames, enhancing multimodal feature sequences based on a spatiotemporal fusion network, and inferring based on a multimodal large language model" is provided, achieving efficient and accurate video authenticity detection and providing reliable technical support for high-security scenarios such as finance and government affairs.

[0062] Figure 2 This diagram illustrates a video detection process as provided in an exemplary embodiment of this specification. See also... Figure 2First, P video frames are sampled from the video to be detected, where P is a positive integer. These P video frames are designated as video frame 1, video frame 2, and video frame P, etc. Multimodal features are extracted from each video frame, and the P video frames are organized into a multimodal feature sequence according to their acquisition time. This multimodal feature sequence is then input into a spatiotemporal fusion network for feature enhancement, resulting in an enhanced multimodal feature sequence. Word embedding processing is performed on the corresponding speech content text of each video frame, yielding word embedding representations for each video frame. The word embedding representations of the P video frames are then organized into a word embedding representation sequence according to their acquisition time. Finally, the image embedding information, word embedding representation sequence, and video detection task prompts corresponding to the enhanced multimodal feature sequence are sequentially organized into a mixed image-text sequence. This mixed image-text sequence is then input into a multimodal large language model for video authenticity detection, yielding the video detection result.

[0063] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 104 can be device A; or the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0064] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0065] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0066] Figure 3 This is a schematic diagram of a video detection device provided as an exemplary embodiment of this specification. See also: Figure 3The device may include: The acquisition module 31 is used to acquire multiple video frames from the video to be detected and extract multimodal features from the multiple video frames; generate a multimodal feature sequence based on the multimodal features of the multiple video frames, and perform word embedding processing on the speech content text of the multiple video frames to obtain a word embedding representation sequence. Feature enhancement module 32 is used to enhance the features of multimodal feature sequences using a spatiotemporal fusion network to obtain enhanced multimodal feature sequences; The video detection module 33 is used to trigger the multimodal large language model to determine the video detection result of the video to be detected based on the enhanced multimodal feature sequence, word embedding representation sequence and video detection task prompt words.

[0067] Optionally, when the acquisition module 31 extracts multimodal features from multiple video frames, it is specifically used for: extracting the frame ID, facial visual features, facial state features, facial action features, and speech-lip-sync information of any one of the multiple video frames; extracting the speech segment of the video frame from the video to be detected and extracting the acoustic features of the speech segment, and performing speech-to-text processing on the speech segment to obtain the speech content text; and unifying the format of the frame ID, facial visual features, facial state features, facial action features, speech content text, acoustic features, and speech-lip-sync information of the video frame to obtain the multimodal features of the video frame.

[0068] Optionally, facial state features include: facial pose information, facial size information, and facial orientation information; facial movement features include mouth movement information and facial expression movement information. Optionally, when the acquisition module 31 extracts the audio-lip-sync information of the video frame, it is specifically used to: acquire the mouth video, including the video frame, and the corresponding audio stream from the video to be detected; input the mouth video and its audio stream into the audio-lip-sync detection model to obtain the audio-lip-sync information of the video frame.

[0069] Optionally, the speech and lip-sync detection model is obtained as follows: positive and negative samples are acquired. Positive samples include lip region image sequences extracted from real videos and their time-synchronized audio segments, while negative samples include lip region image sequences and their time-unsynchronized audio segments. An initial speech and lip-sync detection model containing a visual encoder and an audio encoder is constructed. The visual encoder is used to extract mouth movement features from the lip region image sequences, and the audio encoder is used to extract acoustic representations from the audio segments. Based on the mouth movement features and acoustic representations corresponding to the positive and negative samples, the initial speech and lip-sync detection model is trained through comparative learning to obtain a trained speech and lip-sync detection model.

[0070] Optionally, the spatiotemporal fusion network includes a first spatiotemporal fusion module, a second spatiotemporal fusion module, a third spatiotemporal fusion module, and a feature fusion module; correspondingly, the feature enhancement module 32 is specifically used for: performing a speech and lip-sync detection task on the multimodal feature sequence using the first spatiotemporal fusion module to obtain speech and lip-sync detection results; performing a mouth movement abrupt change detection task on the multimodal feature sequence using the second spatiotemporal fusion module to obtain mouth movement abrupt change detection results; performing a facial expression change naturalness detection task on the multimodal feature sequence using the third spatiotemporal fusion module to obtain facial expression change naturalness detection results; for any multimodal feature in the multimodal feature sequence, the feature fusion module fuses each modality feature in the multimodal feature, the speech and lip-sync detection results, the mouth movement abrupt change detection results, and the facial expression change naturalness detection results to obtain enhanced multimodal features, and obtaining an enhanced multimodal feature sequence based on multiple enhanced multimodal features.

[0071] Optionally, the video detection module 33 is specifically used to: organize the enhanced multimodal feature sequence, word embedding representation sequence, and video detection task prompts into a mixed image-text sequence according to the input requirement information in the video detection task prompts; trigger the multimodal large language model to perform video detection operations based on the mixed image-text sequence according to the output requirement information in the video detection task prompts, and output the video detection result of the video to be detected; wherein, the video detection result includes: video authenticity probability information, natural language explanation information, and information on the degree of speech and lip-sync for the video to be detected; the natural language explanation information is used to explain the reasons for determining the video authenticity probability information of the video to be detected.

[0072] Optionally, the multimodal large language model can be obtained by: acquiring a labeled dataset related to the video authenticity detection task. The labeled dataset includes multiple training samples, each of which includes an enhanced multimodal feature sequence, a word embedding representation sequence, a label, and a fake type label for the sample video. The label indicates whether the sample video is a real video or a fake video. The pre-trained multimodal large language model is fine-tuned using the labeled dataset to obtain the multimodal large language model.

[0073] Optionally, fine-tuning the pre-trained multimodal large language model using an labeled dataset to obtain the multimodal large language model includes: for any training sample, inputting the enhanced multimodal feature sequence, word embedding representation sequence, and video detection task prompts of the sample video into the pre-trained multimodal large language model to obtain the video detection results of the sample video output by the pre-trained multimodal large language model; calculating the authenticity classification loss value based on the video authenticity probability information and corresponding annotation labels in the video detection results of the sample video; extracting keywords representing the forgery type from the natural language interpretation in the video detection results of the sample video; determining the forgery type classification loss value based on the keywords representing the forgery type and their corresponding forgery type labels; adjusting the model parameters of the pre-trained multimodal large language model based on the authenticity classification loss value and the forgery type classification loss value, and repeating the above operations until the model iteration termination condition is met to obtain the multimodal large language model.

[0074] For detailed descriptions of the implementation methods and effects of the above-mentioned device, please refer to the foregoing embodiments, which will not be repeated here.

[0075] Figure 4 This is a schematic diagram of an electronic device provided as an exemplary embodiment of this specification. This electronic device is applicable to the video detection method provided in the foregoing embodiments. Figure 4 As shown, the electronic device 700 mainly consists of a communication interface 702, a user interface 704, a processor 706, and a memory 708. These components are interconnected and communicate with each other through a system bus, network, or other connection mechanism 710. The communication interface 702 enables the device 700 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 702 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 702 can also be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi (Wireless Fidelity), Bluetooth, Global Positioning System (GPS), or wide-area wireless interface such as WiMAX (Wireless Maximum) or LTE (Long Term Evolution). Of course, the communication interface 702 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 702 may also include multiple physical communication interfaces, such as a Wi-Fi interface, a Bluetooth interface, and a wide-area wireless interface.

[0076] User interface 704 includes receiving user input and providing output to the user. Therefore, user interface 704 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT (Cathode Ray Tube), LCD (Liquid Crystal Display), LED (Light Emitting Diode), display using DLP (Digital Light Processing) technology, printer, and other known or future similar devices. User interface 704 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other known or future similar devices. In some embodiments, user interface 704 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, electronic device 700 may support remote access from other devices via communication interface 702 or another physical interface (not shown). User interface 704 can be configured to receive user input, the position and movement of which can be indicated by an indicator or cursor described herein. User interface 704 can also be configured as a display device for rendering or displaying text fragments.

[0077] Processor 706 may include one or more general-purpose processors and / or special-purpose processors. Memory 708 may include one or more volatile and / or non-volatile memory components and may be integrated wholly or partially with processor 706. Memory 708 may include removable and non-removable components.

[0078] The processor 706 is capable of executing program instructions 718 (e.g., compiled or uncompiled program logic and / or machine code) stored in memory 708 to perform the various functions described herein.

[0079] Memory 708 may contain non-transitory computer-readable media, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Memory 708 stores program instructions that, when executed by device 700, enable device 700 to perform any of the methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Processor 706 executing program instructions 718 may cause processor 706 to use data 712.

[0080] For example, program instructions 718 may include an operating system 722 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 700 and one or more applications 720 (e.g., a browser, social application, or game application). Similarly, data 712 may include operating system data 716 and application data 714. Operating system data 716 is primarily accessible to the operating system 722, while application data 714 is primarily accessible to one or more applications 720. Application data 714 may reside in a file system visible or hidden from the user of device 700.

[0081] Application 720 can communicate with operating system 722 through one or more application programming interfaces (APIs). These APIs help application 720 read and / or write application data 714, transmit or receive information via communication interface 702, receive or display information on user interface 704, etc.

[0082] In some terminology, application 720 may be simply referred to as "app". Furthermore, application 720 can be downloaded to device 700 through one or more online app stores or app markets. However, applications can also be installed on device 700 in other ways, such as through a web browser or a physical interface on electronic device 700 (e.g., a USB port).

[0083] Accordingly, embodiments of this specification also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium. Accordingly, embodiments of this specification also provide a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above-described method embodiments. It should be understood that each step or combination of steps in the above-described method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above-described method embodiments.

[0084] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, product, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes that element.

[0085] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0086] The terminology used in the embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” used in the embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one. “A plurality” generally includes at least two, but does not exclude the inclusion of at least one.

[0087] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0088] The above are merely embodiments of this specification and are not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A video detection method, characterized in that, include: Multiple video frames are obtained from the video to be detected, and multimodal features of the multiple video frames are extracted; Based on the multimodal features of the multiple video frames, a multimodal feature sequence is generated, and word embedding processing is performed on the speech content text of the multiple video frames to obtain a word embedding representation sequence. The multimodal feature sequence is enhanced using a spatiotemporal fusion network to obtain an enhanced multimodal feature sequence. Based on the enhanced multimodal feature sequence, the word embedding representation sequence, and the video detection task prompt words, a multimodal large language model is triggered to determine the video detection result of the video to be detected.

2. The method according to claim 1, characterized in that, Extracting multimodal features from the multiple video frames includes: For any one of the multiple video frames, extract the frame ID, facial visual features, facial state features, facial motion features, and speech-lip-sync information of that video frame. The audio segments of the video frames are extracted from the video to be detected, and the acoustic features of the audio segments are extracted. The audio segments are then processed into speech-to-text to obtain the audio content text. The frame ID, facial visual features, facial state features, facial action features, speech content text, acoustic features, and speech-lip synchronization information of the video frame are formatted in a unified manner to obtain the multimodal features of the video frame.

3. The method according to claim 2, characterized in that, The facial state features include facial posture information, facial size information, and facial orientation information, and the facial movement features include mouth movement information and facial expression movement information.

4. The method according to claim 2, characterized in that, Extracting the audio-lip-sync information of the video frames, including: Obtain the mouth video, including the video frame, and the corresponding audio stream from the video to be detected; The mouth video and its audio stream are input into the speech and lip-sync detection model to obtain information on the degree of speech and lip-sync in the video frame.

5. The method according to claim 4, characterized in that, The method for obtaining the speech and lip-sync detection model is as follows: Positive and negative samples are obtained. The positive samples include a sequence of lip region images extracted from a real video and its time-synchronized audio segments. The negative samples include the sequence of lip region images and its time-unsynchronized audio segments. An initial speech and lip-sync detection model is constructed, comprising a visual encoder and an audio encoder. The visual encoder is used to extract mouth movement features from a sequence of lip region images, and the audio encoder is used to extract acoustic representations from audio segments. Based on the mouth movement features and acoustic representations corresponding to the positive and negative samples, the initial speech and lip-sync detection model is trained by comparative learning to obtain a trained speech and lip-sync detection model.

6. The method according to claim 1, characterized in that, The spatiotemporal fusion network includes a first spatiotemporal fusion module, a second spatiotemporal fusion module, a third spatiotemporal fusion module, and a feature fusion module; Accordingly, the multimodal feature sequence is enhanced using a spatiotemporal fusion network to obtain an enhanced multimodal feature sequence, including: The first spatiotemporal fusion module is used to perform a speech and lip-sync detection task on the multimodal feature sequence to obtain the speech and lip-sync detection results; The second spatiotemporal fusion module is used to perform a mouth movement mutation detection task on the multimodal feature sequence to obtain the mouth movement mutation detection results; The third spatiotemporal fusion module is used to perform a facial expression change naturalness detection task on the multimodal feature sequence to obtain the facial expression change naturalness detection results; For any one of the multimodal features in the multimodal feature sequence, the feature fusion module is used to fuse the various modal features in the multimodal feature, the speech and lip-sync detection results, the mouth movement abrupt change detection results, and the facial expression change naturalness detection results to obtain an enhanced multimodal feature, and an enhanced multimodal feature sequence is obtained based on multiple enhanced multimodal features.

7. The method according to claim 1, characterized in that, Based on the enhanced multimodal feature sequence, the word embedding representation sequence, and the video detection task prompt words, a multimodal large language model is triggered to determine the video detection result of the video to be detected, including: According to the input requirements information in the video detection task prompts, the enhanced multimodal feature sequence, the word embedding representation sequence, and the video detection task prompts are organized into a mixed image and text sequence; According to the required information output in the video detection task prompt, the multimodal large language model is triggered to perform video detection operation based on the image and text mixed sequence, and outputs the video detection result of the video to be detected; The video detection results include: video authenticity probability information, natural language interpretation information, and information on the degree of synchronization between speech and lip movements in the video to be detected; the natural language interpretation information is used to explain the reasons for determining the video authenticity probability information of the video to be detected.

8. The method according to claim 7, characterized in that, The method for obtaining the multimodal large language model is as follows: Obtain a labeled dataset related to the video authenticity detection task. The labeled dataset includes multiple training samples. Each training sample includes an enhanced multimodal feature sequence, a word embedding representation sequence, a label, and a fake type label for the sample video. The label indicates whether the sample video is a real video or a fake video. The pre-trained multimodal large language model is fine-tuned using the labeled dataset to obtain the multimodal large language model.

9. The method according to claim 8, characterized in that, The pre-trained multimodal large language model is fine-tuned using the labeled dataset to obtain the multimodal large language model, which includes: For any training sample, the enhanced multimodal feature sequence, word embedding representation sequence, and video detection task prompt words of the sample video are input into a pre-trained multimodal large language model to obtain the video detection result of the sample video output by the pre-trained multimodal large language model; The authenticity classification loss value is calculated based on the video detection results of the sample video and the corresponding annotation labels. Keywords representing forgery types are extracted from the natural language interpretation of the video detection results of the sample videos; the forgery type classification loss value is determined based on the keywords representing forgery types and their corresponding forgery type labels; The model parameters of the pre-trained multimodal large language model are adjusted based on the authenticity classification loss value and the forgery type classification loss value. The above operation is repeated until the model iteration termination condition is met, and the multimodal large language model is obtained.

10. A video detection device, characterized in that, include: The acquisition module is used to acquire multiple video frames from the video to be detected and extract the multimodal features of the multiple video frames; Based on the multimodal features of the multiple video frames, a multimodal feature sequence is generated, and word embedding processing is performed on the speech content text of the multiple video frames to obtain a word embedding representation sequence. The feature enhancement module is used to enhance the features of the multimodal feature sequence using a spatiotemporal fusion network to obtain an enhanced multimodal feature sequence; The video detection module is used to trigger a multimodal large language model to determine the video detection result of the video to be detected based on the enhanced multimodal feature sequence, the word embedding representation sequence, and the video detection task prompt words.

11. An electronic device, characterized in that, include: A memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions for: performing the steps of the method according to any one of claims 1-9.

12. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it is able to perform the steps of the method according to any one of claims 1-9.

13. A computer program product, characterized in that, include: A computer program / instruction that, when executed by a processor, enables the implementation of the steps in the method described in any one of claims 1-9.

Images