Digital human video detection method based on facial semantic features

Abstract

The application discloses a digital person video detection method based on facial semantic features, acquires real person voice visual video to constitute a training data set; constructs a digital person video detection model based on facial reality mode representation, respectively extracts facial reality mode representation features, mouth features and voice features, after multi-modal feature fusion, generates a matching score matrix according to the fused features; trains the digital person video detection model by using the training data set, generates a matching score matrix by using the trained digital person video detection model, and aggregates the matching score matrix to obtain the overall matching score of the to-be-detected video, so that the forgery detection is realized. The application proposes a unified audio and video forgery detection framework based on the facial reality mode, and improves the deep forgery detection performance in different scenes.

Description

Technical Field

[0001] This invention belongs to the field of digital human deepfake detection technology, and more specifically, relates to a digital human video detection method based on facial semantic features. Background Technology

[0002] Regarding data construction and forgery types, representative audio and video forgery datasets typically include GAN-based generation results or partial face editing forgeries. Recent datasets further employ high-quality lip-sync speaking digit generation, posing a more complex cross-modal consistency challenge to existing detection frameworks. Furthermore, a more challenging dataset introduces diffusion model generation to enhance realism and diversity, thereby raising the evaluation threshold. Overall, existing deep forgery detection datasets for audio and video cover a variety of forgery modes and generation models, providing fundamental support for detector development.

[0003] The audio-visual digital human generation task aims to map acoustic signals to temporally coherent facial movements while modeling cross-modal dependencies between audio and visual streams. As digital human generation technology matures, the realism of forgeries has significantly improved, leading to increased detection difficulty and driving a shift in detection paradigms from single-modal feature modeling to joint modeling and collaborative learning of audio and visual data. Existing detection methods primarily rely on supervised learning: one approach directly utilizes labeled real and fake videos for end-to-end training; another involves self-supervised pre-training followed by fine-tuning with labeled data to achieve discriminative power. However, these methods are highly dependent on labeled samples, often exhibiting limited generalization and robustness in cross-dataset scenarios. To alleviate this issue, recent research has begun exploring unsupervised detection approaches using only real data. While these methods can learn certain forgery patterns, their core focus remains on audio-visual alignment consistency.

[0004] Furthermore, most existing methods for detecting deepfake digital humans are still limited to speaking scenarios, failing to adequately cover singing scenarios, which also possess strong audio-visual expressiveness and wider dissemination. This is because existing methods primarily determine authenticity by evaluating the consistency between lip movements and speech; however, in singing scenarios, factors such as accompaniment, rhythm, and melody weaken the alignment between lip movements and speech. Simultaneously, exaggerated facial expressions and more pronounced head movements bring greater temporal and deformation variations, making it difficult for existing detectors to effectively model and adapt them. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a digital human video detection method based on facial semantic features. It proposes a unified audio and video forgery detection framework based on facial authenticity patterns, and jointly models facial semantic information and audio and video consistency in the same model to improve the deep forgery detection performance in different scenarios.

[0006] To achieve the above-mentioned objectives, the digital human video detection method based on facial semantic features of the present invention includes the following steps:

[0007] S1: Acquire several real human voice and visual videos to form a training dataset according to actual needs;

[0008] S2: Construct a digital human video detection model, including a facial realism pattern representation module, a mouth feature extraction module, a speech feature extraction module, a multimodal feature fusion module, and a detection module, wherein:

[0009] The facial realism pattern representation module is used to extract facial realism pattern representation features from the input video. And send it to the multimodal feature fusion module, The preset feature size is indicated; the facial realism pattern representation module includes a text feature extraction module, a facial feature extraction module, and a feature fusion module, wherein:

[0010] The text feature extraction module is used to extract text features from pre-set... Extracting positive text features from positive and negative text pairs and negative text features , Indicate the preset feature size, then the positive sample features Send to the feature fusion module; the method for setting positive and negative text pairs is as follows: set according to actual needs. A granularity for describing the face is defined, and then a real description text and a fake description text are generated for each granularity, which are used as positive text and negative text respectively, thus obtaining the positive and negative text pairs corresponding to that granularity;

[0011] The facial feature extraction module is used to extract facial features of people from the input video. And send it to the feature fusion module;

[0012] The feature fusion module is used to process the features of positive samples. and facial features Perform feature fusion and map to a preset size The facial realism pattern representation features are obtained. ;

[0013] The mouth feature extraction module is used to extract the mouth features of people from the input video. And send it to the multimodal feature fusion module;

[0014] The speech feature extraction module extracts speech features from the input video. And send it to the multimodal feature fusion module;

[0015] The multimodal feature fusion module is used to represent facial realism patterns. Mouth features and speech features By performing fusion, multimodal fusion features are obtained. And send it to the detection module;

[0016] The detection module is used to detect multimodal fusion features. Obtain the matching score matrix , of which elements Represents speech frames With video frames Match score, , Indicates the number of frames in the input video;

[0017] S3: Train the digital human video detection model using the training dataset from step S1 to obtain the trained digital human video detection model; the loss function for each training sample during training. The following method is used for calculation:

[0018] ,

[0019] in, Indicates the preset weight. The audio / video alignment loss function is expressed by the following formula:

[0020] ,

[0021] in, Indicates the first The temporal neighborhood of a frame;

[0022] The binary contrastive loss function, representing facial realism pattern features, is calculated as follows:

[0023] ,

[0024] in, This indicates the calculation of feature similarity;

[0025] S4: When performing deepfake detection on the voice and visual video of the person to be detected, sample the voice and visual video of the person to be detected into frames with a frequency of [missing information]. The video is then input into the digital human video detection model trained in step S3 to obtain a matching score matrix. The overall matching score of the video to be detected is then obtained by aggregating the matching score matrix. When the overall matching score If the value exceeds a preset threshold, the video to be detected is a real video; otherwise, it is a deepfake video.

[0026] This invention presents a digital human video detection method based on facial semantic features. It obtains a training dataset consisting of authentic human speech and visual videos; constructs a digital human video detection model based on facial realism pattern representation; extracts facial realism pattern representation features, mouth features, and speech features respectively; fuses these multimodal features; and generates a matching score matrix based on the fused features. The digital human video detection model is trained using the training dataset. The trained model generates a matching score matrix for the video to be detected, and the matching score matrices are then aggregated to obtain the overall matching score of the video, thereby achieving forgery detection.

[0027] The present invention has the following beneficial effects:

[0028] 1) Whether it is singing or speaking, real samples present richer facial semantics than synthetic samples. This invention incorporates highly distinguishable facial semantics into the detection, which can effectively improve the robustness of forgery detection.

[0029] 2) This invention proposes a unified audio and video forgery detection framework, which jointly models facial semantic information and audio and video consistency in the same model, so that the constructed digital human video detection model has high depth forgery detection performance in different scenarios. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating a specific implementation of the digital human video detection method based on facial semantic features of the present invention.

[0031] Figure 2 This is a structural diagram of the digital human video detection model in this invention;

[0032] Figure 3 This is a flowchart of the construction process of the singing digital human deepfake dataset in this embodiment. Detailed Implementation

[0033] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0034] Example

[0035] Figure 1 This is a flowchart illustrating a specific implementation of the digital human video detection method based on facial semantic features according to the present invention. Figure 1 As shown, the specific steps of the digital human video detection method based on facial semantic features of the present invention include:

[0036] S101: Obtain the training dataset:

[0037] Based on actual needs, a training dataset is constructed by acquiring several real-life human voice and visual videos.

[0038] S102: Constructing a digital human video detection model:

[0039] To adapt to different human activities, this invention learns stable facial realism patterns from real faces, so that when a forged sample disrupts this pattern, a detectable deviation signal is generated, thereby improving the accuracy of deepfake detection of digital humans. Based on this, this invention designs a digital human video detection model based on facial realism pattern learning. Figure 2 This is a structural diagram of the digital human video detection model in this invention. (See diagram below.) Figure 2 As shown, the digital human video detection model in this invention includes a facial realism pattern representation module, a mouth feature extraction module, a speech feature extraction module, a multimodal feature fusion module, and a detection module. Each module will be described in detail below.

[0040] The facial realism pattern representation module is used to extract facial realism pattern representation features from the input video. And send it to the multimodal feature fusion module, This indicates the preset feature size. For example... Figure 2 As shown, the facial realism pattern representation module in this invention includes a text feature extraction module, a facial feature extraction module, and a feature fusion module, wherein:

[0041] The text feature extraction module is used to extract text features from pre-set... Extracting positive text features from positive and negative text pairs. and negative text features , Indicate the preset feature size, then the positive sample features Send to the feature fusion module. The method for setting positive and negative text pairs is as follows: set according to actual needs. A granularity for describing the face is defined, and then a real descriptive text and a fake descriptive text are generated for each granularity, serving as positive text and negative text respectively, thus obtaining the positive and negative text pairs corresponding to that granularity.

[0042] This invention employs positive and negative text sets to provide contrast constraints in the face-text space. In this embodiment, three granularities are set: the entire face, the eyes, and the mouth. To avoid subjective factors from human intervention, this embodiment uses a large language model (chatgpt in this embodiment) to generate positive and negative text pairs. The resulting positive samples include "a humanface", "a bonafide face with expressive eyes", and "a genuine face with a natural mouth", while the corresponding negative texts are "a fake human face", "a spoof face with dull eyes", and "a forged face with an unnatural mouth".

[0043] In this embodiment, the text feature extraction module uses a text concatenation module and a text encoder ( ), across granularity average modules and linear layers, where:

[0044] The text concatenation module is used to... Each text in a positive-negative text pair is associated with a learnable text. To splice together, A concatenated text input text encoder. This embodiment introduces learnable text, which allows text features to contain learnable, realistic semantic expressions.

[0045] The text encoder is used to encode each concatenated text separately to obtain concatenated text features and send them to the cross-granularity averaging module.

[0046] The cross-granularity averaging module is used to perform cross-granularity averaging on the concatenated text features obtained from both positive and negative text to obtain the positive text features. and negative text features And it is sent to the linear layer. The formula for cross-granularity averaging is as follows:

[0047] ,

[0048] ,

[0049] in, Indicates the first The spliced ​​text features obtained from the main text at each granularity Indicates the first The spliced ​​text features obtained from negative text at each granularity , This indicates the goal of finding the norm.

[0050] Linear layers are used to separate the features of the main text. and negative text features Linear mapping to preset size To obtain the features of the main text and negative text features .

[0051] The facial feature extraction module is used to extract facial features of people from the input video. And send it to the multimodal feature fusion module. In this embodiment, the facial feature extraction module includes a facial mask extraction module, a facial encoder, a temporal averaging module, and a linear layer, wherein:

[0052] The face masking module is used to extract each video frame from the input video. Extracting facial masks , , This indicates the number of frames in the input video, then the facial mask for each frame is... With each video frame The mixture is fused to obtain a hybrid sequence. And input the facial encoder.

[0053] Face encoders are used to extract from mixed sequences Extracting facial features The facial features are then sent to the temporal averaging module. In this embodiment, the facial encoder uses the Alpha-CLIP model, which includes an RGB image encoder (…). ) and mask encoder ( Under the combined effect of the attention module () To achieve regional-level semantic modeling.

[0054] The temporal averaging module is used to average facial features over time. Average the facial features And send it to the linear layer.

[0055] Linear layers are used to separate facial features Linear mapping to preset size To obtain facial features .

[0056] The feature fusion module is used to process the features of positive samples. and facial features Perform feature fusion and map to a preset size The facial realism pattern representation features are obtained. In this embodiment, positive sample features are... and facial features After splicing, linear mapping can be performed to achieve fusion.

[0057] The mouth feature extraction module is used to extract the mouth features of people from the input video. And send it to the multimodal feature fusion module. In this embodiment, the mouth feature extraction module includes a mouth region extraction module, a stitching module, a visual encoder, and a linear layer, wherein:

[0058] The mouth region extraction module is used to extract the mouth region from each video frame of the input video. Identify the exit area and crop the opening area image to a preset size. And send it to the splicing module.

[0059] The stitching module is used to stitch together images of the mouth region. The images are stitched together along the time axis to form a sequence of mouth images and then sent to the visual encoder.

[0060] A visual encoder is used to extract visual features from a sequence of mouth images and send them to a linear layer.

[0061] Linear layers are used to linearly map visual features to a preset size. To obtain oral features .

[0062] The speech feature extraction module extracts speech features from the input video. And send it to the multimodal feature fusion module. In this embodiment, the speech feature extraction module includes a Mel spectrum calculation module and a speech encoder ( ) and linear layers, where:

[0063] The Mel spectrum calculation module is used to calculate the audio in the input video. Calculate the Mel spectrum to obtain the Mel feature sequence aligned with the video time. Then it is sent to the voice encoder.

[0064] Speech encoders are used to process Mel feature sequences Speech features are obtained by speech encoding and then sent to the linear layer.

[0065] Linear layers are used to linearly map speech features to a preset size. To obtain oral features .

[0066] The multimodal feature fusion module is used to represent facial realism patterns. Mouth features and speech features By performing fusion, multimodal fusion features are obtained. And send it to the detection module.

[0067] To address the issue of varying reliability across different forgery types, this embodiment employs an adaptive weight generation mechanism. The multimodal feature fusion module includes an initial weight generation module, a modality modulator, and a weighted fusion module, wherein:

[0068] The initial weight generation module is used to represent features based on facial realism patterns. Mouth features and speech features Generate the corresponding initial weights , , This forms the initial weight vector. And send it to the mode modulator.

[0069] In this embodiment, the initial weight generation module includes a feature concatenation module, a multilayer perceptron, and a softmax layer, wherein:

[0070] The feature concatenation module is used to represent facial realism patterns. Mouth features and speech features The data is then stitched together, and the stitched features are sent to the multilayer perceptron.

[0071] The multilayer perceptron is used to perform linear mapping on the concatenated features, mapping them into 3D feature vectors and sending them to the softmax layer.

[0072] The softmax layer is used to process the received 3D feature vector using the softmax function to obtain the initial 3D weight vector. .

[0073] The mode modulator is used to employ a preset 3D modulation vector. For the initial weight vector Apply a bias to obtain the final fusion weights. And send it to the weighted fusion module, where This represents the softmax function. In this embodiment, the 3D modulation vector... By introducing modulation vectors, audio-visual consistency cues can be emphasized while preserving the contribution of facial realism patterns.

[0074] The weighted fusion module is used to employ weight vectors The weights in the facial realism pattern representation feature Mouth features and speech features Weighted fusion is performed to obtain multimodal fusion features. The specific formula for weighted fusion is as follows:

[0075] .

[0076] The detection module is used to detect multimodal fusion features. Obtain the matching score matrix , of which elements Represents speech frames With video frames Match score, , This indicates the number of frames in the input video.

[0077] S103: Training the digital human video detection model:

[0078] The digital human video detection model is trained using the training dataset from step S101 to obtain the trained digital human video detection model.

[0079] In this invention, since the training dataset uses real human speech and visual videos, a binary contrast loss function for faces and text is introduced into the loss function to align facial features with positive text features and distance them from negative text features. Simultaneously, audio-video alignment loss is considered, thus obtaining the loss function for each training sample during the training of the digital human video detection model in this invention. The calculation formula is as follows:

[0080] ,

[0081] in, Indicates the preset weight. This represents the audio-video alignment loss function. To enhance the temporal alignment between audio and visual frames, this invention employs a constraint method that maximizes their corresponding probabilities. The audio-video loss is defined as the negative average of the alignment probabilities over the entire video segment. Therefore, the audio-video alignment loss function... The calculation formula is as follows:

[0082] ,

[0083] in, Indicates the first The temporal neighborhood of a frame.

[0084] The binary contrastive loss function, representing facial realism pattern features, is calculated as follows:

[0085] ,

[0086] in, This indicates the calculation of feature similarity. The formula for calculating feature similarity in this embodiment is as follows:

[0087] ,

[0088] ,

[0089] Wherein, the superscript T indicates transpose. This represents the temperature coefficient.

[0090] The termination condition for training the digital human video detection model can be set according to actual needs. In this embodiment, a validation set is used to validate the current digital human video detection model. The validation set includes real videos and deepfake videos. Then, the deepfake detection accuracy is calculated. When the deepfake detection accuracy reaches a preset threshold, the training ends.

[0091] To address the issue that existing publicly available audio and video deepfake datasets are mainly based on "speaking speech" synthesis and lack dedicated samples for detecting singing digital human forgeries, the deepfake videos in the validation set in this embodiment include the Singing Digital Human Deepfake Dataset (SHDF). Figure 3 This is a flowchart illustrating the construction process of the singing digital human deepfake dataset in this embodiment. For example... Figure 3 As shown, the method for constructing the singing digital human deepfake dataset in this embodiment is as follows:

[0092] S301: Obtain real-life singing videos:

[0093] Obtain according to actual needs Several real singing videos of a particular person.

[0094] S302: Expanding Character Identity:

[0095] Select from existing face generation datasets Each person's identity is used as an extended person. In this embodiment, extended people are selected from the HDTF (High-Quality Deepfake Talking Face) dataset.

[0096] S303: Training a singing model:

[0097] Set according to actual needs Each digital human deepfake model is trained based on the real singing video and the extended facial features of the person in step S302, and a separate singing model is trained for each person. , , .

[0098] In this embodiment, considering both the feasibility of generation and the responsiveness to song-driven head posture and facial expressions, MEMO, Hallo2, EchoMimic, and DreamTalk were selected as candidate models. Then, the generation results of the candidate models were subjectively evaluated by users in terms of visual quality, head movement, and lip-sync accuracy. Based on this, MEMO, Hallo2, and EchoMimic were selected as the generation models for the singing digital human deepfake video, while DreamTalk, which had issues with stiff head posture, was eliminated.

[0099] S304: Obtain driver audio:

[0100] Several real songs are collected, and then several fake songs are synthesized using a preset song synthesis model to form the driving audio set. In this embodiment, the song synthesis model uses the Sonics model.

[0101] S305: Generating Deepfake Videos of Singing Digital Humans

[0102] Each audio track from the audio set is used as the driving signal and input to the singing model of each character. Generate deepfake videos of singing digital humans to form a deepfake dataset of singing digital humans.

[0103] S104: Deepfake Detection

[0104] When performing deepfake detection on the voice and visual video of the person to be detected, the voice and visual video of the person to be detected is sampled into frames with a frequency of [missing information]. The video is then input into the digital human video detection model trained in step S103 to obtain a matching score matrix. The overall matching score of the video to be detected is then obtained by aggregating the matching score matrix. When the overall matching score If the score exceeds a preset threshold, the video to be detected is a real video; otherwise, it is a deepfake video. In this embodiment, the matching score matrix aggregation is implemented using the Softmax function.

[0105] To better illustrate the technical effects of this invention, specific examples are used to conduct experimental verification. This experimental verification covers two typical audio-visual scenarios: speaking and singing. The singing scenario was tested on the SHDF dataset proposed in this invention. In the speaking scenario, three datasets—AVLips, FakeAVCeleb, and TalkingHeadBench—were used for evaluation to cover various types of speaker head generation models. The evaluation on SHDF used a test subset selected from the complete dataset, containing 800 real samples and 1,500 synthetic samples, exhibiting diversity in terms of gender, synthesis method, and speaker identity. In the evaluation of the speaking digital human dataset, AVLips was divided into training, validation, and test sets in a 6:1:3 ratio. For FakeAVCeleb (FKAV), this embodiment constructed a test set containing 500 real samples and 1,000 carefully selected fake samples. The test set for TalkingHeadBench (THB) was obtained by summarizing all test videos generated by the diffusion model and officially released by TalkingHeadBench. Since THB lacks real samples, this embodiment supplements real samples from the test partitions of AVLips to form a complete test set.

[0106] To ensure fair performance comparisons across speaking and singing scenarios, the model in this invention is trained using only real samples from the speaking dataset, thus maintaining consistency in the training data distribution. Furthermore, to examine whether detection performance depends on scene-specific data distributions, AVH-Align and the proposed T-AVFD are retrained on the SHDF training set and evaluated simultaneously on both speaking and singing data.

[0107] In this embodiment, six existing methods are used as comparative methods to verify the present invention. The evaluation metrics used include the area under the curve (AUC) and the average precision (AP). The existing methods are:

[0108] CViT, see the paper "Wodajo D, Atnafu S. Deepfake video detection using convolutional vision transformer. arXiv preprint arXiv:2102.11126, 2021";

[0109] EfficientViT, see the literature "Coccomini D A, Messina N, Gennaro C, et al. Combining efficientnet and vision transformers for video deepfake detection. In: Proceedings of International conference on image analysis and processing. Springer, 2022. 219–229";

[0110] RealForensics, see the literature "Haliassos, A., Mira, R., Petridis, S., and Pantic, M. Leveraging real talking faces via self-supervision for robust forgery detection. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pp. 14950–14962, 2022.";

[0111] LipFD, see the literature "Liu W, She T, Liu J, et al. Lips are lying: Spotting the temporal inconsistency between audio and visual in lip-syncing deepfakes. Advances in Neural Information Processing Systems, 2024, 37: 91131–91155";

[0112] AVAD, see the document "Feng C, Chen Z, Owens A. Self-supervised videoforensics by audio-visual anomaly detection. In: Proceedings of proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2023.10491–10503";

[0113] AVH-Align, see the document "Smeu S, Boldisor DA, Oneata D, et al. Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learning. In: Proceedings of the Computer Vision and Pattern Recognition Conference, 2025. 18815–18825 ".

[0114] Table 1 compares the detection performance of the present invention and the comparative method in different scenarios in this embodiment. Here, sup. indicates supervised training, unsup. indicates unsupervised training, V represents the visual modality, and AV represents the speech-visual modality.

[0115]

[0116] Table 1

[0117] As shown in Table 1, the present invention can achieve optimal performance in different speech and visual detection scenarios.

[0118] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A digital human video detection method based on facial semantic features, characterized in that, Includes the following steps: S1: Obtain several real human voice and visual videos to form a training dataset according to actual needs; S2: Construct a digital human video detection model, including a facial realism pattern representation module, a mouth feature extraction module, a speech feature extraction module, a multimodal feature fusion module, and a detection module, wherein: The facial realism pattern representation module is used to extract facial realism pattern representation features from the input video. And send it to the multimodal feature fusion module, The preset feature size is indicated; the facial realism pattern representation module includes a text feature extraction module, a facial feature extraction module, and a feature fusion module, wherein: The text feature extraction module is used to extract text features from pre-set... Extracting positive text features from positive and negative text pairs and negative text features , Indicate the preset feature size, then the positive sample features Send to the feature fusion module; the method for setting positive and negative text pairs is as follows: set according to actual needs. A granularity for describing the face is defined, and then a real description text and a fake description text are generated for each granularity, which are used as positive text and negative text respectively, thus obtaining the positive and negative text pairs corresponding to that granularity; The facial feature extraction module is used to extract facial features of people from the input video. And send it to the feature fusion module; The feature fusion module is used to process the features of positive samples. and facial features Perform feature fusion and map to a preset size The facial realism pattern representation features are obtained. ; The mouth feature extraction module is used to extract the mouth features of people from the input video. And send it to the multimodal feature fusion module; The speech feature extraction module extracts speech features from the input video. And send it to the multimodal feature fusion module; The multimodal feature fusion module is used to represent facial realism patterns. Mouth features and speech features By performing fusion, multimodal fusion features are obtained. And send it to the detection module; The detection module is used to detect multimodal fusion features. Obtain the matching score matrix , of which elements Represents speech frames With video frames Match score, , Indicates the number of frames in the input video; S3: Train the digital human video detection model using the training dataset from step S1 to obtain the trained digital human video detection model; the loss function for each training sample during training. The following method is used for calculation: ， in, Indicates the preset weight. The audio / video alignment loss function is expressed by the following formula: ， in, Indicates the first The temporal neighborhood of a frame; The binary contrastive loss function, representing facial realism pattern features, is calculated as follows: ， in, This indicates the calculation of feature similarity; S4: When performing deepfake detection on the voice and visual video of the person to be detected, sample the voice and visual video of the person to be detected into frames with a frequency of [missing information]. The video is then input into the digital human video detection model trained in step S3 to obtain a matching score matrix. The overall matching score of the video to be detected is then obtained by aggregating the matching score matrix. When the overall matching score If the value exceeds a preset threshold, the video to be detected is a real video; otherwise, it is a deepfake video.

2. The digital human video detection method according to claim 1, characterized in that, The text feature extraction module includes a text concatenation module, a text encoder, a cross-granularity averaging module, and a linear layer, wherein: The text concatenation module is used to concatenate text into a single file. Each text in a positive-negative text pair is associated with a learnable text. To splice together, A concatenated text input text encoder; The text encoder is used to encode each concatenated text separately to obtain concatenated text features and send them to the cross-granularity averaging module; The cross-granularity averaging module is used to perform cross-granularity averaging on the concatenated text features obtained from both positive and negative text to obtain the positive text features. and negative text features And it is sent to the linear layer. The formula for cross-granularity averaging is as follows: ， ， in, Indicates the first The spliced ​​text features obtained from the main text at each granularity Indicates the first The spliced ​​text features obtained from negative text at each granularity , This indicates the goal of finding the norm. Linear layers are used to separate the features of the main text. and negative text features Linear mapping to preset size To obtain the features of the main text and negative text features .

3. The digital human video detection method according to claim 1, characterized in that, The facial feature extraction module includes a facial masking module, a facial encoder, a temporal averaging module, and a linear layer, wherein: The face masking module is used to extract each video frame from the input video. Extracting facial masks , , This indicates the number of frames in the input video, then the facial mask for each frame is... With each video frame The mixture is fused to obtain a hybrid sequence. And input the facial encoder; A facial encoder is used to extract from a mixed sequence Extracting facial features Then the facial features are sent to the time-series averaging module; The temporal averaging module is used to average facial features over time. Average the facial features And send it to the linear layer; Linear layers are used to separate facial features Linear mapping to preset size To obtain facial features .

4. The digital human video detection method according to claim 1, characterized in that, The mouth feature extraction module includes a mouth region extraction module, a stitching module, a visual encoder, and a linear layer, wherein: The mouth region extraction module is used to extract the mouth region from each video frame of the input video. Identify the exit area and crop the opening area image to a preset size. And send it to the splicing module; The stitching module is used to stitch together images of the mouth region. The images are stitched together along the time axis to form a sequence of mouth images and then sent to the visual encoder. A visual encoder is used to extract visual features from a sequence of mouth images and send them to a linear layer; Linear layers are used to linearly map visual features to a preset size. To obtain oral features .

5. The digital human video detection method according to claim 1, characterized in that, The speech feature extraction module includes a Mel-spectrum calculation module, a speech encoder, and a linear layer, wherein: The Mel spectrum calculation module is used to calculate the audio in the input video. Calculate the Mel spectrum to obtain the Mel feature sequence aligned with the video time. Then it is sent to the voice encoder; Speech encoders are used for Mel feature sequences Speech encoding is performed to obtain speech features, which are then sent to the linear layer. Linear layers are used to linearly map speech features to a preset size. To obtain oral features .

6. The digital human video detection method according to claim 1, characterized in that, The multimodal feature fusion module includes an initial weight generation module, a modality modulator, and a weighted fusion module, wherein: The initial weight generation module is used to represent features based on facial realism patterns. Mouth features and speech features Generate the corresponding initial weights , , This forms the initial weight vector. And send it to the mode modulator; The mode modulator is used to employ a preset 3D modulation vector. For the initial weight vector Apply a bias to obtain the final fusion weights. And send it to the weighted fusion module, where This represents the softmax function; The weighted fusion module is used to employ weight vectors The weights in the facial realism pattern representation feature Mouth features and speech features Weighted fusion is performed to obtain multimodal fusion features. The specific formula for weighted fusion is as follows: 。 7. The digital human video detection method according to claim 6, characterized in that, The initial weight generation module includes a feature concatenation module, a multilayer perceptron, and a softmax layer, wherein: The feature concatenation module is used to represent facial realism patterns. Mouth features and speech features The data is then stitched together, and the stitched features are sent to the multilayer perceptron. The multilayer perceptron is used to linearly map the spliced ​​features into a 3D feature vector and send it to the softmax layer; The softmax layer is used to process the received 3D feature vector using the softmax function to obtain the initial 3D weight vector. .

8. The digital human video detection method according to claim 1, characterized in that, The formula for calculating feature similarity in step S3 is as follows: ， ， Wherein, the superscript T indicates transpose. This represents the temperature coefficient.

9. The digital human video detection method according to claim 1, characterized in that, The termination condition for training the digital human video detection model is as follows: the current digital human video detection model is validated using a validation set, which includes real videos and deepfake videos. Then, the forgery detection accuracy is calculated. When the forgery detection accuracy reaches a preset threshold, the training ends.

10. The digital human video detection method according to claim 9, characterized in that, The deepfake videos in the validation set include a singing digital human deepfake dataset, which is generated as follows: 1) Obtain according to actual needs Several real singing videos of an individual; 2) Select from existing face generation datasets Each character's identity serves as an extended character; 3) Set according to actual needs Each digital human deepfake model is trained based on the real singing video and the extended facial features of the person in step 2), and a separate singing model is trained for each person. , , ; 4) Collect several real songs, and then use a preset song synthesis model to synthesize several fake songs to form a driving audio set; 5) Use each audio signal from the audio set as the driving signal to input into each character's singing model. Generate deepfake videos of singing digital humans, thus forming a deepfake dataset of singing digital humans.

Images