Multi-modal global and local collaboration-based speaking face generation video detection method and device

Abstract

The invention discloses a speaking face generation video detection method and device based on multi-modal global and local collaboration, and the method comprises the steps: carrying out the framing and face region extraction preprocessing of a video, and enabling an input audio to be aligned with an input video frame in a time dimension; a speaking face generation video detection model is constructed and trained, an inconsistent region between video frames is focused on through a preset region attention smooth detection module and a difference capture-time frame aggregation module, and an original audio is input into an audio feature extraction module; carrying out consistency calculation on the input audio and video modal features through a preset audio and video fusion module, fusing the audio and video features so as to discriminate the forged generated video to obtain a video discrimination tag, and then calculating a loss training model; and finally, discriminating a to-be-detected video by using the trained model. According to the method, audio and video features are combined, so that the detection precision and robustness can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimedia digital forensics, and in particular relates to a method and device for detecting speaking face generation videos based on multimodal global and local synergy. Background Art

[0002] In recent years, generative AI technology has made significant progress. Notably, digital human generation is a real-world application of AI-generated technology, widely used in areas such as commercial live streaming, and has promoted social and economic progress. Speaking face generation, a key technology for digital human generation, is fundamentally different from traditional deepfake methods. Traditional deepfakes primarily replace facial images in videos to obfuscate a person's identity, but their operation is often limited by the need for pre-existing video footage similar to the target person as a reference. However, using talking face generation technology, we only need an image and text, or a video and audio of the person speaking, to generate a realistic speech video of the corresponding person, offering greater flexibility and naturalness. Generated talking face videos exhibit superior visual quality and more realistic details, such as lighting uniformity and lip movement. Furthermore, this technology transitions the forgery process from relying on strong references to requiring only weak references, reducing the difficulty of forgery. Consequently, using this technology, it is possible to effortlessly forge videos of prominent politicians making inappropriate remarks, causing public panic and a loss of confidence. For the detection of videos generated by talking faces, existing deep fake detection methods have certain transferability, but still have their limitations. There is currently no detection method for videos generated by talking faces.

[0003] Currently, the detection of deep fake videos mainly focuses on three aspects: spatial visual artifacts, frequency domain forgery traces, and multimodal inconsistencies in audio and video.Spatial visual artifacts are mainly detected by mining the tampering traces left by forged videos in the spatial domain from the original video frames using hand-designed features or deep neural networks, such as local shadow anomalies (B.Peng, W.Wang, J.Dong and T.Tan, "Optimized 3D Lighting Environment Estimation for Image Forgery Detection," in IEEE Transactions on Information Forensics and Security, vol.12, no.2, pp.479-494) and edge fusion artifacts (Y.Nirkin, L.Wolf, Y.Keller and T.Hassner, "DeepFake Detection Based on Discrepancies Between Faces and TheirContext," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.44, no.10, pp.6111-6121). Frequency domain forgery traces are detected by mining the influence of the video on the frequency domain information during the forgery process, such as the spectral checkerboard effect caused by upsampling operations. learning:rethinking face forgery detection infrequency domain[C] / / Proceedings of the IEEE / CVF conference on computervision and pattern recognition.2021:772-781." The spectral information obtained after discrete cosine transform of the video frame is used to detect face forgery videos; the audio and video multimodal detection method mainly extracts the features of audio and video, calculates the inconsistency and asynchrony of the two types of features to distinguish generated videos from real videos. For example, the literature (Zhou Y, Lim SN.Joint audio-visual deepfake detection[C] / / Proceedings of the IEEE / CVFinternational conference on computer vision.2021:14800-14809.) uses the attention mechanism to fuse the audio and video outputs of different intermediate layers of the network, and then align the multimodal features to improve the robustness of detection.

[0004] While existing deepfake detection methods can achieve some success in detecting videos generated by speaking faces, their limited specificity means they perform poorly on high-quality generated speaking faces. Developing efficient and feasible detection methods that address the generation flaws and technical characteristics of generated speaking faces is a pressing challenge for those skilled in the art. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a method and device for detecting speaking face generation videos based on multimodal global and local synergy, which improves the detection capability of low-quality videos by targeting the detection method of multimodal global and local synergy and reduces the computational pressure during detection by inputting a smaller number of frames.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for detecting a speaking face generated video based on multimodal global and local synergy, comprising the following steps:

[0008] Perform pre-processing on the video by framing and extracting the face area, and slice the audio file in the video so that the input audio is aligned with the input video frame in the time dimension;

[0009] A talking face generation video detection model is constructed and trained. Specifically, the original video frame is first input into the regional attention smoothing detection module to extract the inter-frame difference information of the video frame transition, and then input into the difference capture-time aggregation module to capture the difference information between video frames. At the same time, the audio segment corresponding to the original video is input into the audio feature extraction module to obtain audio features, and then the video features and audio features are input into the audio-video interaction module together to obtain audio-video fusion features; the original video is also input into the frequency domain feature extraction module, and the obtained frequency domain features are spliced ​​with the fusion features. The features are then input into the depthwise separable convolution module for feature extraction, and the generated video is discriminated to obtain the video discrimination label; the cross entropy between the video real label and the video discrimination label is used as the loss function of the entire network model, and the model is trained through loss back propagation to obtain a trained talking face generation video detection model;

[0010] The trained speaking face generation video detection model is used to identify the video to be detected and obtain the identification results of the video frames.

[0011] As an optimal technical solution, the region attention smoothing detection module is constructed based on inter-frame difference calculation. By calculating the difference between two adjacent video frames, the inter-frame difference representation of the video frame is obtained, and the inter-frame difference is staggered and added so that each inter-frame difference representation contains the difference between the corresponding frame and the previous frame and the next frame; the inter-frame difference is assigned a channel weight through the convolution layer, and then each input frame is dot-multiplied with the corresponding inter-frame difference representation, and then added to the original input frame, and finally input into a two-layer depth-separable convolution module for deep feature extraction.

[0012] As a preferred technical solution, the difference capture-temporal aggregation module is constructed based on the self-attention mechanism and the mask mechanism, and specifically includes the following steps:

[0013] At each image pixel, the self-attention mechanism is used to calculate the channel similarity matrix between all input video frames, and then the variance of the T dimension is calculated to obtain the variance matrix. The kth minimum variance is then selected as the threshold, and the variance matrix is ​​masked. The pixel area greater than the threshold is set to 1, and vice versa. Finally, the input feature is multiplied pixel by pixel with the variance matrix and added back to the input feature. Finally, it is input into a depthwise separable convolution module to obtain the video feature. In order to achieve the temporal granularity alignment of audio and video features, the difference capture-temporal aggregation module also performs pooling operations of different window sizes on the video features in the horizontal and vertical directions, and then performs convolution and upsampling in the temporal order. Finally, these multiple outputs are added together to aggregate the spatiotemporal information of adjacent frames from multiple granularities.

[0014] As a preferred technical solution, the audio feature extraction module is constructed based on the pre-trained wav2vec model, specifically:

[0015] The original audio sequence is input into the pre-trained wav2vec model, and the input original audio sequence is encoded to obtain an audio feature vector; the audio feature vector is then input into the residual convolutional network, and finally the obtained audio features are dimensionally adjusted to align with the video feature vector.

[0016] As a preferred technical solution, the audio and video interaction module is constructed based on the cross attention model, specifically:

[0017] The video feature vector is input into the 3D convolution layer to obtain the Query, the audio feature vector is input into the 3D convolution layer to obtain the Key, the audio and video feature vectors are linearly summed and input into the convolution layer to obtain the Value, the interactive attention module is used to calculate the similarity matrix between the audio and video feature vectors, and it is multiplied by the Value to obtain the output result. At the same time, the Value is input into the residual convolution layer to further extract the deep features and added to the output result of the interactive attention.

[0018] As a preferred technical solution, the construction of the frequency domain feature extraction module is specifically as follows:

[0019] The input video frame is transformed by DCT, the local frequency response values ​​are extracted, and then the average is calculated and mapped back to the multi-channel space of the same size as the input image.

[0020] As a preferred technical solution, the construction of the depth-separable convolution module and the discriminator is specifically as follows:

[0021] Refer to the setting of the depthwise separable convolution block in the 2D Xception network, which includes five four-layer depthwise separable convolution modules, and then performs a two-layer depthwise separable convolution operation again, and then connects to a fully connected layer; the fully connected layer is used for full connection operation and outputs the discrimination result of the input video frame.

[0022] As a preferred technical solution, the loss function is specifically:

[0023]

[0024] where y i represents the true label of test sample i, the fake video is 1, the real video is 0, represents the probability that sample i is detected as a fake video, and N is the total number of test samples;

[0025] This loss is used for backpropagation to train a talking face generation video detection model.

[0026] In a second aspect, the present invention provides a system for detecting a generated video of a speaking face based on multimodal global and local synergy, which is applied to the method for detecting a generated video of a speaking face based on multimodal global and local synergy, and includes a preprocessing module, a model building and training module, and a test execution module; the model building and training module includes a regional attention smoothing detection module, a difference capture-temporal aggregation module, an audio feature extraction module, an audio-video interaction module, a frequency domain feature extraction module, and a depthwise separable convolution module;

[0027] The preprocessing module is used to perform frame segmentation and face region extraction preprocessing on the video, and slice the audio file in the video so that the input audio is aligned with the input video frame in the time dimension;

[0028] The model construction and training module is used to construct a video detection model for generating talking faces and perform training, specifically: first, the original video frame is input into the regional attention smoothing detection module to extract the inter-frame difference information of the video frame transition, and then the frame difference information is input into the difference capture-time aggregation module to capture the difference information between the video frames. At the same time, the audio segment corresponding to the original video is input into the audio feature extraction module to obtain the audio feature, and then the video feature and the audio feature are input into the audio-video interaction module together to obtain the audio-video fusion feature; the original video will also be input into the frequency domain feature extraction module, and the obtained frequency domain feature and the fusion feature are spliced ​​together, and then input into the depthwise separable convolution module for feature extraction, and the generated video is discriminated to obtain the video discrimination label; the cross entropy between the video real label and the video discrimination label is used as the loss function of the entire network model, and the model is trained through loss back propagation to obtain a trained video detection model for generating talking faces;

[0029] The test execution module uses the trained speaking face generation video detection model to judge the video to be detected and obtain the judgment result of the video frame.

[0030] In a third aspect, the present invention provides an electronic device, comprising:

[0031] at least one processor; and,

[0032] a memory communicatively connected to the at least one processor; wherein,

[0033] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the method for generating a speaking face video based on multimodal global and local synergy.

[0034] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the method for generating a video detection of a speaking face based on multimodal global and local synergy.

[0035] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0036] (1) The present invention extracts inter-frame change information from grouped video frames, captures anomalies during inter-frame transitions from a global temporal dimension, and focuses on subtle pixel differences between frames, capturing inter-frame inconsistencies caused by frame-by-frame video generation. This makes the detection method more conducive to capturing traces of video generation forgery and has stronger robustness.

[0037] (2) The present invention not only utilizes the spatial domain information of the video frame, but also extracts the frequency domain information of the video frame, and further extracts the frequency domain information statistics, so that the detection method can capture the anomalies of the generated video in the frequency domain space.

[0038] (3) This invention not only utilizes the visual modal information of the video, but also the audio modal information. By using the attention mechanism to perform similarity calculation and fusion on the video modal information and the audio modal information, it further captures the incoherence between different modalities and improves the accuracy of the detection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0040] Figure 1 This is a flowchart of a method for detecting a speaking face generated video based on multimodal global and local synergy according to an embodiment of the present invention;

[0041] Figure 2 This is a detailed flow chart of the regional attention smoothing detection module, difference capture-time aggregation module, and audio and video interaction module in an embodiment of the present invention;

[0042] Figure 3 This is a structural diagram of a system for generating a video detection system for a speaking face based on global and local synergy of multiple modalities according to an embodiment of the present invention;

[0043] Figure 4 2 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0045] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0046] See also Figure 1 、 Figure 2 This embodiment provides a method for detecting a speaking face generated video based on multimodal global and local synergy, comprising the following steps:

[0047] S1. Perform pre-processing on the video by framing and extracting the face area, save the audio information of the video separately, and slice the audio file so that the input audio is aligned with the input video frame in the time dimension. Specifically:

[0048] S11. Frame processing of the video at 25 frames per second;

[0049] S12. Use Dlib to detect the face area in the video frame by demarcating a rectangular area. Frame a 256×256 area centered on the detected face.

[0050] Furthermore, the face region of the video frame is of other sizes that are convenient for subsequent detection operations;

[0051] S14. For the processed video frames, each input sample includes 8 video frames, and 8 frames are randomly selected from 20 frames.

[0052] S2. Perform frequency domain feature extraction on the grouped video frames through filters, obtain the high and low frequency features corresponding to each video frame, perform statistics, and convert them into feature maps to be input into the convolutional neural network.

[0053] Optionally, the spatial domain information of the features of the original video frame described in step S3 is specifically:

[0054] S31. For the input video features, we calculate the difference between two adjacent video frames. On each image channel, we subtract the previous frame from the next frame to obtain the inter-frame difference representation of the video frame.

[0055] x c diff(t)=x(t+1,c)-x(t,c),t=0,1,…,T-2

[0056] where x cdiff(t) represents the difference between the adjacent frames of each channel of the input image. Since there is no forward frame difference for the first frame and no backward difference for the last frame, we multiply the backward difference of the first frame by two and the forward difference of the last frame by two, and then perform staggered addition on the differential frame combination to obtain the forward and backward difference sum of the frame in the middle position, so that each frame in the new differential frame group approximately represents the difference between the corresponding frame and the previous frame and the next frame.

[0057]

[0058] The inter-frame differences are convolved and weighted, and then each input frame is dot-multiplied with the corresponding inter-frame difference representation and then added to the original input frame, as shown in the following formula:

[0059] x=x+Conv 1×1 (x diff )⊙x

[0060] Where x is the original input, Conv 1×1 This is a 1×1 two-dimensional convolutional layer. It is then fed into a depthwise separable convolutional module for further feature extraction. The depthwise separable convolutional module consists of two layers of depthwise separable convolutions. For details, see the setup of the depthwise separable convolutional block in the 2D Xception network.

[0061] S32. The construction of the difference capture-time aggregation module further includes the following steps:

[0062] At each image pixel, a self-attention mechanism is employed to calculate the channel-wise similarity matrix between all input video frames. The variance in T dimensions is then calculated to obtain a variance matrix. The k-th minimum variance is then selected as the threshold, and the variance matrix is ​​masked, with pixel regions above the threshold set to 1 and those below the threshold set to 0. Finally, the input features are multiplied pixel-by-pixel by the variance matrix and added back to the input features. This is then fed into a two-layer depthwise separable convolution module to obtain the video features. To better align the temporal granularity of audio and video features, this module also performs 1×1, 1×2, and 1×4 pooling operations on the video features horizontally and vertically, respectively. These features are then convolved and upsampled temporally. These multiple outputs are then summed, aggregating the spatiotemporal information of adjacent frames at multiple granularities.

[0063] S41 processes the raw audio input using a pre-trained model and convolutional network. The raw audio sequence is fed into a pre-trained wav2vec model, which encodes the input to generate an audio feature vector. This feature vector is then fed into a residual convolutional network (residual convolutional network) modeled after the residual module in ResNet50. Finally, the resulting audio features are dimensionally resized to align with the video feature vector.

[0064] The audio and video modality consistency detection module described in S42 is constructed based on the cross-attention mechanism. The video feature vector is input into the 3D convolution layer to obtain Query, the audio feature vector is input into the 3D convolution layer to obtain Key, the audio and video feature vectors are linearly summed and input into the convolution layer to obtain Value, and the interactive attention module is used to calculate the similarity matrix between the audio and video feature vectors, and multiply it with the Value to obtain the output result. At the same time, the Value is input into the residual convolution layer to further extract deep features, and added to the output result of the interactive attention to obtain the audio and video fusion feature.

[0065] S5 Finally, we splice the audio and video fusion features with the frequency domain features in the channel dimension to obtain a feature representation that contains all the information extracted from both.

[0066] In S6, we input the integrated feature representation into a depthwise separable convolutional module to obtain deeper feature information. Referring to the setup of the depthwise separable convolutional block in the 2D Xception network, it includes five four-layer depthwise separable convolutional modules, followed by another two-layer depthwise separable convolution operation, and then connects to a fully connected layer; the fully connected layer is used for full connection operation and outputs the judgment result of the input video frame.

[0067] Optionally, in step S6, the cross entropy between the real video label and the video discriminant label is used as the loss function of the entire network model, and the model is trained by loss back propagation to obtain a trained speaking face generation video detection model, specifically,

[0068]

[0069] where y i represents the true label of test sample i, the fake video is 1, the real video is 0, represents the probability that sample i is detected as a fake video, and N is the total number of test samples.

[0070] S7. Use the trained speaking face generation video detection model to judge the video to be detected and obtain the judgment result of the video frame.

[0071] In a more specific embodiment, the steps are as follows:

[0072] S1: Preprocessing: After dividing the video into frames and extracting the face area, data enhancement operations are performed on the divided training set. Data enhancement methods include Gaussian noise, rotation and filter blur, and the processed video frames are grouped.

[0073] S2: Inter-frame difference extraction: Utilize the constructed region-attention smoothing detection module and difference capture-temporal aggregation module to mine the non-coordinated information of the transition between video frames and the subtle difference areas between the pixels of the frames, and then discover the forged or generated features of the video.

[0074] S3: Audio-Video Interaction: Based on the consistency assumption between video and audio modal information, the cross-attention mechanism is used to calculate and fuse the similarity of audio and video features, thereby mining the audio-visual inconsistency information of the generated video and detecting the authenticity of the video to be tested.

[0075] S4: Feature Integration and Result Discrimination: This module integrates the features extracted by the final fusion feature extraction module based on 2D depthwise separable convolutions to distinguish video authenticity. The feature integration and result discrimination module consists of five depthwise separable convolution blocks, each containing four layers of 2D depthwise separable convolutions, followed by two layers of 2D depthwise separable convolutions and one fully connected layer.

[0076] S5: Training and Testing: Calculate the cross-entropy loss between the true and detected video labels and backpropagate to train the deep face forgery video detection model. After training, the test set is fed into the trained model to determine whether the input video has been forged or tampered with.

[0077] In this embodiment, the discrimination results of the video frames are verified through experiments, as shown in Table 1. Table 1 shows the detection accuracy of the speaking face generation video detection method based on multimodal global and local synergy on the FakeAVCeleb, DFDC and MSTF datasets.

[0078] Table 1

[0079] Dataset FakeAVCeleb DFDC MSTF Accuracy (ACC) 0.9892 0.9258 0.8849

[0080] As shown in Table 1, the multimodal global and local synergy-based speaking face generated video detection method of this embodiment demonstrates outstanding performance on the FakeAVCeleb, DFDC, and MSTF datasets. In particular, its detection accuracy exceeds 90% on the FakeAVCeleb and DFDC datasets. The closest existing solution, a multimodal deepfake detection method, achieves an accuracy of approximately 97.69% on the FakeAVceleb dataset, but less than 90% on the DFDC dataset. In comparison, the multimodal global and local synergy-based speaking face generated video detection method of this embodiment performs better in terms of the proportion of detection results. Consequently, this embodiment exhibits superior energy efficiency compared to other existing generated video detection methods.

[0081] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0082] Based on the same concept as the method for detecting a speaking face generated from a video based on multimodal global and local synergy in the above-mentioned embodiment, the present invention also provides a system for detecting a speaking face generated from a video based on multimodal global and local synergy, which can be used to execute the above-mentioned method for detecting a speaking face generated from a video based on multimodal global and local synergy. For ease of explanation, the structural diagram of the embodiment of the system for detecting a speaking face generated from a video based on multimodal global and local synergy only shows the parts related to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and the device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0083] See also Figure 3 In another embodiment of the present application, a system 10 for detecting a speaking face generated video based on global and local synergy of multimodalities is provided, the system comprising a preprocessing module 11, an inter-frame difference extraction and multimodal interaction module 12, a model building and training module 13, and a test execution module 14;

[0084] The pre-processing module 11 performs framing and face area extraction on the video, performs framing operation on the video at 25 frames per second, uses Dlib to delineate a rectangular area to detect the face area of ​​the video frame, and frames an area of ​​256×256 with the detected face as the center; saves the audio information of the video separately, slices the audio file, and aligns the input audio with the input video frame in the time dimension.

[0085] Furthermore, the face region of the video frame is of other sizes that are convenient for subsequent detection operations;

[0086] The inter-frame difference extraction and multimodal interaction module 12 calculates the difference between two adjacent video frames for the input video features, then performs staggered addition on the difference frame combination so that each frame in the difference frame group approximately represents the difference between the corresponding frame and the previous and next frames. The inter-frame difference is convolved and weighted, and then each input frame is dot-multiplied with the corresponding inter-frame difference representation and then added to the original input frame. At each image pixel, a self-attention mechanism is used to calculate the channel similarity matrix between all input video frames. The variance of the T dimension is then calculated to obtain the variance matrix, and then a threshold k is selected for regional activation. Finally, it is input into a 2-layer depthwise separable convolution module to obtain the video features. The video features are then pooled with 1×1, 1×2, and 1×4 in the horizontal and vertical directions, respectively, and then convolved and upsampled respectively. Finally, these multiple outputs are added together to aggregate the spatiotemporal information of adjacent frames at multiple granularities.

[0087] The model construction and training module 13 is used to construct a forged video detection model, which uses the features of the original video frame and the audio features to input the preset inter-frame difference extraction and multimodal interaction module in a group manner to perform inter-frame temporal feature extraction and multimodal interaction; the extracted fusion features are integrated in the preset feature integration and result discrimination module, and then the features are further extracted and the forged and tampered video is discriminated to obtain a video discrimination label; the cross entropy between the real video label and the video discrimination label is used as the loss function of the entire network model, and the model is trained through loss back propagation to obtain a trained generated video detection model; the generated video detection model includes an inter-frame difference extraction and multimodal interaction module and a feature integration and result discrimination module; the feature integration and result discrimination module is constructed based on a 2D convolution operation;

[0088] The test execution module 14 is used to use the trained generated video detection model to judge the video to be detected and obtain the judgment result of the video frame.

[0089] It should be noted that the speaking face generation video detection system based on multimodal global and local synergy of the present invention corresponds one-to-one to the speaking face generation video detection method based on multimodal global and local synergy of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the speaking face generation video detection method based on multimodal global and local synergy are applicable to the embodiment of the speaking face generation video detection method based on multimodal global and local synergy. For specific details, please refer to the description in the embodiment of the method of the present invention. No further details will be given here. This is hereby declared.

[0090] In addition, in the implementation of the speaking face generation video detection system based on multimodal global and local synergy in the above-mentioned embodiment, the logical division of each program module is only an example. In actual application, the above-mentioned functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the speaking face generation video detection system based on multimodal global and local synergy is divided into different program modules to complete all or part of the functions described above.

[0091] See also Figure 4 In one embodiment, an electronic device is provided for implementing a method for detecting a speaking face generated video based on global and local synergy of multimodality. The electronic device 20 may include a first processor 21, a first memory 22, and a bus. It may also include a computer program stored in the first memory 22 and executable on the first processor 21, such as a speaking face generated video detection program 23 based on global and local synergy of multimodality.

[0092] The first memory 22 includes at least one type of readable storage medium, including flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory 22 may be an internal storage unit of the electronic device 20, such as a mobile hard disk of the electronic device 20. In other embodiments, the first memory 22 may also be an external storage device of the electronic device 20, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc. equipped on the electronic device 20. Furthermore, the first memory 22 may include both an internal storage unit of the electronic device 20 and an external storage device. The first memory 22 can be used not only to store application software installed on the electronic device 20 and various types of data, such as the code of the multimodal global and local synergy-based speaking face generation video detection program 23, but can also be used to temporarily store data that has been output or is about to be output.

[0093] In some embodiments, the first processor 21 may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The first processor 21 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 22, as well as calling data stored in the first memory 22, to perform various functions of the electronic device 20 and process data.

[0094] Figure 4 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 4 The structure shown does not constitute a limitation on the electronic device 20 , and the electronic device 20 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0095] The first memory 22 in the electronic device 20 stores a multimodal global and local synergy-based speaking face generation video detection program 23, which is a combination of multiple instructions. When running in the first processor 21, the program can achieve the following:

[0096] The video is framed and face areas are extracted. The video is framed at 25 frames per second. Dlib is used to draw a rectangular area to detect the face area of ​​the video frame. A 256×256 area is framed with the detected face as the center. The audio information of the video is saved separately, and the audio file is sliced ​​to align the input audio with the input video frame in the time dimension.

[0097] The input video features are generated by calculating the difference between two adjacent video frames. These difference frames are then staggered and added together, ensuring that each frame in the difference frame group approximately represents the difference between the corresponding frame and the previous and next frames. The inter-frame differences are convolved and weighted, and each input frame is dot-multiplied with the corresponding inter-frame difference representation before being added to the original input frame. At each image pixel, a self-attention mechanism is employed to calculate the channel-wise similarity matrix between all input video frames. The variance in T dimensions is then calculated to obtain the variance matrix, and a threshold k is selected for region activation. Finally, the features are fed into a two-layer depthwise separable convolutional module to generate the video features. The video features are then pooled horizontally and vertically with 1×1, 1×2, and 1×4 layers, respectively, followed by convolution and upsampling. Finally, these multiple outputs are summed to aggregate spatiotemporal information from adjacent frames at multiple granularities.

[0098] The features of the original video frames and the audio features are input into a preset inter-frame difference extraction and multimodal interaction module in a grouped manner to perform inter-frame temporal feature extraction and multimodal interaction; the extracted fusion features are integrated in a preset feature integration and result discrimination module, and then the features are further extracted and the forged and tampered videos are discriminated to obtain a video discrimination label; the cross entropy between the real video label and the video discrimination label is used as the loss function of the entire network model, and the model is trained through loss back propagation to obtain a trained generated video detection model; the generated video detection model includes an inter-frame difference extraction and multimodal interaction module and a feature integration and result discrimination module; the feature integration and result discrimination module is constructed based on a 2D convolution operation;

[0099] The trained speaking face generation video detection model is used to identify the video to be detected and obtain the identification results of the video frames.

[0100] Furthermore, if the modules / units integrated in the electronic device 20 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0101] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0102] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0103] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for detecting speaking faces in video based on multimodal global and local synergy, characterized in that: The steps include: Perform pre-processing on the video by framing and extracting the face area, and slice the audio file in the video so that the input audio is aligned with the input video frame in the time dimension; A talking face generation video detection model is constructed and trained. Specifically, the original video frame is first input into the regional attention smoothing detection module to extract the inter-frame difference information of the video frame transition, and then input into the difference capture-time aggregation module to capture the difference information between video frames. At the same time, the audio segment corresponding to the original video is input into the audio feature extraction module to obtain audio features, and then the video features and audio features are input into the audio-video interaction module together to obtain audio-video fusion features; the original video is also input into the frequency domain feature extraction module, and the obtained frequency domain features are spliced ​​with the fusion features. The features are then input into the depthwise separable convolution module for feature extraction, and the generated video is discriminated to obtain the video discrimination label; the cross entropy between the video real label and the video discrimination label is used as the loss function of the entire network model, and the model is trained through loss back propagation to obtain a trained talking face generation video detection model; The trained speaking face generation video detection model is used to identify the video to be detected and obtain the identification results of the video frames.

2. The method for detecting speaking face generation in video based on multimodal global and local synergy according to claim 1, characterized in that: The region attention smoothing detection module is constructed based on inter-frame difference calculation. By calculating the difference between two adjacent video frames, the inter-frame difference representation of the video frame is obtained, and the inter-frame differences are staggered and added so that each inter-frame difference representation contains the difference between the corresponding frame and the previous frame and the next frame; the inter-frame differences are assigned channel weights through the convolution layer, and then each input frame is dot-multiplied with the corresponding inter-frame difference representation, and then added to the original input frame, and finally input into a two-layer depth-separable convolution module for deep feature extraction.

3. The method for detecting speaking face generation in video based on multimodal global and local synergy according to claim 1, characterized in that: The difference capture-temporal aggregation module is constructed based on the self-attention mechanism and the mask mechanism, and specifically includes the following steps: At each image pixel, the self-attention mechanism is used to calculate the channel similarity matrix between all input video frames, and then the variance of the T dimension is calculated to obtain the variance matrix. The kth minimum variance is then selected as the threshold, and the variance matrix is ​​masked. The pixel area greater than the threshold is set to 1, and vice versa. Finally, the input feature is multiplied pixel by pixel with the variance matrix and added back to the input feature. Finally, it is input into a depthwise separable convolution module to obtain the video feature. In order to achieve the temporal granularity alignment of audio and video features, the difference capture-temporal aggregation module also performs pooling operations of different window sizes on the video features in the horizontal and vertical directions, and then performs convolution and upsampling in the temporal order. Finally, these multiple outputs are added together to aggregate the spatiotemporal information of adjacent frames from multiple granularities.

4. The method for detecting speaking face generation in video based on multimodal global and local synergy according to claim 1, characterized in that: The audio feature extraction module is built based on the pre-trained wav2vec model, specifically: The original audio sequence is input into the pre-trained wav2vec model, and the input original audio sequence is encoded to obtain an audio feature vector; the audio feature vector is then input into the residual convolutional network, and finally the obtained audio features are dimensionally adjusted to align with the video feature vector.

5. The method for detecting speaking face generated video based on multimodal global and local synergy according to claim 1, characterized in that: The audio and video interaction module is built based on the cross attention model, specifically: The video feature vector is input into the 3D convolution layer to obtain the query, and the audio feature vector is input into the 3D convolution layer to obtain the key. The audio and video feature vectors are linearly summed and input into the convolution layer to obtain the value. The interactive attention module is used to calculate the similarity matrix between the audio and video feature vectors and multiply it with the value to obtain the output result. At the same time, the value is input into the residual convolution layer to further extract deep features and added with the output result of the interactive attention. The construction of the frequency domain feature extraction module is specifically as follows: The input video frame is transformed by DCT, the local frequency response values ​​are extracted, and then the average is calculated and mapped back to the multi-channel space of the same size as the input image.

6. The method for detecting speaking face generation in video based on multimodal global and local synergy according to claim 1, characterized in that: The construction of the depth-wise separable convolutional module and the discriminator is specifically as follows: Referring to the setting of the depthwise separable convolution block in the 2D Xception network, it includes five four-layer depthwise separable convolution modules, and then performs a two-layer depthwise separable convolution operation again, and then accesses a fully connected layer; the fully connected layer is used for full connection operation and outputs the discrimination result of the input video frame.

7. The method for detecting speaking face generation video based on multimodal global and local synergy according to claim 1, characterized in that: The loss function is specifically: where y i represents the true label of test sample i, the fake video is 1, the real video is 0, represents the probability that sample i is detected as a fake video, and N is the total number of test samples; This loss is used for backpropagation to train a talking face generation video detection model.

8. A multimodal global and local synergy-based video detection system for speaking faces, characterized in that: A method for detecting a generated video of a speaking face based on multimodal global and local synergy, as applied to any one of claims 1-7, comprising a preprocessing module, a model building and training module, and a test execution module; the model building and training module comprises a regional attention smoothing detection module, a difference capture-time aggregation module, an audio feature extraction module, an audio-video interaction module, a frequency domain feature extraction module, and a depthwise separable convolution module; The preprocessing module is used to perform frame segmentation and face region extraction preprocessing on the video, and slice the audio file in the video so that the input audio is aligned with the input video frame in the time dimension; The model construction and training module is used to construct a video detection model for generating talking faces and perform training, specifically: first, the original video frame is input into the regional attention smoothing detection module to extract the inter-frame difference information of the video frame transition, and then the frame difference information is input into the difference capture-time aggregation module to capture the difference information between the video frames. At the same time, the audio segment corresponding to the original video is input into the audio feature extraction module to obtain the audio feature, and then the video feature and the audio feature are input into the audio-video interaction module together to obtain the audio-video fusion feature; the original video will also be input into the frequency domain feature extraction module, and the obtained frequency domain feature and the fusion feature are spliced ​​together, and then input into the depthwise separable convolution module for feature extraction, and the generated video is discriminated to obtain the video discrimination label; the cross entropy between the video real label and the video discrimination label is used as the loss function of the entire network model, and the model is trained through loss back propagation to obtain a trained video detection model for generating talking faces; The test execution module uses the trained speaking face generation video detection model to judge the video to be detected and obtain the judgment result of the video frame.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the method for generating a video detection of a speaking face based on multimodal global and local synergy as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the method for detecting speaking face generation videos based on multimodal global and local synergy described in any one of claims 1 to 7 is implemented.

Images