Video authentication and time positioning method and device, electronic equipment and storage medium

Abstract

The application discloses a video authentication and time positioning method and device, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, and comprises the following steps: performing feature extraction on video and audio in a video file respectively to obtain audio and video features aligned in the time dimension; performing cross-modal cross-reconstruction on the extracted audio and video features, and supervising the cross-modal cross-reconstruction step by using the spatial difference between true and false features, thereby obtaining spatial error features; determining the probability that the corresponding current frame is tampered with according to each feature, and then performing frame classification on each current frame respectively; and performing time positioning on the tampered data frames in the video file according to each feature and the probability that the corresponding current frame is tampered with. The application obtains the error between audio and video modalities by cross-modal cross-reconstruction and supervising the reconstruction process by using the spatial difference between true and false features, and then more accurately focuses on the tampered video frames, so that the accuracy of authentication and time positioning can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to video authentication and time positioning methods, devices, electronic devices and storage media. Background Technology

[0002] With the development of deepfake technology, the generated videos are becoming increasingly realistic, making them difficult to distinguish with the naked eye. Existing technologies often only target single-modal tampering, resulting in a limited scope of identification. Even some existing technologies that utilize multimodal information are limited to binary classification of whether or not tampering has occurred, and cannot determine the specific tampered segment; some technical solutions that utilize multimodal deepfake time localization have low identification and localization accuracy. Summary of the Invention

[0003] The main objective of this application is to propose a video authentication and time-based positioning method, apparatus, electronic device, and storage medium to improve the accuracy of video authentication and time-based positioning.

[0004] To achieve the above objectives, one aspect of this application proposes a video authentication and time-based positioning method, which includes the following steps:

[0005] Feature extraction is performed on the video and audio in the video file respectively to obtain time-aligned input video features and input audio features;

[0006] Cross-modal cross-reconstruction is performed on the input video features and the input audio features, and the spatial difference between real and fake features is used to supervise the cross-modal cross-reconstruction steps to obtain spatial error features.

[0007] The probability of the current frame being tampered with is determined based on the input video features, the input audio features, and the spatial error features, and then each current frame is classified accordingly.

[0008] The tampered data frames in the video file are located temporally based on the input video features, the input audio features, the spatial error features, and the probability that the current frame has been tampered with; wherein the data frame includes at least one of the current frame corresponding to the input video features or the current frame corresponding to the input audio features.

[0009] In some embodiments, before extracting features from the video and audio in the video file to obtain time-aligned input video features and input audio features, the method further includes a preprocessing step for the video file, which includes the following steps:

[0010] The video and audio are extracted from the video file;

[0011] Fill in the video and the audio respectively;

[0012] The size of the filled video is scaled and the channels are adjusted to obtain a video tensor of the first shape;

[0013] The Mel spectrogram is extracted from the filled audio, and then the Mel spectrogram is represented as an audio tensor of a second shape;

[0014] The step of extracting features from the video and audio files separately to obtain time-aligned input video features and input audio features includes the following steps:

[0015] Feature extraction is performed on the video tensor and the audio tensor respectively to obtain time-aligned input video features and input audio features.

[0016] In some embodiments, before the step of performing cross-modal cross-reconstruction on the input video features and the input audio features, and supervising the cross-modal cross-reconstruction using the spatial difference between real and fake features to obtain spatial error features, the method further includes a step of training a cross-modal cross-reconstruction module, the step of training the cross-modal cross-reconstruction module including the following steps:

[0017] A V2A network is trained using video features labeled with both real and tampered data; an A2V network is trained using audio features labeled with both real and tampered data; wherein the cross-modal reconstruction module includes the V2A network and the A2V network; both the A2V network and the V2A network are implemented using a standard 4-layer transformer.

[0018] The expression for the loss function of the cross-modal reconstruction module includes:

[0019]

[0020] P a =V2A(IN) v );

[0021] P v =A2V(IN) a );

[0022] Where N represents the number of video frames, P vi IN represents the features of the i-th video frame predicted using the input audio features. vi Represents the input video features; P ai IN represents the features of the i-th audio frame predicted using the input video features. ai Represents the audio features of the input;

[0023] The expression for the reconstruction supervision function of the cross-modal reconstruction module includes:

[0024]

[0025] Mask pred_a =P a -Real a ;

[0026]

[0027] Mask ref_a =Fake a -Real a =IN a -Real a ;

[0028] Mask ref_v =Fake v -Real v =IN v -Real v ;

[0029] P a =V2A(IN) v );

[0030] P v =A2V(IN) a );

[0031] Among them, Mask pred_v This indicates that the predicted video features are obtained by reconstructing the original video features from the input tampered audio features, and the difference is obtained by subtracting the predicted video features from the reference real video features; Mask ref Mask represents the difference obtained by subtracting the tampered video features from the referenced real video features; pred_a This indicates that the predicted audio features are reconstructed from the input tampered video features, and the difference between the predicted audio features and the reference real audio features is obtained; Mask ref_a This represents the difference obtained by subtracting the tampered audio features from the referenced real audio features; N represents the number of video frames, P v This refers to video features predicted using the input audio features; Fake v =IN v Indicates the tampered video features of the input; P a This refers to the audio features predicted using the input video features; Fake a =IN a Indicates the characteristics of the tampered audio input; Real aIndicates the corresponding real audio features, Real v This indicates the corresponding real-world video features.

[0032] In some embodiments, the step of performing cross-modal cross-reconstruction on the input video features and the input audio features, and supervising the cross-modal cross-reconstruction using the spatial difference between real and fake features to obtain spatial error features, includes the following steps:

[0033] The input video features are input into the V2A network, and the V2A network is used to infer and obtain reconstructed audio features; the input audio features are input into the A2V network, and the A2V network is used to infer and obtain reconstructed video features.

[0034] The difference between the input video features and the reconstructed video features is used to obtain the reconstructed video error features; the difference between the input audio features and the reconstructed audio features is used to obtain the reconstructed audio error features.

[0035] The reconstructed video error features and the reconstructed audio error features are input into a linear layer and linearly weighted to obtain the spatial error features.

[0036] In some embodiments, the method further includes the following steps:

[0037] Once training converges, the expected distribution of the reconstructed features is determined as follows:

[0038]

[0039] Among them, P a,i This represents the audio features predicted using the input video features; P v,i This indicates the features of the video predicted using the input audio features; Fake a,i This represents the features of the tampered audio frames in the training dataset, Fake v,i This represents the features of the tampered video frames in the dataset used for training.

[0040] Furthermore, once the training converges, the distribution of the spatial error characteristics of the output after subtraction is determined as follows:

[0041]

[0042] Among them, Out a,i The audio features output by the cross-modal cross-reconstruction module are used as the first spatial error; Out v,i The video features output by the cross-modal cross-reconstruction module are used as the second spatial error; the first spatial error and the second spatial error are used for subsequent audio and video frame classification and temporal localization tasks;

[0043] The step of classifying each current frame includes the following steps:

[0044] Each current frame is classified according to the loss function for frame classification.

[0045] The loss function for frame classification includes:

[0046]

[0047] Among them, Lossf a Lossf v Lossf s These represent the loss functions used for frame classification based on the input audio features, the input video features, and the spatial error features, respectively. It is the standard cross-entropy loss function.

[0048] In some embodiments, the step of temporally locating the tampered data frame in the video file based on the input video features, the input audio features, the spatial error features, and the probability that the current frame has been tampered with includes the following steps:

[0049] The temporal localization loss function is used to perform temporal localization on the tampered data frames in the video file based on the input video features, the input audio features, the spatial error features, and the probability that the current frame has been tampered with.

[0050] The loss function for the time-based positioning is:

[0051]

[0052] Among them, Lossb a These are used as boundary matching loss functions for audio tampering time localization, video tampering time localization, and any modal tampering time localization, respectively; N is the number of video frames; D is the maximum number of frames for the tampered data frame segment; Y represents the confidence level of the tampered data frame segment from the i-th frame to the (i+j)-th frame. b a,i j represents the corresponding truth value.

[0053] In some embodiments, the method further includes the following steps:

[0054] The tampered data frame and its corresponding time location are visualized.

[0055] To achieve the above objectives, another aspect of this application provides a video authentication and time positioning device, the device comprising:

[0056] The feature extraction unit is used to extract features from the video and audio in the video file respectively, and obtain the input video features and input audio features aligned to the time dimension.

[0057] The feature reconstruction unit is used to perform cross-modal cross-reconstruction on the input video features and the input audio features, and to supervise the cross-modal cross-reconstruction steps by utilizing the spatial difference between real and fake features, thereby obtaining spatial error features;

[0058] The feature classification unit is used to determine the probability that the current frame has been tampered with based on the input video features, the input audio features and the spatial error features, and then classify each current frame accordingly.

[0059] A time localization unit is used to perform time localization of the tampered data frame in the video file based on the input video features, the input audio features, the spatial error features, and the probability that the current frame is tampered with; wherein the data frame includes at least one of the current frame corresponding to the input video features or the current frame corresponding to the input audio features.

[0060] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described video authentication and time positioning method.

[0061] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described video authentication and time positioning method.

[0062] The embodiments of this application include at least the following beneficial effects:

[0063] This application can extract features from the video and audio in a video file separately to obtain time-aligned input video features and input audio features; perform cross-modal cross-reconstruction on the input video and audio features, and supervise the cross-modal cross-reconstruction step using the spatial difference between real and fake features to obtain spatial error features; determine the probability of the corresponding current frame being tampered with based on the input video features, input audio features, and spatial error features, and then classify each current frame separately; perform temporal localization of the tampered data frames in the video file based on the input video features, input audio features, spatial error features, and the probability of the corresponding current frame being tampered with; wherein, the data frame includes at least one of the current frames corresponding to the input video features or the current frames corresponding to the input audio features. This application obtains the error between audio and video modalities through cross-modal cross-reconstruction, thus more accurately focusing on specific tampered video frames and improving the accuracy of counterfeit detection and temporal localization. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 A flowchart illustrating the video authentication and time positioning method provided in this application embodiment;

[0066] Figure 2 Example flowchart of step S100 provided in the embodiments of this application;

[0067] Figure 3 Example flowchart of step S200 provided in the embodiments of this application;

[0068] Figure 4 Example flowchart of step S120 provided in the embodiments of this application;

[0069] Figure 5 Example flowchart of step S130 provided in the embodiments of this application;

[0070] Figure 6 An example flowchart of a video authentication and time positioning method provided in this application embodiment;

[0071] Figure 7 Example flowchart of the training cross-modal cross-reconstruction module provided in the embodiments of this application;

[0072] Figure 8An example flowchart for inference using a cross-modal cross-reconstruction module provided in this application embodiment;

[0073] Figure 9 This is a schematic diagram of the structure of the video authentication and time positioning device provided in the embodiments of this application;

[0074] Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0076] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0077] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0078] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0079] Before providing a detailed description of the embodiments of this application, some of the nouns and terms used in the embodiments of this application will be explained first. The nouns and terms used in the embodiments of this application shall be interpreted as follows:

[0080] Deepfake refers to the use of deep learning technology and neural networks to generate fake audio and video content. This technique typically uses methods such as Generative Adversarial Networks (GANs) to synthesize one person's facial expressions, voice, or movements into another person's audio or video, thereby creating seemingly real but actually manipulated content.

[0081] Cross-modal reconstruction refers to the process of reconstructing one modality (such as audio) from another modality (such as video). This technique is used in multimodal authentication to detect inconsistencies between different modalities, helping to identify tampered content.

[0082] Temporal Localization: In video processing and forgery detection, temporal localization refers to accurately determining the exact time when tampered content occurs within a video. By analyzing cross-modal reconstruction errors in video and audio, it is possible to time-tag and locate tampered segments.

[0083] Mel-Spectrogram: A Mel-Spectrogram is a technique for converting audio signals into a spectral representation. Based on the perceptual characteristics of the human auditory system, it divides frequencies according to the Mel scale. The Mel-Spectrogram first converts the time-domain audio signal into a frequency-domain representation using a Short-Time Fourier Transform (STFT), obtaining the amplitudes of different frequencies. These frequencies are then mapped to the Mel scale. The result is a two-dimensional matrix where the horizontal axis represents time, the vertical axis represents the frequency components after the Mel transformation, and the color or intensity represents the energy of different frequencies within each time segment.

[0084] Deepfake video authentication and time positioning is one of the key technologies required for cybersecurity and forensic identification. It is a technology used to distinguish between videos generated by Deepfake technology and real videos, and to locate specific tampered segments.

[0085] Deepfake videos can be manipulated visually, including but not limited to traditional Generative Adversarial Networks (GANs) and the currently popular Diffusion Model.

[0086] Deepfake videos can be manipulated in the auditory modality, including but not limited to voice cloning, text-to-speech (TTS), etc.

[0087] Deepfake videos can also be manipulated simultaneously in both visual and auditory modalities, for example, by using lip-sync technology to make the video images have lip movements that match the manipulated audio.

[0088] As deepfake technology advances, the videos it generates become increasingly realistic, making them difficult to distinguish with the naked eye. Existing technologies often only target single-modal tampering, neglecting the potential auxiliary role of audio-video correspondence, resulting in limited identification capabilities. Even methods utilizing multimodal information tend to be limited to binary classification of whether tampering has occurred, failing to identify specific tampered segments. Furthermore, methods using multimodal deepfake temporal localization do not employ cross-reconstruction to focus on specific tampered areas, leading to low accuracy in identification and localization.

[0089] This application provides a video authentication and time-based localization method, apparatus, electronic device, and storage medium. The technical solution includes: extracting features from the video and audio in a video file to obtain time-aligned input video features and input audio features; performing cross-modal cross-reconstruction on the input video and audio features, and supervising the cross-modal cross-reconstruction step using the spatial difference between true and false features to obtain spatial error features; determining the probability of the current frame being tampered with based on the input video features, input audio features, and spatial error features, and then classifying each current frame; and performing time-based localization of tampered data frames in the video file based on the input video features, input audio features, spatial error features, and the probability of the corresponding current frame being tampered with; wherein, the data frame includes at least one of the current frame corresponding to the input video features or the current frame corresponding to the input audio features. This application obtains the error between audio and video modalities through cross-modal cross-reconstruction, thereby more accurately focusing on specific tampered video frames and improving the accuracy of authentication and time-based localization.

[0090] This application provides a video authentication and time-based positioning method, apparatus, electronic device, and storage medium, relating to the field of artificial intelligence technology. The video authentication and time-based positioning method, apparatus, electronic device, and storage medium provided in this application can be applied to terminals, servers, or software running on terminals or servers. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application implementing knowledge extraction methods, but is not limited to the above forms.

[0091] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0092] Reference Figure 1 This application provides a video authentication and time positioning method, which may include, but is not limited to, steps S110 to S140, as follows:

[0093] S110: Extract features from the video and audio in the video file to obtain time-aligned input video features and input audio features.

[0094] In some examples, this application may preprocess the video file before executing S110. Therefore, embodiments of this application may further include a step of preprocessing the video file in S100 before S110, see reference. Figure 2 S100 may include the following steps S101 to S104:

[0095] S101: Extract the video and audio from the video file;

[0096] S102: Fill in the video and the audio respectively;

[0097] S103: Scale the size of the filled video and adjust the channels to obtain a video tensor of the first shape;

[0098] S104: Extract the Mel spectrogram from the filled audio, and then represent the Mel spectrogram as an audio tensor of a second shape.

[0099] Furthermore, S110 may further include:

[0100] Feature extraction is performed on the video tensor and the audio tensor respectively to obtain time-aligned input video features and input audio features.

[0101] S120: Perform cross-modal cross-reconstruction on the input video features and the input audio features, and use the spatial difference between real and fake features to supervise the cross-modal cross-reconstruction steps, thereby obtaining spatial error features.

[0102] In some embodiments, this application may further include the step of S200 training the cross-modal cross-reconstruction module, referring to... Figure 3 The steps for training the cross-modal cross-reconstruction module include the following steps S201 to S202:

[0103] S201: Train a V2A network using video features labeled with both real and tampered features;

[0104] S202: Train the A2V network using audio features labeled with real and tampered data; wherein the cross-modal reconstruction module includes the V2A network and the A2V network; both the A2V network and the V2A network are implemented using a 4-layer standard transformer.

[0105] The expression for the loss function of the cross-modal reconstruction module includes:

[0106]

[0107] P a =V2A(IN) v );

[0108] P v =A2V(IN) a );

[0109] Where N represents the number of video frames, P vi IN represents the features of the i-th video frame predicted using the input audio features. vi Represents the input video features; P ai IN represents the features of the i-th audio frame predicted using the input video features. ai Represents the audio features of the input;

[0110] The expression for the reconstruction supervision function of the cross-modal reconstruction module includes:

[0111]

[0112] Mask pred_a =P a -Real a ;

[0113]

[0114] Mask ref_a =Fakea -Real a =IN a -Real a ;

[0115] Mask ref_v =Fake v -Real v =IN v -Real v ;

[0116] P a =V2A(IN) v );

[0117] P v =A2V(IN) a );

[0118] Among them, Mask pred_v This indicates that the predicted video features are obtained by reconstructing the original video features from the input tampered audio features, and the difference between the predicted video features and the reference real video features is obtained; Masl ref This represents the difference obtained by subtracting the tampered video features from the referenced real video features; Masl pred_a This indicates that the predicted audio features are reconstructed from the input tampered video features, and the difference between the predicted audio features and the reference real audio features is obtained; Mask ref_a This represents the difference obtained by subtracting the tampered audio features from the referenced real audio features; N represents the number of video frames, P v This refers to video features predicted using the input audio features; Fake v =IN v Indicates the tampered video features of the input; P a This refers to the audio features predicted using the input video features; fake a =IN a Indicates the characteristics of the tampered audio input; Real a Indicates the corresponding real audio features, Real v This indicates the corresponding real-world video features.

[0119] Reference Figure 4 S120 may further include the following steps S121 to S123:

[0120] S121: Input the input video features into the V2A network and use the V2A network to infer reconstructed audio features; input the input audio features into the A2V network and use the A2V network to infer reconstructed video features;

[0121] S122: Subtract the input video features from the reconstructed video features to obtain the reconstructed video error features; subtract the input audio features from the reconstructed audio features to obtain the reconstructed audio error features;

[0122] S123: Input the reconstructed video error features and the reconstructed audio error features into a linear layer for linear weighting to obtain the spatial error features.

[0123] S130: Determine the probability that the current frame has been tampered with based on the input video features, the input audio features and the spatial error features, and then classify each current frame accordingly.

[0124] Reference Figure 5 The embodiments of this application may further include the following steps S131 to S133:

[0125] S131: After training converges, the expected distribution of the reconstructed features is determined as follows:

[0126]

[0127] Among them, P a,i This represents the audio features predicted using the input video features; P v,i This indicates the features of the video predicted using the input audio features; Fake a,i This represents the features of the tampered audio frames in the training dataset, Fake v,i This represents the features of the tampered video frames in the dataset used for training.

[0128] S132: Furthermore, after training converges, the distribution of the spatial error characteristics of the output after subtraction is determined as follows:

[0129]

[0130] Among them, Out a,i The audio features output by the cross-modal cross-reconstruction module are used as the first spatial error; Out v,i The video features output by the cross-modal cross-reconstruction module are used as the second spatial error; the first spatial error and the second spatial error are used for subsequent audio and video frame classification and temporal localization tasks;

[0131] It should be noted that the output is close to a zero vector if and only if both audio and video modalities are true; otherwise, it follows a specific distribution. The cross-reconstruction module outputs a feature vector Out only when neither the audio nor video modalities have been tampered with. a and Out vThe values ​​tend to 0, but if any modality is tampered with, neither of them will be 0, thus distinguishing between real audio and video input features and tampered audio and video input features;

[0132] The step of classifying each current frame includes the following steps:

[0133] S133: Perform frame classification on each of the current frames according to the frame classification loss function;

[0134] The loss function for frame classification includes:

[0135]

[0136] Among them, Lossf a Lossf v Lossf s These represent the loss functions used for frame classification based on the input audio features, the input video features, and the spatial error features, respectively. It is the standard cross-entropy loss function.

[0137] S140: Based on the input video features, the input audio features, the spatial error features, and the probability that the current frame has been tampered with, the tampered data frame in the video file is located in time; wherein, the data frame includes at least one of the current frame corresponding to the input video features or the current frame corresponding to the input audio features.

[0138] Furthermore, S140 may include the following steps:

[0139] The temporal localization loss function is used to perform temporal localization on the tampered data frames in the video file based on the input video features, the input audio features, the spatial error features, and the probability that the current frame has been tampered with.

[0140] The loss function for the time-based positioning is:

[0141]

[0142] Among them, Lossb a These are used as boundary matching loss functions for audio tampering time localization, video tampering time localization, and any modal tampering time localization, respectively; N is the number of video frames; D is the maximum number of frames for the tampered data frame segment; Y represents the confidence level of the tampered data frame segment from the i-th frame to the (i+j)-th frame. b a,i j represents the corresponding truth value.

[0143] As a further implementation method, the embodiments of this application may also include the following steps:

[0144] S150: Visualize the tampered data frame and the corresponding time location.

[0145] For example, at least one of the audio or video frames determined to have been tampered with, along with the corresponding tampering time, can be visualized, for example, through text, images, video, or voice.

[0146] The following section will provide a detailed introduction and explanation of the solutions in the embodiments of this application, using specific application examples.

[0147] The purpose of this embodiment is to provide a multimodal deepfake video authentication and temporal localization method based on audio-video cross-reconstruction. By utilizing the spatial error of the audio-video cross-reconstruction, the authentication model can focus on specific tampered areas. Specifically, the cross-reconstruction module uses the cross-modal guided contrast loss function provided in this embodiment during the training phase to distinguish between real and tampered data. Specifically, this embodiment includes the following key technical features:

[0148] (1) By using cross-modal cross-reconstruction, the model focuses on the specific tampered area, generating cross-modal spatial error, which is used for frame-level classification and temporal localization, thereby improving the identification and temporal localization capabilities of the anti-counterfeiting model.

[0149] (2) A novel cross-modal guided contrast loss function is proposed. When using input features for cross-modal reconstruction, the reference information of another modality is used to guide the reconstruction process, and the real and fake audio and video features are distinguished at the feature level.

[0150] Reference Figure 6 This embodiment provides an example flowchart of a video authentication and time positioning method.

[0151] Specifically, this embodiment may include the following steps:

[0152] Step 1. Preprocessing of video files:

[0153] For the video file to be detected (including video and audio), it is first preprocessed to unify the length of audio and video files of different lengths. The steps are as follows:

[0154] 1.1 Padding: First, padding is applied to the input audio and video. The video image is padded to frame_padding = 512 frames, and the audio file is padded to audio_padding = int(frame_padding / fps*sps) sample points, where the video frame rate fps = 25, the audio sampling rate sps = 16000, and the total number of audio samples audio_padding is padded to 327680.

[0155] 1.2 Dimensional Adjustment: For the filled video, it is scaled and resized to a size of (96, 96) and channel adjustment is performed. The final video input is represented as a tensor of shape (3, 512, 96, 96), where 3 is the number of RGB channels, 512 is the number of frames, and 96 is the height and width of the video.

[0156] For the padded audio, a Mel spectrogram is extracted using torchaudio.transforms.MelSpectrogram, with a window size of n_fft = 321 and a stride of step = 160. Finally, an audio input is represented as a tensor of shape (64, 2048), where 64 is the number of Mel filters and 2048 is the number of "frames" (width) of the spectrogram.

[0157] Step 2. Feature Extraction and Alignment:

[0158] For the preprocessed video tensor, MViTv2-Base is used as a video encoder to extract features, resulting in a feature vector with shape (256, 512), where 256 is the video vector dimension and 512 is the number of frames.

[0159] For the preprocessed audio tensor, ViT-Base is used as the audio encoder to extract features, resulting in a feature vector with shape (256, 512), where 256 is the audio vector dimension and 512 is the frame number; at this point, the video and audio features are aligned in the temporal feature dimension.

[0160] Step 3. Cross-modal reconstruction:

[0161] For the audio and video features obtained through the video encoder and audio encoder, cross-prediction reconstruction is performed using A2V and V2A networks. Specifically, both A2V and V2A are implemented using a standard 4-layer transformer to achieve a temporal attention mechanism, assigning different attention to real and tampered segments within a video.

[0162] During the training phase, two classes of data from the LAV-DF dataset are used for training, such as... Figure 7As shown:

[0163] (1) The video is completely authentic, and neither the audio nor the video modality has been tampered with.

[0164] (2) The video was completely tampered with. The audio and video were tampered with synchronously. The specific time segment of the tampering and the original video are provided. In this embodiment, the timestamp provided by LAV-DF is used to locate the corresponding segment in the real video to which the tampered segment corresponds.

[0165] During training, this embodiment uses different loss functions for real audio / video frames and tampered audio / video frames:

[0166] For real audio and video frames, this embodiment uses its own cross-modal real data for cross-reconstruction supervision. That is, it uses real video to predict real audio and real audio to predict real video, with the following loss function:

[0167]

[0168] P a =V2A(IN) v );

[0169] P v =A2V(IN) a );

[0170] Where N represents the number of video frames, P vi IN represents the features of the i-th video frame predicted using the input audio. vi Represents the input video feature frame; P ai IN represents the features of the i-th audio frame predicted using the input video. ai This represents the characteristics of the input real audio frame.

[0171] For simultaneously tampered audio and video, this embodiment uses its own cross-modal fake data and corresponding real data for supervised cross-reconstruction. Specifically, for a known fake audio feature input, this embodiment uses it to predict video features, and then subtracts it from a reference real video frame; the difference is called the Mask. pred_v Another supervision mechanism, Mask, uses the difference between the input fake video and the features of a reference real video; this difference is called the Mask value. ref By minimizing the Mask pred_V With 2x Mask ref_v The difference between them enables the supervision of spatial errors in visual modalities.

[0172] Similarly, given a known fake video feature as input, this embodiment uses it to predict audio features, and then subtracts it from a reference real audio feature; the difference is called the Mask. pred_a Another supervised feature, Mask, uses the difference between the input spoof audio features and a reference real audio feature; this difference is called the Mask. ref_a By minimizing the Mask prrd_a With 2x Mask ref_a The difference between them is used to supervise the spatial error of the auditory modality; the relevant loss function is as follows:

[0173]

[0174] Mask pred_a =P a -Real a ;

[0175]

[0176] Mask ref_a =Fake a -Real a =IN a -Real a ;

[0177] Mask ref_v =Fake v -Real v =IN v -Real v ;

[0178] P a =V2A(IN) v );

[0179] P v =A2V(IN) a );

[0180] Where N represents the number of video frames, P v This refers to the features of a video predicted using the input audio; Fake v =IN v Indicates the tampered video features of the input; P a This refers to the features of the audio predicted using the input video, Fake. a =IN a Indicates the characteristics of the tampered audio input, Real a Indicates the corresponding real audio features, Real v This indicates the corresponding real-world video features.

[0181] During the inference phase, an audio feature is predicted using the input video features, and then the difference between the predicted audio feature and the input video feature is used as the output reconstructed audio error feature. Similarly, a video feature is predicted using the input audio features, and then the difference between the predicted audio feature and the input video feature is used as the output reconstructed video error feature. The relevant expressions are as follows:

[0182] Out a =P a -IN a ;

[0183] Out v =P v -IN v ;

[0184] Out combined =Linear(Out) a Out v );

[0185] Here, Linear represents a linear layer that accepts Out. a and Out v As input, after linear weighting, the output dimension is equal to the Out dimension. a and Out v The same feature vectors. Inference stage, such as... Figure 8 As shown.

[0186] Theoretical Analysis: When training converges, real audio input features tend to predict real video input features, and real video input features tend to predict real audio input features; tampered audio input tends to predict tampered video input plus a mask. ref_v Tampered video input tends to predict tampered audio input plus mask. ref_a ;Right now:

[0187]

[0188]

[0189] Among them, Fake a,i This represents the features of the corresponding tampered audio frames in the dataset, Fake v,i This indicates the characteristics of the corresponding tampered video frames in the data.

[0190] The following expression can be derived from determining the truth or falsehood of audio and video input:

[0191]

[0192] As shown in the above formula, the cross-reconstruction module outputs the feature Out only when neither the audio nor video modalities have been tampered with.a and Out v The values ​​tend to be 0, but if any modality is tampered with, neither value will be 0, thus allowing the distinction between genuine audio and video input features and tampered audio and video input features.

[0193] Step 4. Frame classification:

[0194] The frame classification module is implemented using standard one-dimensional convolution. Based on the features of the current frame, it regresses the probability that the frame has been tampered with.

[0195] Still refer to Figure 6 This embodiment uses three independent frame-level classifiers to classify pure audio tampering, pure visual tampering, and audio or video tampering, respectively. The specific loss function is as follows:

[0196]

[0197] Among them, Lossf a Lossf v Lossf s These represent the loss functions for frame classification using audio features, visual features, and spatial error features, respectively. It is the standard cross-entropy loss function.

[0198] Step 5. Time positioning:

[0199] The temporal localization task consists of three relatively independent modules, which utilize audio features output by the audio encoder, video features output by the video encoder, spatial error features output by the cross-modal reconstructed model, and corresponding frame classification results to perform the temporal localization task. Specifically, the temporal localization module mainly includes a boundary matching candidate generation module and a temporal attention transformer module. The boundary matching candidate generation module generates a tensor of shape [T, N, D, T], and the relevant code is as follows:

[0200] #(B,n_feature_in,temporal_dim)->(B,n_feature_in,sample,max_duration,temporal_dim);

[0201] The loss function is as follows:

[0202]

[0203] Among them, Lossb a It can be used as the boundary matching loss function for audio tampering time localization, visual tampering time localization, and time localization of any modality of tampering, respectively. N is the number of frames in the video, and D is the maximum possible duration (frames) of the tampered segment. Y represents the confidence level of the tampered segment starting from frame i and ending at frame i+j. b a,j j represents the corresponding truth value.

[0204] The beneficial effects of this embodiment include:

[0205] By focusing on cross-reconstruction errors, the model can focus on specific tampered areas, thereby improving the accuracy of counterfeit detection and time localization.

[0206] A novel supervisory loss function for cross-modal reconstruction is proposed. When the current modality is known to be tampered with, the true features and tampered features of the other modality are used as supervisory signals to distinguish the true signals from the tampered signals during reconstruction.

[0207] In summary, this embodiment provides a multimodal Deepfake video authentication and temporal localization method based on audio and video cross-reconstruction. It utilizes cross-modal information to improve the ability to identify and locate audio and video tampering, providing a more powerful tool for the development and application of network security and forensic identification, in order to cope with the ever-evolving Deepfake technology.

[0208] Reference Figure 9 This application also provides a video authentication and time positioning device, which can implement the above-mentioned video authentication and time positioning method. The device includes:

[0209] The feature extraction unit is used to extract features from the video and audio in the video file respectively, and obtain the input video features and input audio features aligned to the time dimension.

[0210] The feature reconstruction unit is used to perform cross-modal cross-reconstruction on the input video features and the input audio features, and to supervise the cross-modal cross-reconstruction steps by utilizing the spatial difference between real and fake features, thereby obtaining spatial error features;

[0211] The feature classification unit is used to determine the probability that the current frame has been tampered with based on the input video features, the input audio features and the spatial error features, and then classify each current frame accordingly.

[0212] A time localization unit is used to perform time localization of the tampered data frame in the video file based on the input video features, the input audio features, the spatial error features, and the probability that the current frame is tampered with; wherein the data frame includes at least one of the current frame corresponding to the input video features or the current frame corresponding to the input audio features.

[0213] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0214] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned video authentication and time positioning method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0215] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0216] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0217] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0218] The memory 1002 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 using the video authentication and time positioning method of the embodiments of this application.

[0219] Input / output interface 1003 is used to implement information input and output;

[0220] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0221] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0222] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0223] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described video authentication and time positioning method.

[0224] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0225] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0226] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0227] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0228] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0229] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0230] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0231] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0232] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0233] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0234] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0235] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0236] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A video authentication and time-based positioning method, characterized in that, The method includes the following steps: Feature extraction is performed on the video and audio in the video file respectively to obtain time-aligned input video features and input audio features; Cross-modal cross-reconstruction is performed on the input video features and the input audio features, and the spatial difference between real and fake features is used to supervise the cross-modal cross-reconstruction steps to obtain spatial error features. The probability of the current frame being tampered with is determined based on the input video features, the input audio features, and the spatial error features. Then, each current frame is classified to obtain the corresponding frame classification result. The tampered data frames in the video file are located temporally based on the input video features, the input audio features, the spatial error features, and the corresponding frame classification results; wherein, the data frames include at least one of the current frame corresponding to the input video features or the current frame corresponding to the input audio features; The step of performing cross-modal cross-reconstruction on the input video features and the input audio features, and supervising the cross-modal cross-reconstruction using the spatial difference between real and fake features, to obtain spatial error features, includes the following steps: The input video features are input into a V2A network, and the V2A network is used to infer reconstruct audio features. The input audio features are then input into an A2V network, and the A2V network is used to infer reconstruct video features. The cross-modal reconstruction module includes both the V2A and A2V networks. Both the A2V and V2A networks are implemented using a 4-layer standard transformer to achieve a temporal attention mechanism, assigning different attention to real and tampered segments within a video. The difference between the input video features and the reconstructed video features is used to obtain the reconstructed video error features; the difference between the input audio features and the reconstructed audio features is used to obtain the reconstructed audio error features. The reconstructed video error features and the reconstructed audio error features are input into a linear layer and linearly weighted to obtain the spatial error features; The method further includes a step of training a cross-modal cross-reconstruction module, which includes the following steps: The V2A network was trained using video features labeled with both real and tampered data; the A2V network was trained using audio features labeled with both real and tampered data. The step of classifying each current frame includes the following steps: Each current frame is classified according to the loss function for frame classification.

2. The video authentication and time positioning method according to claim 1, characterized in that, Before extracting features from the video and audio in the video file to obtain time-aligned input video features and input audio features, the method further includes a preprocessing step for the video file. This preprocessing step includes the following steps: The video and audio are extracted from the video file; Fill in the video and the audio respectively; The size of the filled video is scaled and the channels are adjusted to obtain a video tensor of the first shape; The Mel spectrogram is extracted from the filled audio, and then the Mel spectrogram is represented as an audio tensor of a second shape; The step of extracting features from the video and audio files separately to obtain time-aligned input video features and input audio features includes the following steps: Feature extraction is performed on the video tensor and the audio tensor respectively to obtain time-aligned input video features and input audio features.

3. The video authentication and time positioning method according to claim 1, characterized in that, The step of locating the tampered data frames in the video file based on the input video features, the input audio features, the spatial error features, and the corresponding frame classification results includes the following steps: The temporal localization loss function is used to perform temporal localization on the tampered data frames in the video file based on the input video features, the input audio features, the spatial error features, and the corresponding frame classification results.

4. The video authentication and time positioning method according to any one of claims 1 to 3, characterized in that, The method further includes the following steps: The tampered data frame and its corresponding time location are visualized.

5. A video authentication and time-based positioning device, characterized in that, The device is used to implement the video authentication and time positioning method as described in claim 1, and the device includes: The feature extraction unit is used to extract features from the video and audio in the video file respectively, and obtain time-aligned input video features and input audio features. The feature reconstruction unit is used to perform cross-modal cross-reconstruction on the input video features and the input audio features, and to supervise the cross-modal cross-reconstruction steps by utilizing the spatial difference between real and fake features, thereby obtaining spatial error features; The feature classification unit is used to determine the probability that the current frame has been tampered with based on the input video features, the input audio features and the spatial error features, and then classify each current frame to obtain the corresponding frame classification result. A time localization unit is used to perform time localization of tampered data frames in the video file based on the input video features, the input audio features, the spatial error features, and the corresponding frame classification results; wherein, the data frame includes at least one of the current frame corresponding to the input video features or the current frame corresponding to the input audio features.

6. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the video authentication and time positioning method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the video authentication and time positioning method as described in any one of claims 1 to 4.

Images