Authentication method and device of audio and video data, equipment and storage medium

Abstract

The embodiment of the invention provides an audio and video data authentic identification method and device, equipment and a storage medium, and the method comprises the steps: obtaining a video feature and an audio feature which are aligned according to audio and video data; determining a classification label corresponding to the video feature and a classification label corresponding to the audio feature; according to the video features and the classification labels corresponding to the video features, obtaining video counterfeiting features corresponding to the video data, and according to the audio features and the classification labels corresponding to the audio features, obtaining audio counterfeiting features corresponding to the audio data; and obtaining an authentic identification result for determining whether the audio and video data is tampered or not according to the video forgery feature and the audio forgery feature. According to the embodiment of the invention, through feature alignment and classification labels, video and audio counterfeit features with finer granularity can be extracted, more essential counterfeit features can be mined, counterfeit information in audio and video data can be captured more comprehensively in combination with the counterfeit features of the video and the audio, and the accuracy and robustness of counterfeit identification are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for detecting counterfeit audio and video data, an electronic device, and a storage medium. Background Art

[0002] Deep forgery detection mainly relies on deep learning technology, which trains neural networks to identify forgery features in images and videos, and usually uses neural networks to analyze subtle pixel differences to determine the authenticity of the content. Multimodal deep forgery detection combines information from different data sources, such as text, images, and audio. By integrating these different modal data, the accuracy and robustness of detection are improved. This method can more comprehensively capture the characteristics of forged content, making it effective under a variety of attack methods.

[0003] At present, most similar products use deep learning frameworks to model simple classification tasks for audio and video modal information, analyze and judge independently, and output the results of each modality in a unified manner. Some products will model audio and video modalities simultaneously, fuse audio features and video features, build a classification model, and output the analysis and judgment results.

[0004] However, the judgment results output by pure classification methods are often a simple conclusion, that is, whether the audio and video data has been tampered with. The model is unable to learn the human cognitive level, causing the model to overfit to a certain type of scenario, resulting in excellent results during modeling training but a significant drop in results in practical deployment; and for spatiotemporal information such as video and audio, the classification labels are too coarse and cannot provide fine-grained information, resulting in the model being unable to learn more essential forgery features, resulting in a significant drop in results in practice. Summary of the invention

[0005] The embodiment of the present application provides a method for authenticating audio and video data to solve the problem that the authentication result obtained by the authentication based on pure classification method is too rough.

[0006] Correspondingly, the embodiments of the present application also provide an audio and video data authentication device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problem, the embodiment of the present application discloses a method for detecting counterfeit audio and video data, wherein the audio and video data includes video data and audio data, and the method includes:

[0008] Obtaining feature-aligned video features and audio features according to the audio and video data;

[0009] Determine a classification label corresponding to the video feature and a classification label corresponding to the audio feature;

[0010] Obtaining a video forgery feature corresponding to the video data according to the video feature and the classification label corresponding to the video feature, and obtaining an audio forgery feature corresponding to the audio data according to the audio feature and the classification label corresponding to the audio feature;

[0011] According to the video forgery feature and the audio forgery feature, an authentication result for determining whether the audio and video data has been tampered with is obtained.

[0012] Optionally, obtaining feature-aligned video features and audio features according to the audio and video data includes:

[0013] Build audio and video encoder and feature alignment model;

[0014] Inputting the audio and video data into the audio and video encoder to obtain video coding features and audio coding features;

[0015] The video coding features and the audio coding features are input into the feature alignment model to obtain the video features and the audio features.

[0016] Optionally, determining the classification label corresponding to the video feature and the classification label corresponding to the audio feature includes:

[0017] Build video frame classifier and audio frame classifier;

[0018] The video feature is input into the video frame classifier to obtain a classification label corresponding to the video feature, and the audio feature is input into the audio frame classifier to obtain a classification label corresponding to the audio feature.

[0019] Optionally, obtaining a video forgery feature corresponding to the video data according to the video feature and a classification label corresponding to the video feature, and obtaining an audio forgery feature corresponding to the audio data according to the audio feature and a classification label corresponding to the audio feature, includes:

[0020] Build video tampering recognition model and audio tampering recognition model;

[0021] After fusing the video feature and the classification label corresponding to the video feature, a video fusion feature is obtained, and the video fusion feature is input into the video tampering recognition model to obtain the video forgery feature; and, after fusing the audio feature and the classification label corresponding to the audio feature, an audio fusion feature is obtained, and the audio fusion feature is input into the audio tampering recognition model to obtain the audio forgery feature.

[0022] Optionally, obtaining, according to the video forgery feature and the audio forgery feature, an authentication result for determining whether the audio and video data has been tampered with includes:

[0023] Construct an audio and video feature fusion model;

[0024] Inputting the video forgery feature and the audio forgery feature into the audio-video feature fusion model to obtain an audio-video fusion feature;

[0025] According to the audio and video fusion features, obtaining the tampering probability corresponding to the audio and video data;

[0026] Determining whether the audio and video data has been tampered with according to the tampering probability corresponding to the audio and video data and a preset probability threshold;

[0027] If the audio and video data is tampered with, determining a first authentication result;

[0028] If the audio and video data has not been tampered with, determining a second authentication result;

[0029] The first authentication result or the second authentication result is determined as the authentication result.

[0030] Optionally, the feature alignment model has corresponding model parameters, the feature alignment model includes feature channels for processing the video coding features and the audio coding features, and the feature alignment model is trained by the following steps:

[0031] Acquire first sample data for training the feature alignment model;

[0032] Calculating a feature distance between the video coding feature and the audio coding feature;

[0033] Calculate the loss value corresponding to the feature alignment model according to the first sample data and the feature distance;

[0034] The model parameters corresponding to the feature alignment model are adjusted according to the loss value corresponding to the feature alignment model to complete the training of the feature alignment model.

[0035] Optionally, the video frame classifier and the audio frame classifier respectively have corresponding model parameters, and the video frame classifier and the audio frame classifier are trained through the following steps:

[0036] Acquire second sample data for training the video frame classifier and third sample data for training the audio frame classifier;

[0037] Calculating a first difference value between the classification label corresponding to the video feature and the second sample data, and calculating a second difference value between the classification label corresponding to the audio feature and the third sample data;

[0038] Calculating a loss value corresponding to the video frame classifier according to the second sample data and the first difference value, and calculating a loss value corresponding to the audio frame classifier according to the third sample data and the second difference value;

[0039] Adjust the model parameters corresponding to the video frame classifier according to the loss value corresponding to the video frame classifier to complete the training of the video frame classifier; and adjust the model parameters corresponding to the audio frame classifier according to the loss value corresponding to the audio frame classifier to complete the training of the audio frame classifier.

[0040] The embodiment of the present application further discloses a device for detecting counterfeit audio and video data, wherein the audio and video data includes video data and audio data, and the device includes:

[0041] A feature alignment module, used for obtaining video features and audio features for feature alignment according to the audio and video data;

[0042] A classification label module, used to determine the classification label corresponding to the video feature and the classification label corresponding to the audio feature;

[0043] a forgery feature module, configured to obtain a video forgery feature corresponding to the video data according to the video feature and the classification label corresponding to the video feature, and to obtain an audio forgery feature corresponding to the audio data according to the audio feature and the classification label corresponding to the audio feature;

[0044] The authentication result module is used to obtain an authentication result for determining whether the audio and video data has been tampered with according to the video forgery feature and the audio forgery feature.

[0045] An embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the method for authenticating audio and video data as described in any one of the embodiments of the present application.

[0046] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, a processor executes the method for authenticating audio and video data as described in any one of the embodiments of the present application.

[0047] Compared with the prior art, the embodiments of the present application have the following advantages:

[0048] In an embodiment of the present application, based on the audio and video data, feature-aligned video features and audio features are obtained; the classification labels corresponding to the video features and the classification labels corresponding to the audio features are determined; based on the video features and the classification labels corresponding to the video features, the video forgery features corresponding to the video data are obtained, and, based on the audio features and the classification labels corresponding to the audio features, the audio forgery features corresponding to the audio data are obtained; based on the video forgery features and the audio forgery features, an authentication result for determining whether the audio and video data has been tampered with is obtained.

[0049] The embodiments of the present application can extract more fine-grained video and audio forgery features through feature alignment and determination of classification labels, avoiding the problem of overly coarse classification labels in traditional methods, and can further mine more essential forgery features. In addition, by combining the forgery features of video and audio, it can more comprehensively capture forgery information in audio and video data, thereby improving the accuracy and robustness of forgery detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flowchart of a method for detecting counterfeit audio and video data according to an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of an authentication framework for audio and video data of the present application;

[0052] Figure 3 This is a structural block diagram of an embodiment of an audio and video data authentication device of the present application;

[0053] Figure 4 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0055] Reference Figure 1 , is a flowchart of an embodiment of a method for detecting counterfeit audio and video data of the present application, comprising the following steps:

[0056] Step 101: Obtain feature-aligned video features and audio features according to the audio and video data;

[0057] In the embodiment of the present application, audio and video data is multimodal data, specifically including video data and audio data.

[0058] In actual application scenarios, video data and audio data are usually collected by different acquisition devices (such as cameras and microphones). If these acquisition devices are not strictly synchronized, the video data and audio data may not match in time. In addition, during the transmission of audio and video data, the video data and audio data may pass through different transmission paths or network nodes, and factors such as network delays and bandwidth limitations may cause the video data and audio data to arrive at the destination at different times, resulting in a time mismatch between the video data and audio data. In addition, due to the different processing speeds of video data and audio data, the processing time will also be different, which will also cause a time mismatch between the video data and audio data.

[0059] The temporal mismatch between video data and audio data may affect subsequent authentication, leading to misjudgment or missed judgment. In addition, the inability to fully utilize the complementarity of video and audio features during the authentication process will reduce the accuracy and comprehensiveness of multimodal data analysis, leading to erroneous authentication results.

[0060] Therefore, in order to ensure the efficiency and accuracy of audio and video data processing, in the embodiment of the present application, it is necessary to perform feature alignment on the features extracted from the audio and video data to obtain feature-aligned video features and audio features (audio and video features), which not only ensures the temporal consistency of video features and audio features, but also enhances the ability of multimodal data fusion, thereby improving the accuracy and reliability of authentication.

[0061] Step 102: Determine the classification label corresponding to the video feature and the classification label corresponding to the audio feature;

[0062] In the embodiment of the present application, the video features and audio features extracted from the audio and video data are classified respectively. The classification label indicates the category or state to which the feature belongs, such as "real" or "fake".

[0063] Specifically, the video features and audio features have multiple time frames, and each time frame of the video features and audio features will be classified to obtain the classification labels corresponding to the video features and audio features in each time frame (the classification labels corresponding to the audio and video features respectively).

[0064] By classifying the video features and the audio features, the classification labels corresponding to the video features and the classification labels corresponding to the audio features are obtained. In the related art, the classification labels are often used as a counterfeit detection result. However, if the classification labels are directly used as the counterfeit detection result, the counterfeit detection result will appear too rough, resulting in low accuracy and reliability of the counterfeit detection.

[0065] Therefore, after obtaining the classification label, the embodiment of the present application also needs to further determine more essential forgery features from the fine-grained information based on the audio and video features and the classification labels corresponding to the audio and video features, and further determine the authentication result based on the forgery features.

[0066] Step 103: obtaining a video forgery feature corresponding to the video data according to the video feature and the classification label corresponding to the video feature, and obtaining an audio forgery feature corresponding to the audio data according to the audio feature and the classification label corresponding to the audio feature;

[0067] In the embodiment of the present application, video features and their corresponding classification labels are used to generate forged features of video data (video forged features). Similarly, audio features and their corresponding classification labels are used to generate forged features of audio data (audio forged features). Video forged features and audio forged features are the more essential forged features to be obtained in the embodiment of the present application.

[0068] Specifically, the video forgery features include video forgery spatial area labels and video forgery timing labels, and the audio forgery features refer to audio forgery timing labels.

[0069] By analyzing the audio and video features and the classification labels corresponding to the audio and video features and generating forgery features, we can more carefully analyze the anomalies or signs of forgery in the video data and audio data, thereby improving the ability to detect data tampering and enhancing the accuracy and reliability of authentication.

[0070] Step 104: Obtain an authentication result for determining whether the audio and video data has been tampered with based on the video forgery feature and the audio forgery feature.

[0071] In the embodiment of the present application, after obtaining the forgery features under different modes, it is necessary to combine the forgery features under different modes for analysis. Specifically, the video forgery features and the audio forgery features are combined for comprehensive analysis, and finally it can be determined whether the audio and video data has been tampered with.

[0072] By combining the forgery features under different modalities, we can not only make full use of the complementarity of multimodal data and improve the accuracy of forgery detection, but also more comprehensively evaluate the authenticity of the data and reduce the possible misjudgment of a single modality.

[0073] The embodiments of the present application can extract more fine-grained video and audio forgery features through feature alignment and determination of classification labels, avoiding the problem of overly coarse classification labels in traditional methods, and can further mine more essential forgery features. In addition, by combining the forgery features of video and audio, it can more comprehensively capture forgery information in audio and video data, thereby improving the accuracy and robustness of forgery detection.

[0074] In one embodiment of the present application, the step 101 of obtaining feature-aligned video features and audio features according to the audio and video data includes:

[0075] Build audio and video encoder and feature alignment model;

[0076] Inputting the audio and video data into the audio and video encoder to obtain video coding features and audio coding features;

[0077] The video coding features and the audio coding features are input into the feature alignment model to obtain the video features and the audio features.

[0078] In the embodiment of the present application, it is first necessary to construct an audio and video encoder for extracting original features from audio and video data, and a feature alignment model for aligning features output by the audio and video encoder. The audio and video encoder specifically includes a video encoder for extracting original features from video data and an audio encoder for extracting original features from audio data.

[0079] Since the original features in the audio and video data may contain a large amount of redundant information, the audio and video encoder in the embodiment of the present application will also perform feature encoding on the original features in the audio and video data. Feature encoding can reduce the dimension of the features, remove redundant information, avoid the influence of redundant information on counterfeit identification, and improve the accuracy and reliability of counterfeit identification. In addition, reducing the dimension of the features can reduce the computational complexity, improve processing efficiency, and avoid overfitting.

[0080] Specifically, the video data is input into a video encoder, which processes the video data, extracts original features from the video data, and performs feature encoding on the extracted original video features to obtain video encoding features. Similarly, the audio data is input into an audio encoder, which processes the audio data, extracts original features from the audio data, and performs feature encoding on the extracted original audio features to obtain audio encoding features.

[0081] For example, in a video encoder, methods such as convolutional neural network (CNN) and optical flow method can be used to extract original features in video data. Convolutional neural network is a deep learning model. Through multi-layer convolution and pooling operations, convolutional neural network can extract features such as texture, color, and edge in video frames, and is particularly suitable for feature extraction of image and video data; optical flow method is used to analyze motion information between video frames. By calculating the pixel displacement between adjacent frames, optical flow method can extract motion features in the video.

[0082] As another example, in an audio encoder, methods such as Mel-frequency cepstral coefficients (MFCC) and short-time Fourier transform (STFT) can be used to extract features from audio data. Mel-frequency cepstral coefficients are a commonly used audio feature extraction method. By converting an audio signal into Mel-frequency cepstral coefficients, the Mel-frequency cepstral coefficients can extract the spectral features of the audio signal, which is particularly suitable for speech recognition and audio classification; short-time Fourier transform is a time-frequency analysis method used to analyze the time-frequency characteristics of an audio signal. By dividing the audio signal into short-time windows and performing Fourier transform on each window, the time-frequency features of the audio signal can be extracted.

[0083] After obtaining the video coding features and audio coding features, it is also necessary to achieve temporal consistency of the video coding features and audio coding features. Therefore, the video coding features and audio coding features are input into a feature alignment model to output temporally consistent video features and coding features.

[0084] The video data in the embodiment of the present application refers to a video sequence having multiple video frames, and the audio data refers to an audio sequence having multiple audio frames.

[0085] A video frame is a single image in a video sequence, which is usually played continuously at a fixed time interval (such as 24 frames per second, 30 frames per second, etc.) to form a dynamic video. A video frame contains visual information in the video, such as the texture, color, motion, etc. of the image.

[0086] An audio frame is a segment of an audio signal in time, usually divided at fixed time intervals (such as every 20 milliseconds, 30 milliseconds, etc.). An audio frame contains the time-frequency information of the audio signal, such as spectrum, pitch, volume, etc.

[0087] Therefore, in the embodiment of the present application, feature extraction, feature encoding and feature alignment are performed specifically for each video frame in the video data and each audio frame in the audio data, that is, the final video features and audio features are obtained for each frame.

[0088] In the embodiment of the present application, original features in the audio and video data are extracted by an audio and video encoder, and feature encoding processing is performed on the original features to obtain video encoding features and audio encoding features. The video encoding features and audio encoding features are then feature aligned by a feature alignment model to obtain feature-aligned video features and audio features, which not only ensures the temporal consistency of the video features and audio features, but also enhances the ability of multimodal data fusion, thereby improving the accuracy and reliability of authentication.

[0089] In one embodiment of the present application, the step 102 of determining the classification label corresponding to the video feature and the classification label corresponding to the audio feature includes:

[0090] Build video frame classifier and audio frame classifier;

[0091] The video feature is input into the video frame classifier to obtain a classification label corresponding to the video feature, and the audio feature is input into the audio frame classifier to obtain a classification label corresponding to the audio feature.

[0092] In the embodiment of the present application, a video frame classifier for classifying video features and an audio frame classifier for classifying audio features are constructed.

[0093] The video frame classifier and the audio frame classifier can identify features of different categories. Therefore, the video features of each video frame can be input into the video frame classifier, and the classification label corresponding to the video feature can be output. Similarly, the audio features of each audio frame can be input into the audio frame classifier, and the classification label corresponding to the audio feature can be output.

[0094] In the embodiment of the present application, the classification labels corresponding to the video features and the classification labels corresponding to the audio features are not directly output as the authentication results, but need to be combined with the video features and the audio features to further explore more essential counterfeit features.

[0095] In one embodiment of the present application, the step 103, obtaining the video forgery feature corresponding to the video data according to the video feature and the classification label corresponding to the video feature, and obtaining the audio forgery feature corresponding to the audio data according to the audio feature and the classification label corresponding to the audio feature, includes:

[0096] Build video tampering recognition model and audio tampering recognition model;

[0097] After fusing the video feature and the classification label corresponding to the video feature, a video fusion feature is obtained, and the video fusion feature is input into the video tampering recognition model to obtain the video forgery feature; and, after fusing the audio feature and the classification label corresponding to the audio feature, an audio fusion feature is obtained, and the audio fusion feature is input into the audio tampering recognition model to obtain the audio forgery feature.

[0098] In an embodiment of the present application, a video tampering recognition model for identifying whether there are signs of tampering or forgery (forgery features) in video data and an audio tampering recognition model for identifying whether there are signs of tampering or forgery in audio data are constructed.

[0099] In actual application scenarios, individual video features or audio features may not be sufficient to fully reflect the authenticity of the data. Therefore, in an embodiment of the present application, the representation capability of features is enhanced by fusing features and classification labels, thereby discovering more essential counterfeit features and improving the accuracy of tampering identification.

[0100] Specifically, the video features and the corresponding classification labels are fused to obtain video fusion features, which are then input into the video tampering recognition model, and the video tampering recognition model outputs video forgery features. Similarly, the audio features and the corresponding classification labels are fused to obtain audio fusion features, which are then input into the audio tampering recognition model, and the audio tampering recognition model outputs audio forgery features.

[0101] Therefore, the fused video fusion features and audio fusion features are more stable, contain more information, can describe the data more comprehensively, can reduce the misjudgment caused by the insufficiency of a single feature, and improve the reliability of tampering identification.

[0102] In the embodiment of the present application, the features and classification labels can be fused by feature fusion methods such as feature splicing, feature weighting or feature cascading. For example, the video features and classification labels are spliced ​​together to form video fusion features; or the audio features and classification labels are weighted and fused to obtain audio fusion features.

[0103] Furthermore, the video tampering recognition model and the audio tampering recognition model in the embodiments of the present application can be implemented by deep learning models (such as convolutional neural networks, recurrent neural networks) or traditional machine learning models (such as support vector machines, random forests) and other models. Exemplarily, a convolutional neural network is used to analyze video fusion features to identify signs of tampering in the video; a recurrent neural network is used to analyze audio fusion features to identify signs of tampering in the audio.

[0104] The embodiments of the present application construct a video tampering recognition model and an audio tampering recognition model, and fuse video features and classification labels, audio features and classification labels, so as to more accurately extract forgery features in video and audio data, improve the accuracy of forgery feature extraction and the ability to identify forgeries, and further improve the accuracy and reliability of audio and video data identification.

[0105] In one embodiment of the present application, the step 104, obtaining a forgery detection result for determining whether the audio and video data has been tampered with according to the video forgery feature and the audio forgery feature, includes:

[0106] Construct an audio and video feature fusion model;

[0107] Inputting the video forgery feature and the audio forgery feature into the audio-video feature fusion model to obtain an audio-video fusion feature;

[0108] According to the audio and video fusion features, obtaining the tampering probability corresponding to the audio and video data;

[0109] Determining whether the audio and video data has been tampered with according to the tampering probability corresponding to the audio and video data and a preset probability threshold;

[0110] If the audio and video data is tampered with, determining a first authentication result;

[0111] If the audio and video data has not been tampered with, determining a second authentication result;

[0112] The first authentication result or the second authentication result is determined as the authentication result.

[0113] In the embodiment of the present application, an audio-video feature fusion model for fusing video forgery features and audio forgery features is constructed. Specifically, the video forgery features and audio forgery features are input into the audio-video feature fusion model, and the audio-video feature fusion model outputs the audio-video fusion features.

[0114] The audio and video fusion features will then be analyzed, the tampering probability corresponding to the audio and video data will be calculated, and the calculated tampering probability will be compared with the preset probability threshold to determine whether the audio and video data has been tampered with.

[0115] If the tampering probability exceeds the preset threshold, the audio and video data is determined to be tampered with, and the first authentication result (such as "forged") is output; if the tampering probability does not exceed the preset threshold, the audio and video data is determined to be not tampered with, and the second authentication result (such as "true") is output.

[0116] Finally, the final authentication result is determined based on the tampering judgment result.

[0117] Exemplarily, a classification model (such as support vector machine, random forest, deep learning model, etc.) can be used to classify the audio and video fusion features and output the tampering probability, or a regression model (such as linear regression, logistic regression, deep learning model, etc.) can be used to perform regression analysis on the audio and video fusion features and output the tampering probability.

[0118] The tampering probability in the embodiment of the present application is a numerical value used to indicate the possibility that the audio and video data has been tampered with, and is typically a numerical value between 0 and 1, with a larger numerical value indicating a higher possibility that the audio and video data has been tampered with.

[0119] For example, assuming that the preset probability threshold is 0.5, if the tampering probability is 0.8, it means that the possibility of the audio and video data being tampered is very high, and at this time, the first authentication result is determined, that is, the audio and video data is tampered; if the tampering probability is 0.3, it means that the possibility of the audio and video data being tampered is low, and at this time, the second authentication result is determined, that is, the audio and video data is not tampered.

[0120] By fusing video forgery features and audio forgery features, the embodiments of the present application can make full use of the complementarity of multimodal data and improve the accuracy of forgery detection. Comprehensive analysis of video forgery features, audio forgery features, and audio and video fusion features can more comprehensively evaluate the authenticity of the data and reduce possible misjudgments of a single modality. By calculating the probability of tampering, the possibility of data tampering can be quantified to improve the reliability of the forgery detection results. Using a preset probability threshold for judgment can avoid misjudgments caused by a single feature and improve the robustness of forgery detection.

[0121] In one embodiment of the present application, the feature alignment model has corresponding model parameters, the feature alignment model includes a feature channel for processing the video coding feature and the audio coding feature, and the feature alignment model is trained by the following steps:

[0122] Acquire first sample data for training the feature alignment model;

[0123] Calculating a feature distance between the video coding feature and the audio coding feature;

[0124] Calculate the loss value corresponding to the feature alignment model according to the first sample data and the feature distance;

[0125] The model parameters corresponding to the feature alignment model are adjusted according to the loss value corresponding to the feature alignment model to complete the training of the feature alignment model.

[0126] In the embodiment of the present application, it is necessary to obtain first sample data for training the feature alignment model, and the first sample data includes untampered first sample data and tampered first sample data. In each first sample data, the video data and the audio data are divided into a plurality of frame segments, and the number of the frame segments is called the number of first sample frame segments.

[0127] The feature alignment model includes multiple feature channels for processing video coding features and audio coding features. Each feature channel is responsible for processing a specific type of feature and outputting the processed features.

[0128] Specifically, it is necessary to supervise the video features and audio features after feature alignment and calculate the loss value L corresponding to the feature alignment model. c After that, based on L cTrain the feature alignment model, and the loss value L corresponding to the feature alignment model c Specifically, it is calculated by the following formula (1):

[0129]

[0130] Where N represents the number of first sample data, C f represents the number of feature channels in the feature alignment model, T represents the number of first sample frame segments, d i represents the feature distance, y i Indicates the true label value of whether the first sample data has been tampered with, δ represents the nearest distance hyperparameter between the audio and video features in the tampered first sample data, and max represents obtaining the maximum value.

[0131] If the first sample data is tampered with, then y i is 0, if the first sample data has not been tampered with, then y i is 1. δ is a parameter used to measure the distance between the audio and video features of the tampered samples. This parameter is usually used to determine the similarity or difference between the tampered samples, thereby helping the model to identify and classify the tampered samples.

[0132] Specifically, the characteristic distance d i It refers to the Euclidean distance between video features and audio features, which is calculated by the following formula (2):

[0133] d i =‖F vi -F ai ‖ 2 (2)

[0134] Among them, F vi represents the video feature, F ai Represents audio features.

[0135] The parameters of the feature alignment model are adjusted through optimization algorithms (such as gradient descent, Adam, etc.) to minimize the loss value, thereby completing the model training.

[0136] In the embodiment of the present application, the loss value L corresponding to the feature alignment model is calculated. c After that, according to L c The model parameters corresponding to the feature alignment model are adjusted to minimize the loss value, so that the adjusted feature alignment model can more accurately align video features and audio features, improve the temporal consistency of video and audio features, and reduce time offset and mismatch problems. Therefore, the accuracy and reliability of audio and video data authentication can be improved based on the adjusted feature alignment model.

[0137] In one embodiment of the present application, the video frame classifier and the audio frame classifier respectively have corresponding model parameters, and the video frame classifier and the audio frame classifier are trained through the following steps:

[0138] Acquire second sample data for training the video frame classifier and third sample data for training the audio frame classifier;

[0139] Calculating a first difference value between the classification label corresponding to the video feature and the second sample data, and calculating a second difference value between the classification label corresponding to the audio feature and the third sample data;

[0140] Calculating a loss value corresponding to the video frame classifier according to the second sample data and the first difference value, and calculating a loss value corresponding to the audio frame classifier according to the third sample data and the second difference value;

[0141] Adjust the model parameters corresponding to the video frame classifier according to the loss value corresponding to the video frame classifier to complete the training of the video frame classifier; and adjust the model parameters corresponding to the audio frame classifier according to the loss value corresponding to the audio frame classifier to complete the training of the audio frame classifier.

[0142] In the embodiment of the present application, the second sample data is a video data sample for training a video frame classifier, which generally includes video features and corresponding true classification labels. The third sample data is an audio data sample for training an audio frame classifier, which generally includes audio features and corresponding true classification labels.

[0143] Each second sample data and third sample data is also divided into a plurality of frame segments, and the number of the frame segments is referred to as the second sample frame segment number or the third sample frame segment number.

[0144] The first difference value is a measure for measuring the difference between the classification label corresponding to the video feature and the real classification label in the second sample data. The second difference value is a measure for measuring the difference between the classification label corresponding to the audio feature and the real classification label in the third sample data. Both the first difference value and the second difference value are obtained by the cross entropy function.

[0145] The loss value corresponding to the video frame classifier is an indicator used to measure the difference between the predicted result of the video frame classifier and the actual result. The loss value corresponding to the audio frame classifier is an indicator used to measure the difference between the predicted result of the audio frame classifier and the actual result.

[0146] The parameters of the video frame classifier and the audio frame classifier can be adjusted through optimization algorithms (such as gradient descent, Adam, etc.) to minimize the loss value, thereby completing the training of the model.

[0147] Specifically, the true classification label corresponding to the second sample data and the true classification label corresponding to the third sample data are determined by the following formula (3):

[0148] Y m =α m Y a +(1-α m )Y v (3)

[0149] Among them, m represents the modality corresponding to the sample data, m∈{a,v}, a belongs to the video modality, v belongs to the audio modality, α m represents an adjustable parameter, Y a Represents the true classification label corresponding to the video data, Y v Represents the true classification label corresponding to the audio data.

[0150] At this time, it is necessary to adjust α m The value of Y m , if α m Adjust to 1, then Y m is the true classification label corresponding to the second sample data. If α m Adjust to 0, then Y m is the true classification label corresponding to the third sample data.

[0151] The first difference value and the second difference value are calculated by the following formula (4):

[0152]

[0153] in, represents the first difference value or the second difference value, which is determined by the mode corresponding to m, Y mij Y m , which is determined by the above formula (3), Indicates the classification label corresponding to the video feature or the classification label corresponding to the audio feature.

[0154] The loss value corresponding to the video frame classifier and the loss value corresponding to the audio frame classifier are calculated by the following formula (5):

[0155]

[0156] Among them, L f It represents the loss value corresponding to the video frame classifier or the loss value corresponding to the audio frame classifier, which is also determined by the mode corresponding to m. N represents the number of second sample data or third sample data, and T represents the number of second sample frame segments or the number of third sample frame segments.

[0157] The embodiment of the present application can quantify the classification error by calculating the difference value between the classification label corresponding to the video feature and the audio feature and the sample data, and help the model (video frame classifier and audio frame classifier) ​​to better classify. By optimizing the loss value and adjusting the model parameters, the video frame classifier and the audio frame classifier can be classified more accurately. By training the video frame classifier and the audio frame classifier, the reliability of the classification result can be enhanced and the classification error can be reduced. By training the video frame classifier and the audio frame classifier respectively, the complementarity of multimodal data can be fully utilized to improve the accuracy of classification.

[0158] In one embodiment of the present application, it is also necessary to train the video tampering recognition model and the audio tampering recognition model respectively. The video tampering recognition model and the audio tampering recognition model respectively have corresponding model parameters, and the video tampering recognition model and the audio tampering recognition model are trained by the following steps:

[0159] Acquire fourth sample data for training a video tampering recognition model and fifth sample data for training an audio tampering recognition model;

[0160] Calculating a loss value corresponding to a video tampering recognition model based on the fourth sample data and the video forgery feature, and calculating a loss value corresponding to an audio tampering recognition model based on the fifth sample data and the audio forgery feature;

[0161] Adjust the model parameters corresponding to the video tampering recognition model according to the loss value corresponding to the video tampering recognition model to complete the training of the video tampering recognition model; and adjust the model parameters corresponding to the audio tampering recognition model according to the loss value corresponding to the audio tampering recognition model to complete the training of the audio tampering recognition model.

[0162] In the embodiment of the present application, the fourth sample data is the forged feature corresponding to the video sample data used to train the video tampering recognition model. The fifth sample data is the forged feature corresponding to the audio sample data used to train the audio tampering recognition model.

[0163] Each of the fourth sample data and the fifth sample data is also divided into a plurality of frame segments, and the number of the frame segments is referred to as the fourth sample frame segment number or the fifth sample frame segment number.

[0164] The loss value corresponding to the video tampering recognition model is an indicator used to measure the difference between the predicted results of the video tampering recognition model and the actual results. The loss value corresponding to the audio tampering recognition model is an indicator used to measure the difference between the predicted results of the audio tampering recognition model and the actual results.

[0165] The parameters of the video tampering recognition model and the audio tampering recognition model are adjusted through optimization algorithms (such as gradient descent, Adam, etc.) to minimize the loss value, thereby completing the training of the model.

[0166] Specifically, the loss value corresponding to the video tampering recognition model and the loss value corresponding to the audio tampering recognition model are calculated by the following formula (6):

[0167]

[0168] Among them, L b represents the loss value corresponding to the video tampering recognition model or the loss value corresponding to the audio tampering recognition model, which is specifically determined by the mode corresponding to m, N represents the number of the fourth sample data or the fifth sample data, D represents the number of all frame segments to be selected in the fourth sample data or the fifth sample data, T represents the number of the fourth sample frame segments or the number of the fifth sample frame segments, M ijk Indicates the forged features corresponding to the video sample data or the forged features corresponding to the audio sample data, It represents the video forgery feature or audio forgery feature, which is also determined by the mode corresponding to m.

[0169] By calculating the difference between the video forgery features and audio forgery features and the sample data, the tampering identification error can be quantified to help the model better identify tampering. By optimizing the loss value and adjusting the model parameters, the video tampering identification model and the audio tampering identification model can more accurately identify tampering. By training the video tampering identification model and the audio tampering identification model, the reliability of the tampering identification results can be enhanced and the tampering identification error can be reduced. By training the video tampering identification model and the audio tampering identification model separately, the complementarity of multimodal data can be fully utilized to improve the accuracy of tampering identification.

[0170] In the embodiment of the present application, the audio and video feature fusion model can also be supervised by the above formula (6) to obtain the loss value corresponding to the audio and video feature fusion model, so as to adjust the model parameters corresponding to the audio and video feature fusion model.

[0171] The embodiment of the present application optimizes the loss value and adjusts the model parameters so that the audio and video feature fusion model can more accurately fuse video and audio features, improve the accuracy of audio and video feature fusion, reduce feature fusion errors, improve the accuracy of multimodal data analysis, and further improve the accuracy and reliability of audio and video data authentication.

[0172] Reference Figure 2, is a schematic diagram of an audio and video data authentication framework of the present application, including multiple modules such as video data and audio data input, feature encoding, feature alignment, feature classification, tampering identification, and feature fusion. Each module has a corresponding supervision mechanism for calculating the loss value and adjusting the model parameters to optimize the performance of the model. Through these modules and supervision mechanisms, feature extraction, alignment, classification, feature fusion, and tampering identification of audio and video data can be achieved, thereby improving the accuracy and reliability of audio and video data authentication.

[0173] Specifically, the video encoder performs feature encoding on the input video data to obtain video encoding features, the audio encoder performs feature encoding on the input audio data to obtain audio encoding features, and the video encoding features and audio encoding features are input into the feature alignment model to obtain feature-aligned video features and audio features. The feature alignment model is supervised based on the feature-aligned video features and audio features, the loss value corresponding to the model is calculated, and the corresponding model parameters are adjusted.

[0174] The input video features are classified by the video frame classifier to obtain the classification labels corresponding to the video features. The input audio features are classified by the audio frame classifier to obtain the classification labels corresponding to the audio features. The video frame classifier is supervised according to the classification labels corresponding to the video features, the loss value corresponding to the video frame classifier is calculated, and the corresponding model parameters are adjusted; the audio frame classifier is supervised according to the classification labels corresponding to the audio features, the loss value corresponding to the audio frame classifier is calculated, and the corresponding model parameters are adjusted.

[0175] According to the video features and the classification labels corresponding to the video features, identify whether the video data is tampered with. According to the audio features and the classification labels corresponding to the audio features, identify whether the audio data is tampered with. Supervise the video tampering identification model and the audio tampering identification model respectively, calculate the loss value corresponding to the model, and adjust the model parameters.

[0176] It is also necessary to fuse the video forgery features and the audio forgery features through the audio and video feature fusion model to obtain the audio and video fusion features, supervise the audio and video feature fusion model, calculate the loss value corresponding to the model, and adjust the model parameters.

[0177] The embodiments of the present application can extract more fine-grained video and audio forgery features through feature alignment and determination of classification labels, avoiding the problem of overly coarse classification labels in traditional methods, and can further mine more essential forgery features. In addition, by combining the forgery features of video and audio, it can more comprehensively capture forgery information in audio and video data, thereby improving the accuracy and robustness of forgery detection.

[0178] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0179] On the basis of the above embodiments, this embodiment further provides a device for detecting counterfeit audio and video data, which is applied to electronic devices such as terminal devices and servers.

[0180] Reference Figure 3 , shows a structural block diagram of an embodiment of an audio and video data authentication device of the present application, which may specifically include the following modules:

[0181] A feature alignment module 301 is used to obtain video features and audio features for feature alignment according to the audio and video data;

[0182] A classification label module 302, used to determine a classification label corresponding to the video feature and a classification label corresponding to the audio feature;

[0183] a forgery feature module 303, configured to obtain a video forgery feature corresponding to the video data according to the video feature and the classification label corresponding to the video feature, and to obtain an audio forgery feature corresponding to the audio data according to the audio feature and the classification label corresponding to the audio feature;

[0184] The authentication result module 304 is used to obtain an authentication result for determining whether the audio and video data has been tampered with according to the video forgery feature and the audio forgery feature.

[0185] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0186] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0187] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Figure 4 An exemplary apparatus 400 that may be used to implement various embodiments described in this application is schematically illustrated.

[0188] For one embodiment, Figure 4 An exemplary apparatus 400 is shown having one or more processors 402, a control module (chip set) 404 coupled to at least one of the processor(s) 402, a memory 406 coupled to the control module 404, a non-volatile memory (NVM) / storage device 408 coupled to the control module 404, one or more input / output devices 410 coupled to the control module 404, and a network interface 412 coupled to the control module 404.

[0189] The processor 402 may include one or more single-core or multi-core processors, and the processor 402 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 400 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0190] In some embodiments, the apparatus 400 may include one or more computer-readable media (e.g., memory 406 or NVM / storage device 408) having instructions 414 and one or more processors 402 combined with the one or more computer-readable media and configured to execute the instructions 414 to implement a module to perform the actions described in the present disclosure.

[0191] For one embodiment, control module 404 may include any suitable interface controller to provide any suitable interface to at least one of processor(s) 402 and / or any suitable device or component in communication with control module 404 .

[0192] The control module 404 may include a memory controller module to provide an interface to the memory 406. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0193] The memory 406 may be used, for example, to load and store data and / or instructions 414 for the device 400. For one embodiment, the memory 406 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 406 may include a double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0194] For one embodiment, control module 404 may include one or more input / output controllers to provide interfaces to NVM / storage device 408 and input / output device(s) 410 .

[0195] For example, NVM / storage 408 may be used to store data and / or instructions 414. NVM / storage 408 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0196] NVM / storage device 408 may include storage resources that are physically part of the device on which apparatus 400 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 408 may be accessed via input / output device(s) 410 over a network.

[0197] (One or more) input / output devices 410 may provide an interface for the apparatus 400 to communicate with any other appropriate device, and the input / output device 410 may include a communication component, an audio component, a sensor component, etc. The network interface 412 may provide an interface for the apparatus 400 to communicate through one or more networks, and the apparatus 400 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0198] For one embodiment, at least one of the processor(s) 402 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 404. For one embodiment, at least one of the processor(s) 402 may be packaged together with the logic of one or more controllers of the control module 404 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 402 may be integrated on the same die with the logic of one or more controllers of the control module 404. For one embodiment, at least one of the processor(s) 402 may be integrated on the same die with the logic of one or more controllers of the control module 404 to form a system-on-chip (SoC).

[0199] In various embodiments, the device 400 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 400 may have more or fewer components and / or different architectures. For example, in some embodiments, the device 400 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0200] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0201] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0202] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0203] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable audio and video data authentication terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable audio and video data authentication terminal device generate instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0204] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable audio and video data authentication terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0205] These computer program instructions can also be loaded onto a computer or other programmable audio and video data authentication terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable terminal device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0206] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0207] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0208] The above is a detailed introduction to a method and device for detecting counterfeit audio and video data, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for detecting counterfeit audio and video data, characterized in that: The audio and video data includes video data and audio data, and the method includes: Obtaining feature-aligned video features and audio features according to the audio and video data; Determine a classification label corresponding to the video feature and a classification label corresponding to the audio feature; Obtaining a video forgery feature corresponding to the video data according to the video feature and the classification label corresponding to the video feature, and obtaining an audio forgery feature corresponding to the audio data according to the audio feature and the classification label corresponding to the audio feature; According to the video forgery feature and the audio forgery feature, an authentication result for determining whether the audio and video data has been tampered with is obtained.

2. The method according to claim 1, characterized in that The step of obtaining the video features and audio features of feature alignment according to the audio and video data includes: Build audio and video encoder and feature alignment model; Inputting the audio and video data into the audio and video encoder to obtain video coding features and audio coding features; The video coding features and the audio coding features are input into the feature alignment model to obtain the video features and the audio features.

3. The method according to claim 1, characterized in that The determining the classification label corresponding to the video feature and the classification label corresponding to the audio feature includes: Build video frame classifier and audio frame classifier; The video feature is input into the video frame classifier to obtain a classification label corresponding to the video feature, and the audio feature is input into the audio frame classifier to obtain a classification label corresponding to the audio feature.

4. The method according to claim 1, characterized in that: The step of obtaining a video forgery feature corresponding to the video data according to the video feature and the classification label corresponding to the video feature, and obtaining an audio forgery feature corresponding to the audio data according to the audio feature and the classification label corresponding to the audio feature, comprises: Build video tampering recognition model and audio tampering recognition model; After fusing the video feature and the classification label corresponding to the video feature, a video fusion feature is obtained, and the video fusion feature is input into the video tampering recognition model to obtain the video forgery feature; and, after fusing the audio feature and the classification label corresponding to the audio feature, an audio fusion feature is obtained, and the audio fusion feature is input into the audio tampering recognition model to obtain the audio forgery feature.

5. The method according to claim 1, characterized in that The obtaining, according to the video forgery feature and the audio forgery feature, a forgery detection result for determining whether the audio and video data has been tampered with includes: Construct an audio and video feature fusion model; Inputting the video forgery feature and the audio forgery feature into the audio-video feature fusion model to obtain an audio-video fusion feature; According to the audio and video fusion features, obtaining the tampering probability corresponding to the audio and video data; Determining whether the audio and video data has been tampered with according to the tampering probability corresponding to the audio and video data and a preset probability threshold; If the audio and video data is tampered with, determining a first authentication result; If the audio and video data has not been tampered with, determining a second authentication result; The first authentication result or the second authentication result is determined as the authentication result.

6. The method according to claim 2, characterized in that The feature alignment model has corresponding model parameters, the feature alignment model includes feature channels for processing the video coding features and the audio coding features, and the feature alignment model is trained by the following steps: Acquire first sample data for training the feature alignment model; Calculating a feature distance between the video coding feature and the audio coding feature; Calculate the loss value corresponding to the feature alignment model according to the first sample data and the feature distance; The model parameters corresponding to the feature alignment model are adjusted according to the loss value corresponding to the feature alignment model to complete the training of the feature alignment model.

7. The method according to claim 3, characterized in that The video frame classifier and the audio frame classifier respectively have corresponding model parameters, and the video frame classifier and the audio frame classifier are trained by the following steps: Acquire second sample data for training the video frame classifier and third sample data for training the audio frame classifier; Calculating a first difference value between the classification label corresponding to the video feature and the second sample data, and calculating a second difference value between the classification label corresponding to the audio feature and the third sample data; Calculating a loss value corresponding to the video frame classifier according to the second sample data and the first difference value, and calculating a loss value corresponding to the audio frame classifier according to the third sample data and the second difference value; Adjust the model parameters corresponding to the video frame classifier according to the loss value corresponding to the video frame classifier to complete the training of the video frame classifier; and adjust the model parameters corresponding to the audio frame classifier according to the loss value corresponding to the audio frame classifier to complete the training of the audio frame classifier.

8. A device for detecting counterfeit audio and video data, characterized in that: The audio and video data includes video data and audio data, and the device includes: A feature alignment module, used for obtaining video features and audio features for feature alignment according to the audio and video data; A classification label module, used to determine the classification label corresponding to the video feature and the classification label corresponding to the audio feature; a forgery feature module, configured to obtain a video forgery feature corresponding to the video data according to the video feature and the classification label corresponding to the video feature, and to obtain an audio forgery feature corresponding to the audio data according to the audio feature and the classification label corresponding to the audio feature; The authentication result module is used to obtain an authentication result for determining whether the audio and video data has been tampered with according to the video forgery feature and the audio forgery feature.

9. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to execute the method for authenticating audio and video data as claimed in any one of claims 1 to 7.

10. One or more machine-readable media having executable codes stored thereon, which, when executed, enable a processor to execute the method for authenticating audio and video data according to any one of claims 1 to 7.

Images