An audio and video data authentication method, device, equipment and storage medium

Abstract

The embodiment of the application provides a kind of audio and video data's method, device and equipment and storage medium of authenticating, according to audio and video data, obtain the video feature and audio feature of feature alignment;Determine the classification label corresponding to video feature and the classification label corresponding to audio feature;According to video feature and the classification label corresponding to video feature, obtain the video data corresponding video forgery feature, and, according to audio feature and the classification label corresponding to audio feature, obtain the audio data corresponding audio forgery feature;According to video forgery feature and audio forgery feature, obtain the authentication result for determining whether audio and video data is tampered with.The embodiment of the application can extract more fine-grained video and audio forgery features by feature alignment and classification label, can mine more essential forgery features, and combined with video and audio forgery features, can more comprehensively capture the forgery information in audio and video data, improve the accuracy and robustness of authentication.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a method and device for identifying audio and video data, an electronic device and a storage medium. BACKGROUND

[0002] Deep fake detection mainly relies on deep learning techniques, which identify fake features in images and videos by training neural networks. Typically, neural networks are used to analyze subtle pixel differences to determine the authenticity of the content. Multi-modal deep fake detection combines information from different data sources, such as text, images and audio, by integrating these different modalities of data to improve the accuracy and robustness of detection. This method can more comprehensively capture the features of fake content, making it effective under various attack methods.

[0003] Most similar products currently use deep learning frameworks to model simple classification tasks for audio and video modalities, analyze and distinguish independently, and output unified results based on the analysis and judgment of each modality. Some products model audio and video modalities simultaneously, fuse audio and video features to build a classification model, and output analysis and judgment results.

[0004] However, the analysis and judgment results output by the pure classification-based method are often a simple conclusion, i.e., whether the audio and video data has been tampered with. The model cannot learn human cognitive levels, leading to overfitting to certain scenarios, resulting in excellent results during modeling and training but a significant decline in practical deployment. Moreover, for video and audio temporal-spatial information, the classification label is too rough to provide fine-grained information, preventing the model from learning more essential fake features, which leads to a significant decline in practical results. SUMMARY

[0005] The embodiments of the present application provide a method for identifying audio and video data to solve the problem of too rough identification results obtained by pure classification-based identification.

[0006] Correspondingly, the embodiments of the present application also provide an audio and video data identification device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0007] To solve the above problems, the embodiments of the present application disclose a method for identifying audio and video data, the audio and video data comprising video data and audio data, the method comprising:

[0008] According to the audio and video data, video features and audio features are obtained.

[0009] determine a classification label corresponding to the video feature and a classification label corresponding to the audio feature;

[0010] obtain a video forgery feature corresponding to the video data according to the video feature and the classification label corresponding to the video feature, and obtain an audio forgery feature corresponding to the audio data according to the audio feature and the classification label corresponding to the audio feature;

[0011] obtain a forgery result for determining whether the audio-video data is tampered according to the video forgery feature and the audio forgery feature.

[0012] Optionally, the obtaining of the video feature and the audio feature in feature alignment according to the audio-video data comprises:

[0013] constructing an audio-video encoder and a feature alignment model;

[0014] inputting the audio-video data into the audio-video encoder to obtain a video encoding feature and an audio encoding feature;

[0015] inputting the video encoding feature and the audio encoding feature into the feature alignment model to obtain the video feature and the audio feature.

[0016] Optionally, the determining of the classification label corresponding to the video feature and the classification label corresponding to the audio feature comprises:

[0017] constructing a video frame classifier and an audio frame classifier;

[0018] inputting the video feature into the video frame classifier to obtain the classification label corresponding to the video feature, and inputting the audio feature into the audio frame classifier to obtain the classification label corresponding to the audio feature.

[0019] Optionally, the obtaining of the video forgery feature corresponding to the video data according to the video feature and the classification label corresponding to the video feature, and the obtaining of the audio forgery feature corresponding to the audio data according to the audio feature and the classification label corresponding to the audio feature comprises:

[0020] constructing a video tampering identification model and an audio tampering identification model;

[0021] fusing the video feature and the classification label corresponding to the video feature to obtain a video fusion feature, and inputting the video fusion feature into the video tampering identification model to obtain the video forgery feature; and fusing the audio feature and the classification label corresponding to the audio feature to obtain an audio fusion feature, and inputting the audio fusion feature into the audio tampering identification model to obtain the audio forgery feature.

[0022] Optionally, the obtaining, according to the video forgery feature and the audio forgery feature, a forgery result for determining whether the audio-video data is tampered with comprises:

[0023] constructing an audio-video feature fusion model;

[0024] inputting the video forgery feature and the audio forgery feature into the audio-video feature fusion model to obtain an audio-video fusion feature;

[0025] obtaining a tampering probability corresponding to the audio-video data according to the audio-video fusion feature;

[0026] determining whether the audio-video data is tampered with according to the tampering probability corresponding to the audio-video data and a preset probability threshold;

[0027] if the audio-video data is tampered with, determining a first forgery result;

[0028] if the audio-video data is not tampered with, determining a second forgery result;

[0029] determining the first forgery result or the second forgery result as the forgery result.

[0030] Optionally, the feature alignment model has corresponding model parameters, the feature alignment model comprises feature channels for processing the video encoding feature and the audio encoding feature, and the feature alignment model is trained by the following steps:

[0031] obtaining first sample data for training the feature alignment model;

[0032] calculating a feature distance between the video encoding feature and the audio encoding feature;

[0033] calculating a loss value corresponding to the feature alignment model according to the first sample data and the feature distance;

[0034] adjusting the model parameters corresponding to the feature alignment model according to the loss value corresponding to the feature alignment model, so as to complete the training of the feature alignment model.

[0035] Optionally, the video frame classifier and the audio frame classifier respectively have corresponding model parameters, and the video frame classifier and the audio frame classifier are trained by the following steps:

[0036] obtaining second sample data for training the video frame classifier and third sample data for training the audio frame classifier;

[0037] calculate a first difference value between the classification label corresponding to the video feature and the second sample data, and calculate a second difference value between the classification label corresponding to the audio feature and the third sample data;

[0038] calculate a loss value corresponding to the video frame classifier according to the second sample data and the first difference value, and calculate a loss value corresponding to the audio frame classifier according to the third sample data and the second difference value;

[0039] adjust the model parameter corresponding to the video frame classifier according to the loss value corresponding to the video frame classifier to complete the training of the video frame classifier, and adjust the model parameter corresponding to the audio frame classifier according to the loss value corresponding to the audio frame classifier to complete the training of the audio frame classifier.

[0040] The embodiment of the present application further discloses an audio-video data authentication device, wherein the audio-video data comprises video data and audio data, and the device comprises:

[0041] a feature alignment module configured to obtain video features and audio features in feature alignment according to the audio-video data;

[0042] a classification label module configured to determine a classification label corresponding to the video features and a classification label corresponding to the audio features;

[0043] a fake feature module configured to obtain video fake features corresponding to the video data according to the video features and the classification label corresponding to the video features, and obtain audio fake features corresponding to the audio data according to the audio features and the classification label corresponding to the audio features;

[0044] an authentication result module configured to obtain an authentication result for determining whether the audio-video data is tampered according to the video fake features and the audio fake features.

[0045] The embodiment of the present application further discloses an electronic device comprising a processor and a memory having executable codes stored thereon, wherein the executable codes, when executed, cause the processor to perform the audio-video data authentication method according to any one of the embodiments of the present application.

[0046] The embodiment of the present application further discloses one or more machine readable media having executable codes stored thereon, wherein the executable codes, when executed, cause the processor to perform the audio-video data authentication method according to any one of the embodiments of the present application.

[0047] Compared with the prior art, the embodiment of the present application has the following advantages:

[0048] In the embodiment of the present application, according to the audio-video data, video features and audio features in feature alignment are obtained; classification labels corresponding to the video features and classification labels corresponding to the audio features are determined; according to the video features and the classification labels corresponding to the video features, video forgery features corresponding to the video data are obtained, and according to the audio features and the classification labels corresponding to the audio features, audio forgery features corresponding to the audio data are obtained; according to the video forgery features and the audio forgery features, a forgery judgment result for determining whether the audio-video data is tampered is obtained.

[0049] The embodiment of the present application can extract more fine-grained video and audio forgery features through feature alignment and classification label determination, avoids the problem of too rough classification labels in traditional methods, can further mine more essential forgery features, and can more comprehensively capture forgery information in audio-video data by combining video and audio forgery features, thereby improving the accuracy and robustness of forgery judgment. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is a step flowchart of an audio-video data forgery judgment method embodiment of the present application;

[0051] Figure 2 is a schematic diagram of an audio-video data forgery judgment framework of the present application;

[0052] Figure 3 is a structural block diagram of an audio-video data forgery judgment device embodiment of the present application;

[0053] Figure 4 is a structural schematic diagram of a device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0055] Reference Figure 1 is a step flowchart of an audio-video data forgery judgment method embodiment of the present application, including the following steps:

[0056] Step 101, according to the audio-video data, video features and audio features in feature alignment are obtained;

[0057] In the embodiment of the present application, the audio-video data belongs to multi-modal data, specifically including video data and audio data.

[0058] In actual application scenarios, video data and audio data are usually collected by different collection devices (such as a camera and a microphone) respectively. If these collection devices are not strictly synchronized, the video data and the audio data can be mismatched in time. Moreover, in the transmission process of audio-video data, the video data and the audio data can pass through different transmission paths or network nodes, and factors such as network delay and bandwidth limitation can cause the video data and the audio data to arrive at the destination at different times, thereby causing the video data and the audio data to be mismatched in time. In addition, due to different processing speeds of the video data and the audio data, the processing time is also different, which also causes the video data and the audio data to be mismatched in time.

[0059] The mismatch of the video data and the audio data in time can affect the subsequent authentication, leading to misjudgment or missed judgment. Moreover, the complementarity of the video features and the audio features cannot be fully utilized in the authentication process, which reduces the accuracy and comprehensiveness of multi-modal data analysis, thereby leading to an incorrect authentication result.

[0060] Therefore, in order to ensure the efficiency and accuracy of audio-video data processing, the features extracted from the audio-video data need to be aligned in the embodiments of the present application to obtain the video features and the audio features (audio-video features) that are aligned in time. This not only ensures the consistency of the video features and the audio features in time, but also enhances the ability of multi-modal data fusion, thereby improving the accuracy and reliability of authentication.

[0061] Step 102, determining a classification label corresponding to the video features and a classification label corresponding to the audio features;

[0062] In the embodiments of the present application, the video features and the audio features extracted from the audio-video data are classified respectively. The classification label represents the category or state to which the features belong, such as "real" or "fake".

[0063] Specifically, the video features and the audio features have multiple time frames, and each time frame of the video features and the audio features is classified to obtain the classification labels corresponding to the video features and the audio features (the classification labels corresponding to the audio-video features) respectively under each time frame.

[0064] By classifying the video features and the audio features, the classification labels corresponding to the video features and the classification labels corresponding to the audio features are obtained. In related technologies, the classification label is often used as an authentication result. However, directly using the classification label as an authentication result can make the authentication result too rough, which can lead to low accuracy and reliability of authentication.

[0065] Therefore, after obtaining the classification label, the embodiment of the present application further needs to determine more essential fake features from the fine-grained information based on the audio and video features and the classification labels corresponding to the audio and video features respectively, and further determine the forgery result based on the fake features.

[0066] Step 103, obtaining a video fake feature corresponding to the video data according to the video feature and the classification label corresponding to the video feature, and obtaining an audio fake feature corresponding to the audio data according to the audio feature and the classification label corresponding to the audio feature;

[0067] In the embodiment of the present application, the video fake feature (video fake feature) of the video data will be generated by using the video feature and the classification label corresponding thereto. Similarly, the audio fake feature (audio fake feature) of the audio data will also be generated by using the audio feature and the classification label corresponding thereto. The video fake feature and the audio fake feature are more essential fake features to be obtained in the embodiment of the present application.

[0068] Specifically, the video fake feature includes a video fake spatial region label and a video fake time sequence label, and the audio fake feature refers to an audio fake time sequence label.

[0069] By analyzing the audio and video features and the classification labels corresponding to the audio and video features, the fake features are generated, which can more meticulously analyze the abnormal or fake signs in the video data and the audio data, thereby improving the detection capability of data tampering and enhancing the accuracy and reliability of forgery identification.

[0070] Step 104, obtaining a forgery result for determining whether the audio and video data are tampered according to the video fake feature and the audio fake feature.

[0071] In the embodiment of the present application, after obtaining the fake features in different modalities, the fake features in different modalities need to be combined for analysis. Specifically, the video fake feature and the audio fake feature are comprehensively analyzed, and finally it can be determined whether the audio and video data are tampered.

[0072] By combining the fake features in different modalities, not only the complementarity of multi-modal data can be fully utilized to improve the accuracy of forgery identification, but also the authenticity of data can be more comprehensively evaluated to reduce the possible misjudgment of a single modality.

[0073] The embodiment of the present application can extract more fine-grained video and audio fake features by feature alignment and classification label determination, avoid the problem of too rough classification label in the traditional method, further mine more essential fake features, and combine the video and audio fake features to more comprehensively capture the fake information in the audio and video data, thereby improving the accuracy and robustness of forgery identification.

[0074] In an embodiment of the present application, the step 101 of obtaining the video feature and the audio feature in alignment comprises:

[0075] constructing an audio-video encoder and a feature alignment model;

[0076] inputting the audio-video data into the audio-video encoder to obtain video encoding features and audio encoding features;

[0077] inputting the video encoding features and the audio encoding features into the feature alignment model to obtain the video feature and the audio feature.

[0078] In an embodiment of the present application, an audio-video encoder for extracting original features in audio-video data and a feature alignment model for performing feature alignment on the features output by the audio-video encoder are first constructed. The audio-video encoder specifically includes a video encoder for extracting original features in video data and an audio encoder for extracting original features in audio data.

[0079] Since the original features in the audio-video data can contain a large amount of redundant information, the original features in the audio-video data are further encoded in the audio-video encoder in the embodiment of the present application. Feature encoding can reduce the dimension of the features, remove redundant information, avoid the influence of redundant information on the authenticity verification, improve the accuracy and reliability of the authenticity verification, and reduce the computational complexity and improve the processing efficiency while avoiding overfitting.

[0080] Specifically, the video data is input into the video encoder, the video encoder processes the video data, extracts original features in the video data, and encodes the extracted video original features to obtain video encoding features. Similarly, the audio data is input into the audio encoder, the audio encoder processes the audio data, extracts original features in the audio data, and encodes the extracted audio original features to obtain audio encoding features.

[0081] For example, the original features in the video data can be extracted in the video encoder using a convolutional neural network (CNN), an optical flow method, or the like. The convolutional neural network is a deep learning model that can extract texture, color, edge, and other features in a video frame through multiple convolution and pooling operations, and is particularly suitable for feature extraction of image and video data. The optical flow method is used to analyze the motion information between video frames, and can extract motion features in the video by calculating the pixel displacement between adjacent frames.

[0082] In another example, in the audio encoder, the Mel Frequency Cepstral Coefficients (MFCC), Short-Time Fourier Transform (STFT) and other methods can be used to extract features in the audio data. Mel Frequency Cepstral Coefficients is a commonly used audio feature extraction method, which can extract the spectral features of the audio signal by converting the audio signal into Mel Frequency Cepstral Coefficients, and is particularly suitable for speech recognition and audio classification; Short-Time Fourier Transform is a time-frequency analysis method used to analyze the time-frequency characteristics of the audio signal, which can extract the time-frequency features of the audio signal by dividing the audio signal into short-time windows and performing Fourier transform on each window.

[0083] After obtaining the video encoding features and the audio encoding features, the consistency of the video encoding features and the audio encoding features in time needs to be realized, so the video encoding features and the audio encoding features are input into the feature alignment model to output the video features and the encoding features consistent in time.

[0084] The video data in the embodiments of the present application refers to a video sequence with multiple video frames, and the audio data refers to an audio sequence with multiple audio frames.

[0085] The video frame is a single image in the video sequence, which is usually played continuously at a fixed time interval (such as 24 frames per second, 30 frames per second, etc.) to form a dynamic video. The video frame contains visual information in the video, such as the texture, color, motion, etc. of the image.

[0086] The audio frame is a segment of the audio signal in time, which is usually divided at a fixed time interval (such as every 20 milliseconds, 30 milliseconds, etc.). The audio frame contains time-frequency information of the audio signal, such as frequency spectrum, tone, volume, etc.

[0087] Therefore, in the embodiments of the present application, feature extraction, feature encoding and feature alignment are performed for each video frame in the video data and each audio frame in the audio data, that is, the final video features and audio features are obtained for each frame.

[0088] In the embodiments of the present application, the original features in the audio-video data are extracted by the audio-video encoder, and the original features are processed by feature encoding to obtain the video encoding features and the audio encoding features. Then, the video encoding features and the audio encoding features are aligned by the feature alignment model to obtain the video features and the audio features aligned in features. This not only ensures the consistency of the video features and the audio features in time, but also enhances the ability of multi-modal data fusion, thereby improving the accuracy and reliability of the authentication.

[0089] In an embodiment of the present application, the step 102 of determining the classification label corresponding to the video features and the classification label corresponding to the audio features comprises:

[0090] constructing a video frame classifier and an audio frame classifier;

[0091] inputting the video features into the video frame classifier to obtain classification labels corresponding to the video features, and inputting the audio features into the audio frame classifier to obtain classification labels corresponding to the audio features.

[0092] In the embodiments of the present application, a video frame classifier for classifying video features and an audio frame classifier for classifying audio features are constructed.

[0093] The video frame classifier and the audio frame classifier can identify features of different categories, so that the video features of each video frame can be input into the video frame classifier, and the classification labels corresponding to the video features can be output. Similarly, the audio features of each audio frame can be input into the audio frame classifier, and the classification labels corresponding to the audio features can be output.

[0094] The classification labels corresponding to the video features and the classification labels corresponding to the audio features in the embodiments of the present application are not directly output as the result of authentication, but further mining of more essential forged features needs to be combined with the video features and the audio features.

[0095] In an embodiment of the present application, the step 103 of obtaining the video forged features corresponding to the video data according to the video features and the classification labels corresponding to the video features, and obtaining the audio forged features corresponding to the audio data according to the audio features and the classification labels corresponding to the audio features, comprises:

[0096] constructing a video tampering identification model and an audio tampering identification model;

[0097] After the video features and the classification labels corresponding to the video features are fused to obtain video fusion features, the video fusion features are input into the video tampering identification model to obtain the video forged features; and after the audio features and the classification labels corresponding to the audio features are fused to obtain audio fusion features, the audio fusion features are input into the audio tampering identification model to obtain the audio forged features.

[0098] In the embodiments of the present application, a video tampering identification model for identifying whether there is tampering or forged evidence (forged features) in the video data, and an audio tampering identification model for identifying whether there is tampering or forged evidence in the audio data are constructed.

[0099] In actual application scenarios, a single video feature or audio feature may not be sufficient to fully reflect the authenticity of the data, and therefore, in the embodiments of the present application, the feature representation capability is enhanced by fusing the feature and the classification label, and more essential fake features can be mined, thereby improving the accuracy of tamper detection.

[0100] Specifically, the video feature and the corresponding classification label are fused to obtain a video fusion feature, and the video fusion feature is input into a video tamper detection model, and the video tamper detection model outputs a video fake feature. Similarly, the audio feature and the corresponding classification label are fused to obtain an audio fusion feature, and the audio fusion feature is input into an audio tamper detection model, and the audio tamper detection model outputs an audio fake feature.

[0101] Therefore, the fused video fusion feature and audio fusion feature are more stable and contain more information, can more comprehensively describe the data, can reduce the misjudgment caused by a single feature, and improve the reliability of tamper detection.

[0102] In the embodiments of the present application, the feature and the classification label can be fused by feature splicing, feature weighting or feature concatenation. For example, the video feature and the classification label are spliced together to form a video fusion feature, or the audio feature and the classification label are weighted and fused to obtain an audio fusion feature.

[0103] Further, the video tamper detection model and the audio tamper detection model in the embodiments of the present application can be implemented by a deep learning model (such as a convolutional neural network, a recurrent neural network) or a traditional machine learning model (such as a support vector machine, a random forest) and the like. For example, a convolutional neural network is used to analyze the video fusion feature to identify tamper evidence in the video, and a recurrent neural network is used to analyze the audio fusion feature to identify tamper evidence in the audio.

[0104] The embodiments of the present application can more accurately extract fake features in video and audio data by constructing a video tamper detection model and an audio tamper detection model and fusing the video feature and the classification label and the audio feature and the classification label, thereby improving the accuracy of fake feature extraction and the ability of fake identification, and further improving the accuracy and reliability of audio and video data identification.

[0105] In an embodiment of the present application, the step 104 of obtaining a fake identification result for determining whether the audio and video data is tampered with according to the video fake feature and the audio fake feature comprises:

[0106] constructing an audio and video feature fusion model;

[0107] input the video forgery feature and the audio forgery feature into the audio-video feature fusion model to obtain an audio-video fusion feature;

[0108] obtain a tampering probability corresponding to the audio-video data according to the audio-video fusion feature;

[0109] determine whether the audio-video data is tampered according to the tampering probability corresponding to the audio-video data and a preset probability threshold;

[0110] if the audio-video data is tampered, determine a first forgery judgment result;

[0111] if the audio-video data is not tampered, determine a second forgery judgment result;

[0112] determine the first forgery judgment result or the second forgery judgment result as the forgery judgment result.

[0113] In the embodiments of the present application, an audio-video feature fusion model for fusing a video forgery feature and an audio forgery feature is constructed. Specifically, the video forgery feature and the audio forgery feature are input into the audio-video feature fusion model, and the audio-video feature fusion model outputs an audio-video fusion feature.

[0114] Subsequently, the audio-video fusion feature is analyzed, the tampering probability corresponding to the audio-video data is calculated, and the calculated tampering probability is compared with a preset probability threshold to determine whether the audio-video data is tampered.

[0115] If the tampering probability exceeds the preset threshold, it is determined that the audio-video data is tampered, and a first forgery judgment result (such as “forgery”) is output; if the tampering probability does not exceed the preset threshold, it is determined that the audio-video data is not tampered, and a second forgery judgment result (such as “real”) is output.

[0116] Finally, according to the tampering judgment result, a final forgery judgment result is determined.

[0117] For example, a classification model (such as a support vector machine, a random forest, a deep learning model, etc.) can be used to classify the audio-video fusion feature to output the tampering probability, or a regression model (such as linear regression, logistic regression, a deep learning model, etc.) can be used to perform regression analysis on the audio-video fusion feature to output the tampering probability.

[0118] The tampering probability in the embodiments of the present application is a numerical value representing the possibility of the audio-video data being tampered, which is usually a value between 0 and 1, and the larger the value, the higher the possibility of the audio-video data being tampered.

[0119] For example, assuming that the preset probability threshold is 0.5. If the tampering probability is 0.8, it indicates that the audio and video data is highly likely to be tampered, and the first authentication result is determined, that is, the audio and video data is tampered. If the tampering probability is 0.3, it indicates that the audio and video data is less likely to be tampered, and the second authentication result is determined, that is, the audio and video data is not tampered.

[0120] The embodiments of the present application can fully utilize the complementarity of multi-modal data by fusing video forgery features and audio forgery features, and improve the accuracy of authentication. Comprehensive analysis of video forgery features, audio forgery features and audio-video fusion features can more comprehensively evaluate the authenticity of data and reduce the possible misjudgment of a single modality. By calculating the tampering probability, the possibility of data tampering can be quantified, and the reliability of the authentication result can be improved. Using a preset probability threshold for judgment can avoid misjudgment caused by a single feature and improve the robustness of authentication.

[0121] In an embodiment of the present application, the feature alignment model has corresponding model parameters, and the feature alignment model includes feature channels for processing the video encoding features and the audio encoding features. The feature alignment model is trained by the following steps:

[0122] Obtain first sample data for training the feature alignment model;

[0123] Calculate the feature distance between the video encoding features and the audio encoding features;

[0124] Calculate the loss value corresponding to the feature alignment model according to the first sample data and the feature distance;

[0125] Adjust the model parameters corresponding to the feature alignment model according to the loss value corresponding to the feature alignment model to complete the training of the feature alignment model.

[0126] In the embodiments of the present application, first sample data for training the feature alignment model is needed, and the first sample data includes first sample data that is not tampered and first sample data that is tampered. In each first sample data, the video data and the audio data are segmented into multiple frame segments, and the number of frame segments is referred to as the first sample frame segment number.

[0127] In the feature alignment model, a plurality of feature channels for processing video encoding features and audio encoding features are included, and each feature channel is responsible for processing a specific type of feature and outputting the processed feature.

[0128] Specifically, the video features and the audio features after feature alignment need to be supervised, and the loss value L corresponding to the feature alignment model is calculated c Then, based on L cThe feature alignment model is trained, and the loss value L corresponding to the feature alignment model is... c Specifically, it is calculated using the following formula (1):

[0129]

[0130] Where N represents the number of the first sample data, C f The number of feature channels in the feature alignment model is represented by T, which represents the number of the first sample frame segment, and d is the number of the first sample frame segment. i Represents the feature distance, y i The true label value indicates whether the first sample data has been tampered with, δ represents the nearest distance hyperparameter between audio and video features in the tampered first sample data, and max represents obtaining the maximum value.

[0131] If the first sample data is tampered with, then y i If the first sample data is not tampered with, then y is 0. i The value is 1. δ is a parameter used to measure the distance between audio and video features of tampered samples. This parameter is usually used to determine the similarity or difference between tampered samples, thereby helping the model to identify and classify tampered samples.

[0132] Specifically, the feature distance d i This refers to the Euclidean distance between video features and audio features, which is specifically calculated using the following formula (2):

[0133] d i =||F vi -F ai ‖twenty two)

[0134] Among them, F vi F represents video features. ai Indicates audio characteristics.

[0135] The parameters of the feature alignment model are adjusted by optimizing algorithms (such as gradient descent, Adam, etc.) to minimize the loss value, thereby completing the training of the model.

[0136] In this embodiment of the application, the loss value L corresponding to the feature alignment model is calculated. c Afterwards, it will be based on L c The model parameters of the feature alignment model are adjusted to minimize the loss value, so that the adjusted feature alignment model can more accurately align video and audio features, improve the temporal consistency of video and audio features, and reduce time offset and mismatch problems. Therefore, the accuracy and reliability of audio and video data authentication can be improved based on the adjusted feature alignment model.

[0137] In an embodiment of the present application, the video frame classifier and the audio frame classifier have corresponding model parameters, respectively, and the video frame classifier and the audio frame classifier are trained by the following steps:

[0138] Obtaining second sample data for training the video frame classifier and third sample data for training the audio frame classifier;

[0139] Calculating a first difference value between the classification label corresponding to the video feature and the second sample data, and calculating a second difference value between the classification label corresponding to the audio feature and the third sample data;

[0140] According to the second sample data and the first difference value, calculating a loss value corresponding to the video frame classifier, and according to the third sample data and the second difference value, calculating a loss value corresponding to the audio frame classifier;

[0141] According to the loss value corresponding to the video frame classifier, adjusting the model parameters corresponding to the video frame classifier to complete the training of the video frame classifier; and according to the loss value corresponding to the audio frame classifier, adjusting the model parameters corresponding to the audio frame classifier to complete the training of the audio frame classifier.

[0142] In an embodiment of the present application, the second sample data is a video data sample for training the video frame classifier, which generally includes a video feature and a corresponding true classification label. The third sample data is an audio data sample for training the audio frame classifier, which generally includes an audio feature and a corresponding true classification label.

[0143] In each of the second sample data and the third sample data, it is also divided into multiple frame segments, and the number of frame segments is referred to as the second sample frame segment number or the third sample frame segment number.

[0144] The first difference value is a measure for measuring the difference between the classification label corresponding to the video feature and the true classification label in the second sample data. The second difference value is a measure for measuring the difference between the classification label corresponding to the audio feature and the true classification label in the third sample data. The first difference value and the second difference value are obtained by a cross-entropy function.

[0145] The loss value corresponding to the video frame classifier is an index for measuring the difference between the prediction result of the video frame classifier and the true result. The loss value corresponding to the audio frame classifier is an index for measuring the difference between the prediction result of the audio frame classifier and the true result.

[0146] The parameters of the video frame classifier and the audio frame classifier can be adjusted by an optimization algorithm (such as gradient descent, Adam, etc.) to minimize the loss value, thereby completing the training of the model.

[0147] Specifically, the real classification label corresponding to the second sample data and the real classification label corresponding to the third sample data are determined by the following formula (3):

[0148] Y m = a m Y a + (1 - a m )Y v (3)

[0149] Wherein, m represents the modality corresponding to the sample data, m e {a, v}, a belongs to the video modality, and v belongs to the audio modality, a m represents an adjustable parameter, Y a represents the real classification label corresponding to the video data, and Y v represents the real classification label corresponding to the audio data.

[0150] At this time, the value of a m needs to be adjusted to determine Y m , if a m is adjusted to 1, Y m is the real classification label corresponding to the second sample data, and if a m is adjusted to 0, Y m is the real classification label corresponding to the third sample data.

[0151] The first difference value and the second difference value are calculated by the following formula (4):

[0152]

[0153] Wherein, represents the first difference value or the second difference value, which is determined by the modality corresponding to m, Y mij is Y m , which is determined by the above formula (3), represents the classification label corresponding to the video feature or the classification label corresponding to the audio feature.

[0154] The loss value corresponding to the video frame classifier and the loss value corresponding to the audio frame classifier are calculated by the following formula (5):

[0155]

[0156] Wherein, L f represents the loss value corresponding to the video frame classifier or the loss value corresponding to the audio frame classifier, which is determined by the modality corresponding to m, N represents the number of second sample data or third sample data, and T represents the number of second sample frame segments or the number of third sample frame segments.

[0157] The embodiments of the present application can quantify the classification error by calculating the difference value between the classification label corresponding to the video feature and the audio feature and the sample data, helping the model (video frame classifier and audio frame classifier) to better classify. By optimizing the loss value, the model parameters are adjusted, so that the video frame classifier and the audio frame classifier can more accurately classify. By training the video frame classifier and the audio frame classifier, the reliability of the classification result can be enhanced, and the classification error can be reduced. By training the video frame classifier and the audio frame classifier respectively, the complementarity of the multi-modal data can be fully utilized, and the accuracy of the classification can be improved.

[0158] In an embodiment of the present application, the video tampering identification model and the audio tampering identification model also need to be trained respectively. The video tampering identification model and the audio tampering identification model have corresponding model parameters respectively, and the video tampering identification model and the audio tampering identification model are trained by the following steps:

[0159] Obtain fourth sample data for training the video tampering identification model and fifth sample data for training the audio tampering identification model;

[0160] Calculate the loss value corresponding to the video tampering identification model according to the fourth sample data and the video forgery feature, and calculate the loss value corresponding to the audio tampering identification model according to the fifth sample data and the audio forgery feature;

[0161] Adjust the model parameters corresponding to the video tampering identification model according to the loss value corresponding to the video tampering identification model to complete the training of the video tampering identification model; and adjust the model parameters corresponding to the audio tampering identification model according to the loss value corresponding to the audio tampering identification model to complete the training of the audio tampering identification model.

[0162] In the embodiments of the present application, the fourth sample data is the forgery feature corresponding to the video sample data for training the video tampering identification model. The fifth sample data is the forgery feature corresponding to the audio sample data for training the audio tampering identification model.

[0163] In each fourth sample data and fifth sample data, it will also be divided into multiple frame segments, and the number of these frame segments is called the fourth sample frame segment number or the fifth sample frame segment number.

[0164] The loss value corresponding to the video tampering identification model is an index for measuring the difference between the prediction result and the true result of the video tampering identification model. The loss value corresponding to the audio tampering identification model is an index for measuring the difference between the prediction result and the true result of the audio tampering identification model.

[0165] The parameters of the video tampering identification model and the audio tampering identification model are adjusted by an optimization algorithm (such as gradient descent, Adam, etc.) to minimize the loss value, thereby completing the training of the model.

[0166] Specifically, the loss value corresponding to the video tampering identification model and the loss value corresponding to the audio tampering identification model are calculated by formula (6) as follows:

[0167]

[0168] wherein, L b represents the loss value corresponding to the video tampering identification model or the loss value corresponding to the audio tampering identification model, which is determined by the modal corresponding to m, N represents the number of fourth sample data or fifth sample data, D represents the number of all candidate frame segments in the fourth sample data or the fifth sample data, T represents the number of fourth sample frame segments or the number of fifth sample frame segments, M ijk represents the forgery feature corresponding to the video sample data or the forgery feature corresponding to the audio sample data, represents the video forgery feature or the audio forgery feature, which is also determined by the modal corresponding to m.

[0169] By calculating the difference between the video forgery feature and the audio forgery feature and the sample data, the tampering identification error can be quantified, which helps the model to better perform tampering identification. By optimizing the loss value, the model parameters are adjusted, so that the video tampering identification model and the audio tampering identification model can more accurately perform tampering identification. By training the video tampering identification model and the audio tampering identification model, the reliability of the tampering identification result can be enhanced, and the tampering identification error can be reduced. By training the video tampering identification model and the audio tampering identification model respectively, the complementarity of multi-modal data can be fully utilized, and the accuracy of tampering identification can be improved.

[0170] In the embodiments of the present application, the audio-video feature fusion model can also be supervised by the above formula (6) to obtain the loss value corresponding to the audio-video feature fusion model, so as to adjust the model parameters corresponding to the audio-video feature fusion model.

[0171] The embodiments of the present application optimize the loss value and adjust the model parameters, so that the audio-video feature fusion model can more accurately fuse video and audio features, improve the accuracy of audio-video feature fusion, reduce feature fusion error, improve the accuracy of multi-modal data analysis, and further improve the authenticity and reliability of audio-video data.

[0172] Referring to Figure 2is a kind of audio and video data authentication framework schematic diagram of the application, including the input of video data and audio data, feature coding, feature alignment, feature classification, tampering identification, feature fusion and multiple modules.For each module, there is a corresponding supervision mechanism for calculating loss value and adjusting model parameters to optimize the performance of the model.Through these modules and supervision mechanisms, feature extraction, alignment, classification, feature fusion and tampering identification of audio and video data can be realized, thereby improving the accuracy and reliability of audio and video data authentication.

[0173] Specifically, the input video data is encoded by the video encoder to obtain video encoding features, and the input audio data is encoded by the audio encoder to obtain audio encoding features.The video encoding features and the audio encoding features are input into the feature alignment model to obtain the video features and the audio features aligned in features.The feature alignment model is supervised according to the video features and the audio features aligned in features, and the corresponding loss value of the model is calculated, and the corresponding model parameters are adjusted.

[0174] The input video features are classified by the video frame classifier to obtain the classification label corresponding to the video features, and the input audio features are classified by the audio frame classifier to obtain the classification label corresponding to the audio features.The video frame classifier is supervised according to the classification label corresponding to the video features, the corresponding loss value of the video frame classifier is calculated, and the corresponding model parameters are adjusted.The audio frame classifier is supervised according to the classification label corresponding to the audio features, the corresponding loss value of the audio frame classifier is calculated, and the corresponding model parameters are adjusted.

[0175] According to the video features and the classification label corresponding to the video features, it is identified whether there is tampering in the video data, according to the audio features and the classification label corresponding to the audio features, it is identified whether there is tampering in the audio data, and the video tampering identification model and the audio tampering identification model are supervised respectively, the corresponding loss value of the model is calculated, and the model parameters are adjusted.

[0176] It is also necessary to fuse the video forgery features and the audio forgery features through the audio and video feature fusion model to obtain the audio and video fusion features, and to supervise the audio and video feature fusion model, calculate the corresponding loss value of the model, and adjust the model parameters.

[0177] The embodiments of the application can extract more fine-grained video and audio forgery features by determining the feature alignment and classification label, avoiding the problem of too rough classification label in traditional methods, and can further mine more essential forgery features.Combining the video and audio forgery features, the forgery information in the audio and video data can be more comprehensively captured, and the accuracy and robustness of authentication are improved.

[0178] It should be noted that, for the method embodiments, the series of acts complement each other to achieve the purpose of this application, therefore, this application is not limited to the described act sequence, because some acts can be performed in other sequences or at the same time. Besides, those skilled in the art will appreciate that the embodiments described in the specification are preferred embodiments, and the acts involved are not necessarily required by this application.

[0179] Based on the above-mentioned embodiments, the present embodiment further provides an audio and video data authentication device, which is applied to electronic devices such as terminal devices and servers.

[0180] Referring to Figure 3 , a structural block diagram of an audio and video data authentication device embodiment of the present application is shown, which can specifically include the following modules:

[0181] The feature alignment module 301 is configured to obtain feature-aligned video features and audio features according to the audio and video data.

[0182] The classification label module 302 is configured to determine classification labels corresponding to the video features and classification labels corresponding to the audio features.

[0183] The counterfeit feature module 303 is configured to obtain video counterfeit features corresponding to the video data according to the video features and the classification labels corresponding to the video features, and obtain audio counterfeit features corresponding to the audio data according to the audio features and the classification labels corresponding to the audio features.

[0184] The authentication result module 304 is configured to obtain an authentication result for determining whether the audio and video data is tampered according to the video counterfeit features and the audio counterfeit features.

[0185] The present embodiment further provides a non-volatile readable storage medium, which stores one or more programs, and when the one or more programs are applied to a device, the device can execute instructions of each method step in the present embodiment.

[0186] The present embodiment provides one or more machine-readable media, which store instructions, and when executed by one or more processors, make an electronic device execute a method as described in one or more of the above embodiments. In the present embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0187] Embodiments of the present disclosure can be implemented as a device configured with desired functionality using any suitable hardware, firmware, software, or any combination thereof, which can include terminal devices, servers (clusters), and the like electronic devices. Figure 4 An exemplary device 400 that can be used to implement various embodiments described herein is shown schematically.

[0188] For one embodiment, Figure 4 An exemplary device 400 is shown having one or more processors 402, a control module (chipset) 404 coupled to at least one of the processor(s) 402, a memory 406 coupled to the control module 404, a non-volatile memory (NVM) / storage device 408 coupled to the control module 404, one or more input / output devices 410 coupled to the control module 404, and a network interface 412 coupled to the control module 404.

[0189] The processor(s) 402 can include one or more single core or multicore processors, which can include any combination of general-purpose processors or dedicated processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 400 can function as a terminal device, a server (cluster), and the like devices described in embodiments herein.

[0190] In some embodiments, the device 400 can include one or more computer- readable media (e.g., the memory 406 or the NVM / storage device 408) having instructions 414 and one or more processors 402 incorporated with the one or more computer-readable media configured to execute the instructions 414 to implement modules to perform the actions described in the present disclosure.

[0191] For one embodiment, the control module 404 can include any suitable interface controllers to provide for any suitable interface to at least one of the processor(s) 402 and / or any suitable device or component in communication with the control module 404.

[0192] The control module 404 can include a memory controller module to provide an interface to the memory 406. The memory controller module can be a hardware module, a software module, and / or a firmware module.

[0193] The memory 406 can be used, for example, to load and store data and / or instructions 414 for the device 400. For one embodiment, the memory 406 can include any suitable volatile memory, such as suitable DRAM. In some embodiments, the memory 406 can include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0194] For one embodiment, control module 404 can include one or more input / output controllers to provide an interface to NVM / storage 408 and input / output device(s) 410.

[0195] For example, NVM / storage 408 can be used to store data and / or instructions 414. NVM / storage 408 can include any suitable non-volatile memory (e.g., flash memory) and / or can include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0196] NVM / storage 408 can include storage resources that are physically part of the device on which the apparatus 400 is installed or that is accessed via the device but that is not necessarily physically part of the device. For example, NVM / storage 408 can be accessed over a network via input / output device(s) 410.

[0197] Input / output device(s) 410 can provide an interface for apparatus 400 to communicate with any other suitable device, and input / output device(s) 410 can include communication components, audio components, sensor components, etc. Network interface 412 can provide an interface for apparatus 400 to communicate over one or more networks, and apparatus 400 can communicate wirelessly with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as to access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or combinations thereof.

[0198] For one embodiment, at least one of processor(s) 402 can be packaged together with logic of one or more controllers of control module 404 (e.g., a memory controller module). For one embodiment, at least one of processor(s) 402 can be packaged together with logic of one or more controllers of control module 404 to form a system in a package (SiP). For one embodiment, at least one of processor(s) 402 can be integrated on the same die with logic of one or more controllers of control module 404. For one embodiment, at least one of processor(s) 402 can be integrated on the same die with logic of one or more controllers of control module 404 to form a system on a chip (SoC).

[0199] In various embodiments, the apparatus 400 can be, but is not limited to, a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet, a netbook, etc.) or the like. In various embodiments, the apparatus 400 can have more or fewer components, and / or different architectures. For example, in some embodiments, the apparatus 400 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including touch screen displays), non- volatile memory port, multiple antennas, a graphics chip, an application specific integrated circuit (ASIC), and a speaker.

[0200] In the detection device, a master control chip can be used as a processor or a control module, sensor data, position information, etc. are stored in a memory or NVM / storage device, a sensor group can be used as an input / output device, and a communication interface can include a network interface.

[0201] For the apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts are described in the method embodiment.

[0202] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other.

[0203] The embodiments of the present application are described with reference to flowcharts and / or block diagrams according to the method, terminal device (system), and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable audio / video data authentication terminal device processor to produce a machine, so that the instructions executed by the computer or other programmable audio / video data authentication terminal device processor produce an article of manufacture including a machine that implements the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks Figure 1 The apparatus that implements the function specified in one block or multiple blocks.

[0204] These computer program instructions can also be stored in a computer-readable memory that can guide the computer or other programmable audio / video data authentication terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction apparatus that implements the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks Figure 1 The apparatus that implements the function specified in one block or multiple blocks.

[0205] These computer program instructions can also be loaded into a computer or other programmable audio / video data authentication terminal device to cause a series of operational steps to be performed on the computer or other programmable terminal device to produce a computer-implemented process such that the instructions which execute on the computer or other programmable terminal device provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0206] Although the preferred embodiments of the application have been described, those skilled in the art will be able to make additional changes and modifications to these embodiments once they have the benefit of the present disclosure. Therefore, it is intended to include all such changes and modifications in the scope of the application as set forth in the claims.

[0207] Finally, it should be noted that, in the specification, relational terms such as first and second, and the like, can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or terminal device. An element proceeded by "comprises a..." does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or terminal device that comprises the identified element.

[0208] The above provides a method and device for audio / video data authentication, an electronic device and a storage medium, and the principles and implementation manners of the application are described by using specific examples. The above description of the embodiments is only used to help understand the method of the application and its core idea; meanwhile, for those skilled in the art, according to the idea of the application, the specific implementation manners and application ranges can be changed, and the above description should not be understood as limiting the application.

Claims

1. An audio / video data authentication method, characterized by comprising: The audio-video data includes video data and audio data, and the method includes: According to the audio-video data, video features and audio features are obtained in alignment; Classify the video features and the audio features to determine the classification labels corresponding to the video features and the classification labels corresponding to the audio features; Fuse the video features and the classification labels corresponding to the video features to obtain video forgery features corresponding to the video data, and fuse the audio features and the classification labels corresponding to the audio features to obtain audio forgery features corresponding to the audio data; According to the video forgery features and the audio forgery features, a forgery result for determining whether the audio-video data is tampered with is obtained.

2. The method of claim 1, wherein, According to the audio-video data, video features and audio features are obtained in alignment, including: An audio-video encoder and a feature alignment model are constructed; The audio-video data is input into the audio-video encoder to obtain video coding features and audio coding features; The video coding features and the audio coding features are input into the feature alignment model to obtain the video features and the audio features.

3. The method of claim 1, wherein, The video features and the audio features are classified to determine the classification labels corresponding to the video features and the classification labels corresponding to the audio features, including: A video frame classifier and an audio frame classifier are constructed; The video features are input into the video frame classifier to obtain the classification labels corresponding to the video features, and the audio features are input into the audio frame classifier to obtain the classification labels corresponding to the audio features.

4. The method of claim 1, wherein, The video features and the classification labels corresponding to the video features are fused to obtain video forgery features corresponding to the video data, and the audio features and the classification labels corresponding to the audio features are fused to obtain audio forgery features corresponding to the audio data, including: A video tampering identification model and an audio tampering identification model are constructed; After the video features and the classification labels corresponding to the video features are fused, video fusion features are obtained, and the video fusion features are input into the video tampering identification model to obtain the video forgery features; and after the audio features and the classification labels corresponding to the audio features are fused, audio fusion features are obtained, and the audio fusion features are input into the audio tampering identification model to obtain the audio forgery features.

5. The method of claim 1, wherein, According to the video forgery features and the audio forgery features, a forgery result for determining whether the audio-video data is tampered with is obtained, including: An audio-video feature fusion model is constructed; The video forgery features and the audio forgery features are input into the audio-video feature fusion model to obtain audio-video fusion features; According to the audio-video fusion features, a tampering probability corresponding to the audio-video data is obtained; According to the tampering probability corresponding to the audio-video data and a preset probability threshold, it is determined whether the audio-video data is tampered with; If the audio-video data is tampered with, a first forgery result is determined; If the audio-video data is not tampered with, a second forgery result is determined; determine the first anti-fake result or the second anti-fake result as the anti-fake result.

6. The method of claim 2, wherein, The feature alignment model has corresponding model parameters, and the feature alignment model includes feature channels for processing the video coding features and the audio coding features, and the feature alignment model is trained by the following steps: obtain first sample data for training the feature alignment model; calculate feature distances between the video coding features and the audio coding features; calculate loss values corresponding to the feature alignment model according to the first sample data and the feature distances; adjust the model parameters corresponding to the feature alignment model according to the loss values corresponding to the feature alignment model, so as to complete the training of the feature alignment model.

7. The method of claim 3, wherein, The video frame classifier and the audio frame classifier have corresponding model parameters respectively, and the video frame classifier and the audio frame classifier are trained by the following steps: obtain second sample data for training the video frame classifier and third sample data for training the audio frame classifier; calculate first difference values between classification labels corresponding to the video features and the second sample data, and calculate second difference values between classification labels corresponding to the audio features and the third sample data; calculate loss values corresponding to the video frame classifier according to the second sample data and the first difference values, and calculate loss values corresponding to the audio frame classifier according to the third sample data and the second difference values; adjust the model parameters corresponding to the video frame classifier according to the loss values corresponding to the video frame classifier, so as to complete the training of the video frame classifier; and adjust the model parameters corresponding to the audio frame classifier according to the loss values corresponding to the audio frame classifier, so as to complete the training of the audio frame classifier.

8. An apparatus for authenticating audiovisual data, characterized by comprising: The audio-video data includes video data and audio data, and the device includes: a feature alignment module configured to obtain video features and audio features aligned in features according to the audio-video data; a classification label module configured to classify the video features and the audio features, and determine classification labels corresponding to the video features and classification labels corresponding to the audio features; a fake feature module configured to fuse the video features and the classification labels corresponding to the video features to obtain video fake features corresponding to the video data, and fuse the audio features and the classification labels corresponding to the audio features to obtain audio fake features corresponding to the audio data; an anti-fake result module configured to obtain an anti-fake result for determining whether the audio-video data is tampered according to the video fake features and the audio fake features.

9. An electronic device, comprising: include: a processor; and a memory having executable code stored thereon, which, when executed, causes the processor to perform the anti-fake method of the audio-video data as claimed in any one of claims 1-7.

10. One or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform the anti-fake method of the audio-video data as claimed in any one of claims 1-7.

Images