Audio depth forgery detection method and device, terminal and storage medium

Abstract

The invention discloses an audio deep forgery detection method and device, a terminal and a storage medium, and relates to the technical field of multimedia information security and artificial intelligence, and the method comprises the steps: constructing an audio deep forgery detection network model; according to the training set, performing decoupling stage training on an audio deep counterfeiting detection network model, and determining an initial training network model; performing meta-learning on the initial training network model according to the training set, and determining a target training network model; and obtaining a to-be-detected audio, inputting the to-be-detected audio into the target training network model, and determining an audio category corresponding to the to-be-detected audio. According to the method, meta-learning is adopted in the training stage of the audio deep forgery detection network model to force the audio deep forgery detection network model to learn general knowledge with higher generalization, so that the problem of poor detection performance caused by model overfitting in the face of an unknown vocoder and a natural scene in the prior art can be effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multimedia information security and artificial intelligence technology, and in particular to an audio deepfake detection method and device, a terminal and a storage medium. BACKGROUND

[0002] In recent years, generative artificial intelligence (GenAI) has made breakthrough progress in deepfake technology. One of the core components of these technologies is the vocoder, which can efficiently convert acoustic features such as mel-spectrograms into high-quality audio waveforms. However, the misuse of audio deepfake technology also poses a serious social security risk. Therefore, developing a general detection technology that can effectively detect unknown and new types of fake audio has become an urgent need in the field of information security.

[0003] Currently, the mainstream audio deepfake detection (ADD) method usually consists of a front-end feature extractor and a back-end classifier. In terms of feature extraction, research has shown that pre-trained models based on self-supervised learning, such as Wav2Vec 2.0, WavLM and HuBERT, significantly outperform traditional manually designed features such as MFCC and LFCC, as they can capture potential acoustic patterns in speech through large-scale data pre-training. In terms of the classifier, researchers have adopted a variety of deep learning models, including convolutional neural networks (CNN), deep residual networks (ResNet), and graph attention networks (GAT), and have achieved good detection results on standard datasets. Although existing technologies have achieved some success, they are prone to overfitting to training data during the training process, resulting in a sharp decline in detection performance when faced with unseen attack types.

[0004] Therefore, the existing technology still needs to be improved and developed. SUMMARY

[0005] The technical problem to be solved by the present application is to provide an audio deepfake detection method, device, terminal and storage medium to solve the problem of poor detection performance caused by model overfitting when facing unknown vocoders and natural scenes.

[0006] The technical solution adopted by the present application to solve the problem is as follows:

[0007] In a first aspect, the present application provides an audio deepfake detection method, wherein the method comprises:

[0008] constructing an audio deepfake detection network model;

[0009] obtaining a training set, performing decoupling stage training on the audio deepfake detection network model according to the training set, and determining an initial training network model;

[0010] performing meta-learning on the initial training network model according to the training set, and determining a target training network model;

[0011] obtaining audio to be detected, inputting the audio to be detected into the target training network model, and determining an audio category corresponding to the audio to be detected.

[0012] In an implementation method, the audio deepfake detection network model includes a first feature extractor, a second feature extractor, an adversarial training decoupling learning module, and a reconstruction decoupling module. The adversarial training decoupling learning module includes a vocoder-specific encoder and a vocoder-independent encoder. The decoupling stage training of the audio deepfake detection network model according to the training set to determine the initial training network model includes:

[0013] performing feature extraction on the training set by the first feature extractor to determine first global features;

[0014] encoding the first global features by the vocoder-specific encoder to determine vocoder-specific features;

[0015] encoding the first global features by the vocoder-independent encoder to determine vocoder-independent features;

[0016] determining a first decoupling stage loss according to the vocoder-specific features and the vocoder-independent features;

[0017] performing feature extraction on the training set by the second feature extractor to determine second global features;

[0018] performing feature decoupling on the second global features by the reconstruction decoupling module to determine synthesized-independent features;

[0019] determining a second decoupling stage loss according to the vocoder-independent features and the synthesized-independent features;

[0020] constructing a target loss function according to the first decoupling stage loss and the second decoupling stage loss, and optimizing parameters of the audio deepfake detection network model based on the target loss function to determine the initial training network model.

[0021] In an implementation method, the adversarial training decoupling learning module further includes a domain classifier, a fake classifier, and a discriminator with gradient reversal, a first decoupling stage loss is determined according to the vocoder-specific features and the vocoder-agnostic features, including:

[0022] A first vocoder class is determined by the domain classifier according to the vocoder-specific features;

[0023] A true or false class is determined by the fake classifier according to the vocoder-agnostic features;

[0024] A second vocoder class is determined by the discriminator according to the vocoder-agnostic features;

[0025] The first decoupling stage loss is determined according to the first vocoder class, the true or false class, and the second vocoder class.

[0026] In an implementation method, the reconstruction decoupling module includes a speaker encoder and a content encoder, a synthesis-agnostic feature is determined by feature decoupling of the second global feature by the reconstruction decoupling module, including:

[0027] A speaker feature is determined by encoding the second global feature by the speaker encoder;

[0028] A content feature is determined by encoding the second global feature by the content encoder;

[0029] The synthesis-agnostic feature is determined according to the speaker feature and the content feature.

[0030] In an implementation method, the reconstruction decoupling module further includes an audio generator, a second decoupling stage loss is determined according to the vocoder-agnostic features and the synthesis-agnostic features, including:

[0031] The speaker encoder and the vocoder-agnostic features are adaptively instance normalized to determine a fusion feature;

[0032] A reconstructed audio signal is determined by the audio generator according to the fusion feature and the content feature;

[0033] The second decoupling stage loss is determined according to the vocoder-agnostic features, the synthesis-agnostic features, and the reconstructed audio signal;

[0034] The parameters of the second feature extractor, the content encoder, and the audio generator are in a frozen state.

[0035] In an implementation method, the initial training network model is meta-learned according to the training set, and a target training network model is determined, including:

[0036] Each parameter in the initial training network model except for parameters in the vocoder-independent encoder and the forgery classifier is set to a frozen state, and a parameter frozen network model is determined;

[0037] The parameter frozen network model is meta-learned based on the training set, and the target training network model is determined.

[0038] In an implementation method, the parameter frozen network model is meta-learned based on the training set, and the target training network model is determined, including:

[0039] The training set is dynamically divided into a meta-training set and a meta-test set;

[0040] The meta-training set is input into the parameter frozen network model, and a meta-training loss is determined;

[0041] The meta-test set is input into the parameter frozen network model, and a meta-test loss is determined;

[0042] A meta-learning loss function is determined according to the meta-training loss and the meta-test loss, and parameters in the vocoder-independent encoder and the forgery classifier in the parameter frozen network model are optimized based on the meta-learning loss function, and the target training network model is determined.

[0043] In a second aspect, the embodiments of the present application also provide an audio deep forgery detection device, wherein the audio deep forgery detection device includes:

[0044] A model construction module is configured to construct an audio deep forgery detection network model;

[0045] A decoupling training module is configured to obtain a training set, and train the audio deep forgery detection network model in a decoupling stage according to the training set, and determine an initial training network model;

[0046] A meta-learning module is configured to meta-learn the initial training network model according to the training set, and determine a target training network model;

[0047] An audio detection module is configured to obtain a to-be-detected audio, input the to-be-detected audio into the target training network model, and determine an audio category corresponding to the to-be-detected audio.

[0048] In a third aspect, an embodiment of the present application further provides a terminal, comprising a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the audio deepfake detection method according to any one of the above; and the processor is configured to execute the programs.

[0049] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a plurality of instructions, wherein the instructions are adapted to be loaded and executed by a processor to implement the audio deepfake detection method according to any one of the above.

[0050] The present application has the following beneficial effects: the embodiment of the present application constructs an audio deepfake detection network model; obtains a training set, decouples and trains the audio deepfake detection network model according to the training set, determines an initial training network model; meta-learns the initial training network model according to the training set, determines a target training network model; obtains an audio to be detected, inputs the audio to be detected into the target training network model, and determines an audio category corresponding to the audio to be detected. Since the embodiment of the present application adopts meta-learning in the training stage of the audio deepfake detection network model, it forces the audio deepfake detection network model to learn more general and universal knowledge, and thus effectively solves the problem of poor detection performance caused by model overfitting when the prior art faces unknown vocoders and natural scenes. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0052] Figure 1 is a flowchart of the audio deepfake detection method provided by the embodiment of the present application.

[0053] Figure 2 is an embodiment flowchart of the audio deepfake detection method provided by the embodiment of the present application.

[0054] Figure 3 is an internal module diagram of the audio deepfake detection device provided by the embodiment of the present application.

[0055] Figure 4 is a principle block diagram of the terminal provided by the embodiment of the present application. DETAILED DESCRIPTION

[0056] The application discloses an audio deepfake detection method and device, a terminal and a storage medium. To make the purpose, technical solution and effect of the application clearer and more explicit, the application is further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and not to limit the application.

[0057] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a," "an," and "the" as used herein can include plural referents. It should be further understood that the use of the term "include" in the specification of the application means that the stated features, integers, steps, operations, elements, and / or components are present, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, "connected" or "coupled" as used herein can include wireless connection or wireless coupling. The phrase "and / or" as used herein includes all or any single unit and all combinations of the associated listed items.

[0058] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0059] Currently, mainstream audio deepfake detection (ADD) methods usually consist of a front-end feature extractor and a back-end classifier. In terms of feature extraction, studies have shown that pre-trained models based on self-supervised learning, such as Wav2Vec 2.0, WavLM, and HuBERT, significantly outperform traditional manually designed features such as MFCC and LFCC in terms of performance, as they are able to capture potential acoustic patterns in speech through large-scale pre-training. In terms of classifiers, researchers have adopted a variety of deep learning models, including convolutional neural networks (CNN), deep residual networks (ResNet), and graph attention networks (GAT), and have achieved good detection results on standard datasets. Although existing technologies have achieved some success, they are prone to overfitting to training data during the training process, resulting in a sharp decline in detection performance when faced with unseen attack types.

[0060] To solve the above problems of the prior art, the present application provides an audio deepfake detection method, which can be applied to a smart terminal, a server or a cloud platform. The method comprises the following steps: constructing an audio deepfake detection network model; obtaining a training set, decoupling stage training the audio deepfake detection network model according to the training set, and determining an initial training network model; meta-learning the initial training network model according to the training set, and determining a target training network model; obtaining a to-be-detected audio, inputting the to-be-detected audio into the target training network model, and determining an audio category corresponding to the to-be-detected audio. Since the present application adopts meta-learning in the training stage of the audio deepfake detection network model, it forces the audio deepfake detection network model to learn more general knowledge, thereby effectively solving the problem of poor detection performance caused by model overfitting when the prior art faces unknown vocoders and natural scenes.

[0061] An exemplary method comprises the following steps:

[0062] As shown in the figure, the method comprises the following steps: Figure 1

[0063] Step S100, constructing an audio deepfake detection network model.

[0064] In simple terms, the audio deepfake detection network model is a network model for identifying whether an audio is fake. The audio deepfake detection network model can be realized based on a convolutional neural network, a recurrent neural network / long short-term memory network, a generative adversarial network, etc.

[0065] Step S200, obtaining a training set, decoupling stage training the audio deepfake detection network model according to the training set, and determining an initial training network model.

[0066] Specifically, the training set is a fake audio dataset. In this embodiment, a dataset containing real audio and fake audio generated by four known vocoders is input into the audio deepfake detection network model for training as a training sample. The decoupling stage training is supervised learning. In the decoupling stage training process, the audio deepfake detection network model performs multi-level decoupling training on the input model data (training set or to-be-detected audio), so that the model separates the multiple influencing factors (such as fake traces, vocoder types, speaker identities, etc.) intertwined in the audio signal, and extracts more pure and general features.

[0067] Step S300, meta-learning the initial training network model according to the training set, and determining a target training network model.

[0068] ​In simple terms, meta-learning can improve the model's "learning ability" and make it perform better in complex real-world scenarios. By combining decoupling stage training and meta-learning, the application simulates cross-vocoder and cross-scene distribution changes, forcing the model to learn more generalizable knowledge, resulting in a detection model with generalization ability for unknown vocoders.

[0069] In an implementation manner, the audio deepfake detection network model comprises a first feature extractor, a second feature extractor, an adversarial training decoupling learning module, and a reconstruction decoupling module, the adversarial training decoupling learning module comprises a vocoder-specific encoder and a vocoder-independent encoder, the audio deepfake detection network model is trained in a decoupling stage according to the training set, and an initial training network model is determined, comprising:

[0070] Step S2011, feature extraction of the training set is performed by the first feature extractor to determine first global features;

[0071] Step S2012, the first global features are encoded by the vocoder-specific encoder to determine vocoder-specific features;

[0072] Step S2013, the first global features are encoded by the vocoder-independent encoder to determine vocoder-independent features;

[0073] Step S2014, a first decoupling stage loss is determined according to the vocoder-specific features and the vocoder-independent features;

[0074] Step S2015, feature extraction of the training set is performed by the second feature extractor to determine second global features;

[0075] Step S2016, the second global features are decoupled according to the reconstruction decoupling module to determine synthesized independent features;

[0076] Step S2017, a second decoupling stage loss is determined according to the vocoder-independent features and the synthesized independent features;

[0077] Step S2018, a target loss function is constructed according to the first decoupling stage loss and the second decoupling stage loss, and the parameters of the audio deepfake detection network model are optimized based on the target loss function to determine the initial training network model.

[0078] Specifically, as shown in Figure 2 The audio deepfake detection network model is a double-layer decoupling (unwinding) deep network model. The audio deepfake detection network model comprises a preset feature extractor, an adversarial training decoupling module, and a reconstruction decoupling module.

[0079] First Feature Extractor E D Second feature extractor E R This involves a self-supervised pre-trained large model (such as the WavLM-Large model). When data (training set or audio to be detected) is input into the audio deep fakery detection network model, pre-trained weights are loaded onto the first and second feature extractors, respectively. The feature extractors then extract features from the input data to obtain the first global feature f corresponding to the input data. D The system extracts both the first global feature and the second global feature, where the first feature extractor extracts the first global feature and the second feature extractor extracts the second global feature. By employing a self-supervised pre-trained large model as the feature extractor, a new paradigm is provided for speech processing tasks. The model trained on a large-scale dataset exhibits robustness and generalization capabilities far exceeding those of conventional models.

[0080] The adversarial training decoupling module includes a vocoder-specific encoder E d Voice coder-independent encoder E a The adversarial training decoupling module is the first layer of decoupling structure in the audio deepfake detection network model. It decouples global features at the low-level signal layer (primarily involving the original waveform and spectral features of the audio), including encoding global features using a vocoder-specific encoder to obtain vocoder-specific features f. d The global features are encoded using a vocoder-independent encoder to obtain the vocoder-independent features f. a The loss corresponding to the first decoupling structure, i.e., the first decoupling stage loss, is calculated based on vocoder-specific and vocoder-independent features. In the adversarial training decoupling module, the vocoder-specific encoder and the vocoder-independent encoder have the same network structure, but their parameters are not shared. For example, the vocoder-specific encoder and the vocoder-independent encoder can consist of a linear layer, two bottleneck blocks, and a global average pooling layer.

[0081] The reconstruction decoupling module forms the second-layer decoupling structure. This module performs high-level semantic decoupling on the second global features, yielding synthesis-independent and synthesis-related features. Based on these vocoder-independent and synthesis-independent features, the loss for the second decoupling stage is calculated. A target loss function for the audio deepfake detection network model is constructed based on the losses for the first and second decoupling stages. Specifically, the target loss function is obtained by weighted summation of the losses included in the first and second decoupling stages. During training, the parameters of the audio deepfake detection network model are optimized based on the target loss function to obtain the initial trained network model.

[0082] In an implementation manner, the adversarial training decoupling learning module further comprises a domain classifier, a fake classifier, and a discriminator with gradient reversal, a first decoupling stage loss is determined according to the vocoder-specific features and the vocoder-independent features, and the first decoupling stage loss comprises:

[0083] Step S20141, a first vocoder category is determined by classifying the vocoder-specific features through the domain classifier;

[0084] Step S20142, a true or false category is determined by classifying the vocoder-independent features through the fake classifier;

[0085] Step S20143, a second vocoder category is determined by classifying the vocoder-independent features through the discriminator;

[0086] Step S20144, the first decoupling stage loss is determined according to the first vocoder category, the true or false category, and the second vocoder category.

[0087] Specifically, the adversarial training decoupling learning module further comprises a domain classifier, a fake classifier, and a discriminator with gradient reversal. The vocoder-specific features are input into the domain classifier to obtain a first vocoder category; the vocoder-independent features are input into the fake classifier to obtain a true or false category; and the vocoder-independent features are input into the discriminator (domain discriminator) with gradient reversal to obtain a second vocoder category. In the training process, through adversarial training, the vocoder-independent encoder is guided to learn to extract vocoder-independent features, so that the features are difficult to be identified by the discriminator as domain-related information (such as vocoder type or real speech category), and can be effectively distinguished by the fake classifier as true or false, thereby reducing the domain information (vocoder-specific information) in the vocoder-independent features. The discriminator can be composed of two linear blocks, one linear layer, and a GRL layer. The fake classifier and the domain classifier can be composed of linear layers.

[0088] The first decoupling stage loss is determined according to the first vocoder category, the true or false category, and the second vocoder category, comprising: a domain classification loss is calculated by a cross-entropy loss function according to the first vocoder category and a true category corresponding to input data (a training set or audio to be detected); a true or false classification loss is determined by a cross-entropy function according to the true or false category and a true or false category corresponding to the input data; a domain adversarial loss is calculated by a cross-entropy function according to the first vocoder category and the true category; a first feature decoupling loss is determined according to an L2 norm of a covariance matrix between the vocoder-specific features and the vocoder-independent features, the first feature decoupling loss is used to measure the redundancy between the features, so that the model learns more independent and non-redundant feature representations; and the first decoupling stage loss is determined according to the domain classification loss, the true or false classification loss, the domain adversarial loss, and the first feature decoupling loss.

[0089] In each loss function, the authenticity classification loss is represented as follows:

[0090]

[0091] wherein, is a cross-entropy loss function based on logit normalization, is a cross-entropy loss function, λ1 is a hyperparameter of the domain classification loss, is an authenticity classification loss function, is a domain classification loss function, C a is a fake classifier, y i is an authenticity label of a training sample, C d is a domain classifier, a i is a label of the first vocoder and the second vocoder, i is the i-th training sample.

[0092] vocoder-specific feature f d vocoder-agnostic feature f a the first feature decoupling loss of:

[0093]

[0094] wherein, Cov is a covariance matrix calculated for two feature vectors.

[0095] domain adversarial loss:

[0096]

[0097] wherein, is the expectation of the joint distribution of the sample (x i , a i ) according to the (X, A) distribution, a i is a label of the first vocoder and the second vocoder, x i is the i-th training sample, E D is a first feature extractor, E a is a vocoder-agnostic encoder, D is a domain discriminator, X is a training sample set, and A is a label set of the first vocoder and the second vocoder.

[0098] In an implementation manner, the reconstruction decoupling module includes a speaker encoder and a content encoder, and the second global feature is subjected to feature decoupling according to the reconstruction decoupling module to determine a synthesis-agnostic feature, including:

[0099] Step S20161, encoding the second global feature by the speaker encoder to determine a speaker feature;

[0100] Step S20162, encoding the second global feature by the content encoder to determine a content feature;

[0101] Step S20163, determining the synthesis-independent feature according to the speaker feature and the content feature.

[0102] Specifically, the reconstruction decoupling module decouples the second global feature in the second stage to classify the synthesis-dependent feature and the synthesis-independent feature from the second global feature again, and further purifies the features. In this embodiment, the synthesis-independent feature is composed of the speaker feature and the content feature. The reconstruction decoupling module includes an independent content encoder and a speaker encoder. The content feature is extracted from the vocoder-independent feature by the content encoder, and the speaker feature is extracted from the vocoder-independent feature by the speaker encoder. This process aims to suppress the semantic and speaker information irrelevant to the synthesis task, so that the features used for detection finally pay more attention to the essential traces of forgery.

[0103] In an implementation manner, the reconstruction decoupling module further includes an audio generator configured to determine a second decoupling stage loss according to the vocoder-independent feature and the synthesis-independent feature, including:

[0104] Step S20171, performing adaptive instance normalization operation on the speaker encoder and the vocoder-independent feature to determine a fusion feature;

[0105] Step S20172, reconstructing by the audio generator according to the fusion feature and the content feature to determine a reconstructed audio signal;

[0106] Step S20173, determining a second decoupling stage loss according to the vocoder-independent feature, the synthesis-independent feature, and the reconstructed audio signal;

[0107] The parameters of the second feature extractor, the content encoder, and the audio generator are in a frozen state.

[0108] Specifically, the reconstruction decoupling module further includes an audio generator. The speaker feature in the synthesis-independent feature (including the speaker feature and the content feature) and the vocoder-independent feature are fused by adaptive instance normalization (AdaIN) operation to obtain a fusion feature. The audio generator reconstructs the audio according to the fusion feature and the content feature in the synthesis-independent feature to obtain a reconstructed audio signal. The content encoder, the speaker encoder, and the generator in the reconstruction decoupling module can be composed of standard network components such as linear layers, convolutional layers, and recurrent layers.

[0109] The second decoupling stage loss is determined according to the vocoder-independent feature, the synthesis-independent feature and the reconstructed audio signal, including: determining the second feature decoupling loss according to the vocoder-independent feature and the synthesis-independent feature or determining the second feature decoupling loss according to the vocoder-independent feature and the speaker feature; measuring the consistency of the speaker information by calculating the L1 norm of the learned speaker feature and the speaker feature extracted by the pre-training model to obtain the speaker consistency loss; obtaining the reconstruction loss by calculating the L1 norm between the spectrogram of the reconstructed audio signal and the spectrogram of the audio signal of the input data; determining the second decoupling stage loss according to the second feature decoupling loss, the consistency loss and the reconstruction loss.

[0110] In each loss function, the speaker consistency loss is expressed as:

[0111]

[0112] wherein f s is the learned speaker feature, is the speaker feature extracted by the pre-training model;

[0113] The second feature decoupling loss of the vocoder-independent feature f a and the speaker feature f s :

[0114]

[0115] The speech reconstruction loss:

[0116]

[0117] wherein x sp and are the spectrograms of the input audio signal and the reconstructed audio signal respectively.

[0118] In an implementation manner, the initial training network model is meta-learned according to the training set to determine a target training network model, including:

[0119] In step S2021, parameters in the initial training network model except for parameters in the vocoder-independent encoder and the fake classifier are set to a frozen state to determine a parameter frozen network model.

[0120] In step S2022, the parameter frozen network model is meta-learned based on the training set to determine the target training network model.

[0121] Briefly, in the audio deepfake detection network model, the deepfake classification output by the deepfake classifier based on the vocoder-independent feature is taken as the audio classification corresponding to the model input data. In order to obtain more accurate audio classification detection results, all other parameters in the initial training model except the parameters of the vocoder-independent encoder and the parameters of the deepfake classifier are set to a frozen state to obtain a parameter frozen network model. By meta-learning the parameter frozen network model, the parameters of the vocoder-independent encoder and the deepfake classifier are further optimized to improve the generalization ability of the model, so that the model can have good performance in different environments.

[0122] In an implementation manner, the parameter frozen network model is meta-learned based on the training set to determine a target training network model, including:

[0123] Step S20221, dynamically dividing the training set into a meta-training set and a meta-test set;

[0124] Step S20222, inputting the meta-training set into the parameter frozen network model to determine a meta-training loss;

[0125] Step S20223, inputting the meta-test set into the parameter frozen network model to determine a meta-test loss;

[0126] Step S20224, determining a meta-learning loss function according to the meta-training loss and the meta-test loss, optimizing the parameters of the vocoder-independent encoder and the deepfake classifier in the parameter frozen network model based on the meta-learning loss function, and determining the target training network model.

[0127] Specifically, the training set (data domain) is dynamically divided into a meta-training set and a meta-test set by a preset vocoder-agnostic meta-learning (VAML). By inputting the meta-training set into the parameter frozen network model, the parameters of the vocoder-independent encoder and the deepfake classifier in the parameter frozen network model are updated based on the target loss function of the parameter frozen network model to obtain an adapted model. Then, the performance of the adapted model is evaluated on the meta-test set, and a meta-test loss is calculated. Finally, the meta-training loss and the meta-test loss are combined for double-layer optimization to improve the generalization ability of the model to unknown vocoders.

[0128] Step S400, obtaining a to-be-detected audio, inputting the to-be-detected audio into the target training network model, and determining an audio category corresponding to the to-be-detected audio.

[0129] Specifically, the target training network model is an audio deep fake detection network model that has been trained. By inputting the audio to be detected into the target training network model, the feature extractor extracts features from the audio to be detected to obtain global features; the vocoder-agnostic encoder extracts features from the global features to obtain a vocoder-agnostic feature; the fake classifier classifies and detects according to the vocoder-agnostic feature to obtain a true or false category (a probability that the audio is fake); and the true or false category output by the fake classifier is taken as the audio classification corresponding to the audio to be detected.

[0130] In the process of training the audio deep fake detection network model, PyTorch 2.0 (an open-source Python machine learning library) can be used for implementation, and all experiments can be performed on an NVIDIA A100 GPU. A public dataset containing real audio and four kinds of fake audio (such as Voc.v4) is used as a training set, and other public datasets containing unknown vocoders (such as ASVspoof2019 LA, 21DF, WaveFake, etc.) are used as test sets. The types of vocoders and the sources of real speech in the test set and the training set are not completely the same. In the training and testing stages, the input audio is resampled to 16 kHz and cropped to 4 seconds, the Adam optimizer is used, the batch size is 8, the learning rate is set to 5x10 -5 , and the epochs in the training stage can be set to 100. The detection performance is evaluated mainly by the equal error rate (EER).

[0131] The embodiment trains the network model constructed based on decoupling learning and meta-learning by using a training dataset containing multiple vocoder fake samples, so as to obtain a network model that can accurately determine whether an audio has been subjected to a fake operation. With the general fake traces learned by double-layer decoupling and the generalization ability obtained by meta-learning, the trained model can efficiently detect fake audios generated by unknown vocoders and audios in a natural scene, and plays an important role in improving the audio deep fake detection performance and overcoming the core problem of weak generalization and robustness in current field research.

[0132] Based on the above embodiment, the present application further provides an audio deep fake detection device, as shown in Figure 3 , the device comprises:

[0133] A model construction module 01 is configured to construct an audio deep fake detection network model.

[0134] A decoupling training module 02 is configured to obtain a training set, train the audio deep fake detection network model in a decoupling stage according to the training set, and determine an initial training network model.

[0135] The meta-learning module 03 is configured to perform meta-learning on the initial training network model according to the training set to determine a target training network model.

[0136] The audio detection module 04 is configured to acquire a to-be-detected audio, input the to-be-detected audio into the target training network model, and determine an audio category corresponding to the to-be-detected audio.

[0137] Based on the above embodiments, the present application further provides a terminal, a principle block diagram of which can be shown in Figure 4 The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. The processor of the terminal is configured to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the terminal is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement the audio deepfake detection method. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0138] Those skilled in the art can understand that Figure 4 The principle block diagram shown in the above embodiments is only a block diagram of part of the structure related to the present application scheme, and does not constitute a limitation on the terminal to which the present application scheme is applied. The specific terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0139] In an implementation manner, the memory of the terminal stores more than one program, and is configured to execute the more than one program by more than one processor, which includes instructions for performing the audio deepfake detection method.

[0140] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of each method can be included. Any reference to memory, storage, database or other medium used in each embodiment of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0141] In summary, the present application discloses an audio depth forgery detection method, device, terminal and storage medium. The method constructs an audio depth forgery detection network model, obtains a training set, decouples and trains the audio depth forgery detection network model according to the training set to determine an initial training network model, performs meta-learning on the initial training network model according to the training set to determine a target training network model, obtains an audio to be detected, inputs the audio to be detected into the target training network model, and determines an audio category corresponding to the audio to be detected. Since the present application adopts meta-learning in the training stage of the audio depth forgery detection network model to force the audio depth forgery detection network model to learn more general and universal knowledge, the problem of poor detection performance caused by model overfitting when the prior art faces unknown vocoders and natural scenes can be effectively solved.

[0142] It should be understood that the application of the present application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all these improvements and changes shall belong to the protection scope of the claims of the present application.

Claims

1. A method for detecting deepfake audio, characterized in that, The method includes: Construct a network model for detecting deep audio forgery; Obtain a training set, and perform decoupling phase training on the audio deep forgery detection network model based on the training set to determine the initial training network model; Meta-learning is performed on the initial trained network model based on the training set to determine the target trained network model; The audio to be detected is obtained, and the audio to be detected is input into the target training network model to determine the audio category corresponding to the audio to be detected.

2. The audio depth forgery detection method according to claim 1, characterized in that, The audio deepfake detection network model includes a first feature extractor, a second feature extractor, an adversarial training decoupling learning module, and a reconstruction decoupling module. The adversarial training decoupling learning module includes a vocoder-specific encoder and a vocoder-independent encoder. The audio deepfake detection network model is trained in a decoupling phase based on the training set to determine the initial training network model, including: The first feature extractor is used to extract features from the training set to determine the first global feature. The first global feature is encoded by the vocoder-specific encoder to determine the vocoder-specific feature; The first global feature is encoded by the vocoder-independent encoder to determine the vocoder-independent feature; The loss of the first decoupling stage is determined based on the vocoder-specific features and the vocoder-independent features. The training set is used to extract features through the second feature extractor to determine the second global feature; The reconstruction and decoupling module performs feature decoupling on the second global feature to determine the synthesized irrelevant features. The loss of the second decoupling stage is determined based on the vocoder-independent features and the synthesis-independent features; A target loss function is constructed based on the first decoupling stage loss and the second decoupling stage loss, and the parameters of the audio depth forgery detection network model are optimized based on the target loss function to determine the initial training network model.

3. The audio depth forgery detection method according to claim 2, characterized in that, The adversarial training decoupling learning module further includes a domain classifier, a spoofing classifier, and a discriminator with gradient inversion. It determines the loss for the first decoupling stage based on the vocoder-specific features and the vocoder-independent features, including: The domain classifier classifies the vocoder based on its specific features to determine the first vocoder category. The forgery classifier classifies forgeries based on vocoder-independent features to determine the authenticity category; The discriminator classifies the second vocoder category based on the vocoder-independent features. The loss of the first decoupling stage is determined based on the first vocoder category, the true / false category, and the second vocoder category.

4. The audio depth forgery detection method according to claim 2, characterized in that, The reconstruction decoupling module includes a speaker encoder and a content encoder. Based on the reconstruction decoupling module, the second global feature is decoupled to determine synthesized irrelevant features, including: The second global feature is encoded using the speaker encoder to determine the speaker feature; The second global feature is encoded using the content encoder to determine the content feature; The synthesis-irrelevant features are determined based on the speaker features and the content features.

5. The audio depth forgery detection method according to claim 4, characterized in that, The reconstruction and decoupling module further includes an audio generator, which determines the loss of the second decoupling stage based on the vocoder-independent features and the synthesis-independent features, including: The speaker encoder and the vocoder-independent features are subjected to adaptive instance normalization to determine the fused features; The audio generator reconstructs the audio signal based on the fusion features and the content features to determine the reconstructed audio signal. The loss of the second decoupling stage is determined based on the vocoder-independent features, the synthesis-independent features, and the reconstructed audio signal. The parameters of the second feature extractor, the content encoder, and the audio generator are frozen.

6. The audio deepfake detection method according to claim 3, characterized in that, Meta-learning is performed on the initial trained network model based on the training set to determine the target trained network model, including: All parameters in the initial trained network model, except those in the vocoder-independent encoder and the spoofing classifier, are set to a frozen state to determine the parameter-frozen network model. Meta-learning is performed on the parameter-frozen network model based on the training set to determine the target training network model.

7. The audio depth forgery detection method according to claim 6, characterized in that, Meta-learning is performed on the parameter-frozen network model based on the training set to determine the target training network model, including: The training set is dynamically divided into a meta-training set and a meta-test set; The meta-training set is input into the parameter-freezing network model to determine the meta-training loss; The meta-test set is input into the parameter-frozen network model to determine the meta-test loss; The meta-learning loss function is determined based on the meta-training loss and the meta-testing loss. Based on the meta-learning loss function, the parameters in the vocoder-independent encoder and the spoofing classifier in the parameter-frozen network model are optimized to determine the target training network model.

8. An audio deepfake detection device, characterized in that, The device includes: The model building module is used to build a network model for detecting deep audio forgery. The decoupled training module is used to acquire a training set, perform decoupled phase training on the audio deep forgery detection network model based on the training set, and determine the initial training network model. The meta-learning module is used to perform meta-learning on the initial trained network model based on the training set to determine the target trained network model; The audio detection module is used to acquire the audio to be detected, input the audio to be detected into the target training network model, and determine the audio category corresponding to the audio to be detected.

9. A terminal, characterized in that, The terminal includes a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the audio depth forgery detection method as described in any one of claims 1-7; the processors are used to execute the programs.

10. A computer-readable storage medium storing a plurality of instructions thereon, characterized in that, The instructions are loaded and executed by the processor to implement the steps of the audio depth forgery detection method according to any one of claims 1-7.

Images