An audio deepfake detection method and device, a terminal and a storage medium

Abstract

The application discloses an audio deepfake detection method and device, a terminal and a storage medium, relates to the technical field of multimedia information security and artificial intelligence, and the method constructs an audio deepfake detection network model; the audio deepfake detection network model is trained in a decoupling stage according to the training set, and an initial training network model is determined; meta-learning is performed on the initial training network model according to the training set, and a target training network model is determined; a to-be-detected audio is acquired, the to-be-detected audio is input into the target training network model, and the audio category corresponding to the to-be-detected audio is determined. Since the application adopts meta-learning in the training stage of the audio deepfake detection network model, the audio deepfake detection network model is forced to learn more general and universal knowledge, and therefore, the problem that the detection performance is poor due to model overfitting when the prior art faces unknown vocoders and natural scenes can be effectively solved.

Description

Technical Field

[0001] This invention relates to the fields of multimedia information security and artificial intelligence technology, and in particular to an audio deepfake detection method, device, terminal and storage medium. Background Technology

[0002] In recent years, deepfake technology, centered on Generative Artificial Intelligence (GenAI), has made groundbreaking progress. One of the core components of these technologies is the vocoder, which efficiently converts acoustic features such as Mel-spectrum data into high-quality audio waveforms. However, the misuse of audio deepfake technology has also brought serious social security risks. Therefore, developing a universal detection technology capable of effectively detecting unknown and novel forged audio has become an urgent need in the field of information security.

[0003] Currently, mainstream audio deepfake detection (ADD) methods typically consist of a front-end feature extractor and a back-end classifier. Regarding feature extraction, research has demonstrated that pre-trained models based on self-supervised learning, such as Wav2Vec 2.0, WavLM, and HuberT, significantly outperform traditional manually designed features like MFCC and LFCC due to their ability to capture latent acoustic patterns in speech through large-scale pre-training. For classifiers, researchers have employed various deep learning models, including Convolutional Neural Networks (CNNs), Deep Residual Networks (ResNets), and Graph Attention Networks (GATs), achieving good detection results on standard datasets. Despite these successes, existing techniques are prone to overfitting to training data during training, leading to a sharp decline in detection performance when faced with unseen attack types.

[0004] Therefore, existing technologies still need improvement and development. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an audio deepfake detection method, device, terminal and storage medium to address the above-mentioned defects of the prior art, aiming to solve the problem of poor detection performance caused by model overfitting when the prior art is faced with unknown vocoders and natural scenes.

[0006] The technical solution adopted by this invention to solve the problem is as follows:

[0007] In a first aspect, embodiments of the present invention provide an audio depth forgery detection method, wherein the method includes:

[0008] Construct a network model for detecting deep audio forgery;

[0009] Obtain a training set, and perform decoupling phase training on the audio deep forgery detection network model based on the training set to determine the initial training network model;

[0010] Meta-learning is performed on the initial trained network model based on the training set to determine the target trained network model;

[0011] The audio to be detected is obtained, and the audio to be detected is input into the target training network model to determine the audio category corresponding to the audio to be detected.

[0012] In one implementation method, the audio deepfake detection network model includes a first feature extractor, a second feature extractor, an adversarial training decoupling learning module, and a reconstruction decoupling module. The adversarial training decoupling learning module includes a vocoder-specific encoder and a vocoder-independent encoder. The audio deepfake detection network model is trained in a decoupling phase based on the training set to determine an initial training network model, including:

[0013] The first feature extractor is used to extract features from the training set to determine the first global feature.

[0014] The first global feature is encoded by the vocoder-specific encoder to determine the vocoder-specific feature;

[0015] The first global feature is encoded by the vocoder-independent encoder to determine the vocoder-independent feature;

[0016] The loss of the first decoupling stage is determined based on the vocoder-specific features and the vocoder-independent features.

[0017] The training set is used to extract features through the second feature extractor to determine the second global feature;

[0018] The reconstruction and decoupling module performs feature decoupling on the second global feature to determine the synthesized irrelevant features.

[0019] The loss of the second decoupling stage is determined based on the vocoder-independent features and the synthesis-independent features;

[0020] A target loss function is constructed based on the first decoupling stage loss and the second decoupling stage loss, and the parameters of the audio depth forgery detection network model are optimized based on the target loss function to determine the initial training network model.

[0021] In one implementation, the adversarial training decoupling learning module further includes a domain classifier, a spoofing classifier, and a discriminator with gradient inversion, determining a first decoupling stage loss based on vocoder-specific features and vocoder-independent features, including:

[0022] The domain classifier classifies the vocoder based on its specific features to determine the first vocoder category.

[0023] The forgery classifier classifies forgeries based on vocoder-independent features to determine the authenticity category;

[0024] The discriminator classifies the second vocoder category based on the vocoder-independent features.

[0025] The loss of the first decoupling stage is determined based on the first vocoder category, the true / false category, and the second vocoder category.

[0026] In one implementation, the reconstruction decoupling module includes a speaker encoder and a content encoder. The reconstruction decoupling module performs feature decoupling on the second global feature to determine synthesized irrelevant features, including:

[0027] The second global feature is encoded using the speaker encoder to determine the speaker feature;

[0028] The second global feature is encoded using the content encoder to determine the content feature;

[0029] The synthesis-irrelevant features are determined based on the speaker features and the content features.

[0030] In one implementation, the reconstruction decoupling module further includes an audio generator that determines a second decoupling stage loss based on the vocoder-independent features and the synthesis-independent features, including:

[0031] The speaker encoder and the vocoder-independent features are subjected to adaptive instance normalization to determine the fused features;

[0032] The audio generator reconstructs the audio signal based on the fusion features and the content features to determine the reconstructed audio signal.

[0033] The loss of the second decoupling stage is determined based on the vocoder-independent features, the synthesis-independent features, and the reconstructed audio signal.

[0034] The parameters of the second feature extractor, the content encoder, and the audio generator are frozen.

[0035] In one implementation method, meta-learning is performed on an initial trained network model based on the training set to determine a target trained network model, including:

[0036] All parameters in the initial trained network model, except those in the vocoder-independent encoder and the spoofing classifier, are set to a frozen state to determine the parameter-frozen network model.

[0037] Meta-learning is performed on the parameter-frozen network model based on the training set to determine the target training network model.

[0038] In one implementation method, meta-learning is performed on the parameter-frozen network model based on the training set to determine the target training network model, including:

[0039] The training set is dynamically divided into a meta-training set and a meta-test set;

[0040] The meta-training set is input into the parameter-freezing network model to determine the meta-training loss;

[0041] The meta-test set is input into the parameter-frozen network model to determine the meta-test loss;

[0042] The meta-learning loss function is determined based on the meta-training loss and the meta-testing loss. Based on the meta-learning loss function, the parameters in the vocoder-independent encoder and the spoofing classifier in the parameter-frozen network model are optimized to determine the target training network model.

[0043] Secondly, embodiments of the present invention also provide an audio depth spoofing detection device, wherein the audio depth spoofing detection device includes:

[0044] The model building module is used to build a network model for detecting deep audio forgery.

[0045] The decoupled training module is used to acquire a training set, perform decoupled phase training on the audio deep forgery detection network model based on the training set, and determine the initial training network model.

[0046] The meta-learning module is used to perform meta-learning on the initial trained network model based on the training set to determine the target trained network model;

[0047] The audio detection module is used to acquire the audio to be detected, input the audio to be detected into the target training network model, and determine the audio category corresponding to the audio to be detected.

[0048] Thirdly, embodiments of the present invention also provide a terminal, the terminal including a memory and one or more processors; the memory stores one or more programs; the programs include instructions for executing the audio depth forgery detection method as described above; the processor is used to execute the programs.

[0049] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a plurality of instructions, wherein the instructions are adapted to be loaded and executed by a processor to implement any of the audio depth forgery detection methods described above.

[0050] The beneficial effects of this invention are as follows: This invention constructs an audio deepfake detection network model; obtains a training set; performs decoupling training on the audio deepfake detection network model based on the training set to determine an initial training network model; performs meta-learning on the initial training network model based on the training set to determine a target training network model; obtains the audio to be detected; inputs the audio to be detected into the target training network model to determine the audio category corresponding to the audio to be detected. Because this invention uses meta-learning in the training phase of the audio deepfake detection network model, forcing the model to learn more generalized and universal knowledge, it can effectively solve the problem of poor detection performance due to model overfitting when facing unknown vocoders and natural scenes in existing technologies. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating the audio depth forgery detection method provided in this embodiment of the invention.

[0053] Figure 2 This is a schematic flowchart of an embodiment of the audio depth forgery detection method provided in this invention.

[0054] Figure 3 This is a schematic diagram of the internal modules of the audio depth forgery detection device provided in an embodiment of the present invention.

[0055] Figure 4 This is a schematic diagram of the terminal provided in an embodiment of the present invention. Detailed Implementation

[0056] This invention discloses a method, apparatus, terminal, and storage medium for detecting deepfake audio. To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.

[0057] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0058] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0059] Currently, mainstream audio deepfake detection (ADD) methods typically consist of a front-end feature extractor and a back-end classifier. Regarding feature extraction, research has demonstrated that pre-trained models based on self-supervised learning, such as Wav2Vec 2.0, WavLM, and HuberT, significantly outperform traditional manually designed features like MFCC and LFCC due to their ability to capture latent acoustic patterns in speech through large-scale pre-training. For classifiers, researchers have employed various deep learning models, including Convolutional Neural Networks (CNNs), Deep Residual Networks (ResNets), and Graph Attention Networks (GATs), achieving good detection results on standard datasets. Despite these successes, existing techniques are prone to overfitting to training data during training, leading to a sharp decline in detection performance when faced with unseen attack types.

[0060] To address the aforementioned shortcomings of existing technologies, this invention provides an audio deepfake detection method applicable to smart terminals, servers, or cloud platforms. The method involves constructing an audio deepfake detection network model; acquiring a training set; performing decoupling-stage training on the audio deepfake detection network model based on the training set to determine an initial training network model; performing meta-learning on the initial training network model based on the training set to determine a target training network model; acquiring the audio to be detected; inputting the audio to be detected into the target training network model to determine the audio category corresponding to the audio to be detected. Because this invention employs meta-learning during the training phase of the audio deepfake detection network model, forcing the model to learn more generalized and universal knowledge, it effectively solves the problem of poor detection performance due to model overfitting when facing unknown vocoders and natural scenes in existing technologies.

[0061] Exemplary method:

[0062] like Figure 1 As shown, the method includes:

[0063] Step S100: Construct an audio deep forgery detection network model.

[0064] In simple terms, a deepfake audio detection network model is a network model used to identify whether audio is fake. Deepfake audio detection network models can be implemented based on convolutional neural networks, recurrent neural networks / long short-term memory networks, generative adversarial networks, etc.

[0065] Step S200: Obtain the training set, and perform decoupling training on the audio deep forgery detection network model based on the training set to determine the initial training network model.

[0066] Specifically, the training set is a dataset of fake audio. In this embodiment, a dataset containing real audio and fake audio generated by four known vocoders is used as training samples and input into the audio deep forgery detection network model for training. The decoupling phase training is supervised learning. During the decoupling phase training, the audio deep forgery detection network model performs multi-level decoupling training on the data input to the model (training set or audio to be detected), enabling the model to separate various entangled influencing factors in the audio signal (such as forgery traces, vocoder type, speaker identity, etc.) and extract purer and more general features.

[0067] Step S300: Perform meta-learning on the initial training network model based on the training set to determine the target training network model.

[0068] In simple terms, meta-learning enhances a model's "learning ability," enabling it to perform better in complex real-world scenarios. This invention combines decoupled training with meta-learning to simulate distribution variations across vocoders and scenarios, forcing the model to learn more generalizable knowledge, thus obtaining a detection model with the ability to generalize to unknown vocoders.

[0069] In one implementation, the audio deepfake detection network model includes a first feature extractor, a second feature extractor, an adversarial training decoupling learning module, and a reconstruction decoupling module. The adversarial training decoupling learning module includes a vocoder-specific encoder and a vocoder-independent encoder. The audio deepfake detection network model is trained in a decoupling phase based on the training set to determine the initial training network model, including:

[0070] Step S2011: Extract features from the training set using the first feature extractor to determine the first global feature;

[0071] Step S2012: Encode the first global feature using the vocoder-specific encoder to determine the vocoder-specific feature;

[0072] Step S2013: Encode the first global feature using the vocoder-independent encoder to determine the vocoder-independent feature;

[0073] Step S2014: Determine the first decoupling stage loss based on the vocoder-specific features and the vocoder-independent features;

[0074] Step S2015: Extract features from the training set using the second feature extractor to determine the second global feature;

[0075] Step S2016: Decouple the second global feature according to the reconstruction and decoupling module to determine the synthesized irrelevant features;

[0076] Step S2017: Determine the loss of the second decoupling stage based on the vocoder-independent features and the synthesis-independent features;

[0077] Step S2018: Construct a target loss function based on the first decoupling stage loss and the second decoupling stage loss, and optimize the parameters of the audio depth forgery detection network model based on the target loss function to determine the initial training network model.

[0078] Specifically, such as Figure 2 As shown, the audio deepfake detection network model is a two-layer decoupled (unentangled) deep network model. The audio deepfake detection network model includes a pre-defined feature extractor, an adversarial training decoupling module, and a reconstruction decoupling module.

[0079] First Feature Extractor E D Second feature extractor E R This involves a self-supervised pre-trained large model (such as the WavLM-Large model). When data (training set or audio to be detected) is input into the audio deep fakery detection network model, pre-trained weights are loaded onto the first and second feature extractors, respectively. The feature extractors then extract features from the input data to obtain the first global feature f corresponding to the input data. D The system extracts both the first global feature and the second global feature, where the first feature extractor extracts the first global feature and the second feature extractor extracts the second global feature. By employing a self-supervised pre-trained large model as the feature extractor, a new paradigm is provided for speech processing tasks. The model trained on a large-scale dataset exhibits robustness and generalization capabilities far exceeding those of conventional models.

[0080] The adversarial training decoupling module includes a vocoder-specific encoder E d Voice coder-independent encoder E a The adversarial training decoupling module is the first layer of decoupling structure in the audio deepfake detection network model. It decouples global features at the low-level signal layer (primarily involving the original waveform and spectral features of the audio), including encoding global features using a vocoder-specific encoder to obtain vocoder-specific features f. d The global features are encoded using a vocoder-independent encoder to obtain the vocoder-independent features f. a The loss corresponding to the first decoupling structure, i.e., the first decoupling stage loss, is calculated based on vocoder-specific and vocoder-independent features. In the adversarial training decoupling module, the vocoder-specific encoder and the vocoder-independent encoder have the same network structure, but their parameters are not shared. For example, the vocoder-specific encoder and the vocoder-independent encoder can consist of a linear layer, two bottleneck blocks, and a global average pooling layer.

[0081] The reconstruction decoupling module forms the second-layer decoupling structure. This module performs high-level semantic decoupling on the second global features, yielding synthesis-independent and synthesis-related features. Based on these vocoder-independent and synthesis-independent features, the loss for the second decoupling stage is calculated. A target loss function for the audio deepfake detection network model is constructed based on the losses for the first and second decoupling stages. Specifically, the target loss function is obtained by weighted summation of the losses included in the first and second decoupling stages. During training, the parameters of the audio deepfake detection network model are optimized based on the target loss function to obtain the initial trained network model.

[0082] In one implementation, the adversarial training decoupling learning module further includes a domain classifier, a spoofing classifier, and a discriminator with gradient inversion, determining the first decoupling stage loss based on the vocoder-specific features and the vocoder-independent features, including:

[0083] Step S20141: Classify the vocoder according to its specific features using the domain classifier to determine the first vocoder category;

[0084] Step S20142: The forgery classifier classifies the forgery based on the vocoder-independent features to determine the authenticity category;

[0085] Step S20143: The discriminator classifies the second vocoder category based on the vocoder-independent features.

[0086] Step S20144: Determine the first decoupling stage loss based on the first vocoder category, the true / false category, and the second vocoder category.

[0087] Specifically, the adversarial training decoupled learning module also includes a domain classifier, a spoofing classifier, and a discriminator with gradient inversion. Vocoder-specific features are input into the domain classifier to obtain the first vocoder category; vocoder-independent features are input into the spoofing classifier to obtain the true / false category; and vocoder-independent features are input into the discriminator (domain discriminator) with gradient inversion to obtain the second vocoder category. During training, adversarial training guides the vocoder-independent encoder to learn to extract encoder-independent features, making them difficult for the discriminator to identify as having domain-related information (such as vocoder type or real speech category) while also being effectively distinguishable as true / false by the spoofing classifier, thereby reducing the domain information (vocoder-specific information) in the vocoder-independent features. The discriminator can consist of two linear blocks, one linear layer, and a GRL layer. The spoofing classifier and domain classifier can both consist of linear layers.

[0088] The loss for the first decoupling stage is determined based on the first vocoder category, the true / false category, and the second vocoder category. This includes: calculating the domain classification loss using the cross-entropy loss function based on the first vocoder category and the true category corresponding to the input data (training set or audio to be detected); determining the true / false classification loss using the cross-entropy function based on the true / false category and the true / false category corresponding to the input data; calculating the domain adversarial loss using the cross-entropy function based on the first vocoder category and the true category; determining the first feature decoupling loss based on the L2 norm of the covariance matrix between vocoder-specific features and vocoder-independent features, using the first feature decoupling loss to measure the redundancy between features, enabling the model to learn more independent and non-redundant feature representations; and determining the first decoupling stage loss based on the domain classification loss, the true / false classification loss, the domain adversarial loss, and the first feature decoupling loss.

[0089] Among the various loss functions, the true / false classification loss is expressed as follows:

[0090]

[0091] in, The cross-entropy loss function is based on logit normalization. Let λ be the cross-entropy loss function, and λ1 be a hyperparameter of the domain classification loss. For true / false classification loss function, For the domain classification loss function, C a To fake the classifier, y i C represents the true and false labels for the training samples. d For the domain classifier, a i Let i be the label for the first and second vocoder categories, and let i be the i-th training sample.

[0092] Vocoder specific features f d Vocoder-independent feature f a The first characteristic decoupling loss:

[0093]

[0094] Where Cov is the covariance matrix used to calculate the two eigenvectors.

[0095] Domain adversarial losses:

[0096]

[0097] in, For the sample (x) i ,a i Take the expectation and a according to the joint distribution of (X,A) distribution. i Labels for the first and second vocoder categories, x i For the i-th training sample, E D For the first feature extractor, E a Let X be a vocoder-independent encoder, D be a domain discriminator, X be the training sample set, and A be the tag set for the first and second vocoder categories.

[0098] In one implementation, the reconstruction decoupling module includes a speaker encoder and a content encoder. The reconstruction decoupling module performs feature decoupling on the second global feature to determine synthesized irrelevant features, including:

[0099] Step S20161: Encode the second global feature using the speaker encoder to determine the speaker feature;

[0100] Step S20162: Encode the second global feature using the content encoder to determine the content feature;

[0101] Step S20163: Determine the synthetic irrelevant features based on the speaker features and the content features.

[0102] Specifically, the reconstruction and decoupling module performs a second-stage decoupling based on the second global features, further refining the features by classifying and synthesizing relevant and irrelevant features from the second global features. In this embodiment, the synthesized irrelevant features consist of speaker features and content features. The reconstruction and decoupling module includes independent content encoders and speaker encoders. The content encoder extracts content features from the vocoder-irrelevant features, and the speaker encoder extracts speaker features from the vocoder-irrelevant features. This process aims to suppress semantic and speaker information irrelevant to the synthesis task, making the features ultimately used for detection more focused on essential forgery traces.

[0103] In one implementation, the reconstruction decoupling module further includes an audio generator that determines a second decoupling stage loss based on the vocoder-independent features and the synthesis-independent features, including:

[0104] Step S20171: Perform adaptive instance normalization on the speaker encoder and the vocoder-independent features to determine the fused features;

[0105] Step S20172: The audio generator reconstructs the audio signal based on the fusion features and the content features to determine the reconstructed audio signal;

[0106] Step S20173: Determine the second decoupling stage loss based on the vocoder-independent features, the synthesis-independent features, and the reconstructed audio signal;

[0107] The parameters of the second feature extractor, the content encoder, and the audio generator are frozen.

[0108] Specifically, the reconstruction decoupling module also includes an audio generator. It utilizes Adaptive Instance Normalization (AdaIN) to fuse speaker features and vocoder-independent features from synthesis-independent features (which include speaker features and content features) to obtain fused features. The audio generator then reconstructs the audio signal based on the fused features and the content features from the synthesis-independent features. The content encoder, speaker encoder, and generator in the reconstruction decoupling module can be composed of standard network components such as linear layers, convolutional layers, and recurrent layers.

[0109] The loss for the second decoupling stage is determined based on vocoder-independent features, synthesis-independent features, and the reconstructed audio signal. This includes: determining the second feature decoupling loss based on vocoder-independent features and synthesis-independent features, or determining the second feature decoupling loss based on vocoder-independent features and speaker features; measuring the consistency of speaker information by calculating the L1 norm of the learned speaker features and the speaker features extracted by the pre-trained model to obtain the speaker consistency loss; calculating the reconstruction loss based on the L1 norm between the reconstructed audio signal and the spectrogram of the audio signal of the input data; and determining the second decoupling stage loss based on the second feature decoupling loss, consistency loss, and reconstruction loss.

[0110] Among the various loss functions, the speaker consistency loss is expressed as:

[0111]

[0112] Among them, f s To learn speaker features, Speaker features extracted for the pre-trained model;

[0113] Vocoder-independent feature f a With speaker characteristics f s The second characteristic of decoupling loss:

[0114]

[0115] Speech reconstruction loss:

[0116]

[0117] Where, x sp and These are the spectrograms of the input audio signal and the reconstructed audio signal, respectively.

[0118] In one implementation, meta-learning is performed on the initial trained network model based on the training set to determine the target trained network model, including:

[0119] Step S2021: Set all parameters in the initial training network model except for those in the vocoder-independent encoder and the spoofing classifier to a frozen state, and determine the parameter-frozen network model;

[0120] Step S2022: Perform meta-learning on the parameter-frozen network model based on the training set to determine the target training network model.

[0121] In simple terms, the deep audio forgery detection network model uses the forgery classification output by the forgery classifier based on vocoder-independent features as the audio classification corresponding to the model's input data. To obtain more accurate audio classification and detection results, all parameters in the initial training model, except for the parameters of the vocoder-independent encoder and the forgery classifier, are frozen, resulting in a parameter-frozen network model. Meta-learning is then performed on the parameter-frozen network model to further optimize the parameters of the vocoder-independent encoder and the forgery classifier, thereby improving the model's generalization ability and enabling it to perform well in different environments.

[0122] In one implementation, meta-learning is performed on the parameter-frozen network model based on the training set to determine the target training network model, including:

[0123] Step S20221: Dynamically divide the training set into a meta-training set and a meta-test set;

[0124] Step S20222: Input the meta-training set into the parameter-frozen network model and determine the meta-training loss;

[0125] Step S20223: Input the meta-test set into the parameter-frozen network model to determine the meta-test loss;

[0126] Step S20224: Determine the meta-learning loss function based on the meta-training loss and the meta-testing loss, and optimize the parameters in the vocoder-independent encoder and the spoofing classifier in the parameter-frozen network model based on the meta-learning loss function to determine the target training network model.

[0127] Specifically, the training set (data domain) is dynamically divided into a meta-training set and a meta-test set through a pre-defined meta-learning module (vocoder-agnostic meta-learning, VAML). The meta-training set is input into the parameter-frozen network model, and the loss is calculated based on the target loss function of the parameter-frozen network model. The parameters of the vocoder-independent encoder and spoofing classifier in the parameter-frozen network model are then updated to obtain the adapted model. The performance of the adapted model is then evaluated on the meta-test set, and the cloud test loss is calculated. Finally, a two-layer optimization is performed by combining the meta-training loss and the meta-test loss to improve the model's generalization ability to unknown vocoders.

[0128] Step S400: Obtain the audio to be detected, input the audio to be detected into the target training network model, and determine the audio category corresponding to the audio to be detected.

[0129] Specifically, the target training network model is a pre-trained audio deep forgery detection network model. The audio to be detected is input into the target training network model, and a feature extractor extracts features from the audio to obtain global features. A vocoder-independent encoder then extracts features from the global features to obtain a vocoder-independent encoder. A forgery classifier performs classification detection based on the vocoder-independent features to obtain the true / false category (the probability that the audio is forged). The true / false category output by the forgery classifier is used as the audio classification corresponding to the audio to be detected.

[0130] The training of the audio deep fakery detection network model was implemented using PyTorch 2.0 (an open-source Python machine learning library), and all experiments were conducted on an NVIDIA A100 GPU. A publicly available dataset containing real audio and four fake audio sources (such as Voc.v4) was used as the training set, and other publicly available datasets containing unknown vocoders (such as ASVspoof2019 LA, 21DF, WaveFake, etc.) were used as the test set. The vocoder types and real speech sources in the test set were not exactly the same as those in the training set. During both training and testing, the input audio was resampled to 16kHz and cropped to 4 seconds, using the Adam optimizer with a batch size of 8 and a learning rate of 5×10⁻⁶. -5 The number of epochs during the training phase can be set to 100. Detection performance is primarily evaluated using equal error rate (EER).

[0131] This embodiment trains a network model based on decoupling learning and meta-learning on a training dataset containing various vocoder forgery samples, thereby obtaining a network model capable of accurately determining whether audio has been forged. Leveraging the general forgery traces learned through two-layer unwrapping and the generalization ability gained through meta-learning, the trained model can efficiently detect forged audio generated by unknown vocoders as well as audio in natural scenes. This plays a crucial role in improving the performance of deep forgery detection in audio and overcoming the core problems of poor generalization and robustness in current research.

[0132] Based on the above embodiments, the present invention also provides an audio depth forgery detection device, such as... Figure 3 As shown, the device includes:

[0133] Model building module 01 is used to build an audio deep fake detection network model;

[0134] Decoupling training module 02 is used to acquire a training set, perform decoupling phase training on the audio depth forgery detection network model based on the training set, and determine the initial training network model.

[0135] Meta-learning module 03 is used to perform meta-learning on the initial trained network model based on the training set to determine the target trained network model;

[0136] The audio detection module 04 is used to acquire the audio to be detected, input the audio to be detected into the target training network model, and determine the audio category corresponding to the audio to be detected.

[0137] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 4 As shown, the terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an audio deepfake detection method. The display screen can be a liquid crystal display (LCD) or an e-ink display.

[0138] Those skilled in the art will understand that Figure 4 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0139] In one implementation, the terminal's memory stores one or more programs, and these programs are configured to be executed by one or more processors, and the programs contain instructions for performing an audio depth spoofing detection method.

[0140] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0141] In summary, this invention discloses an audio deepfake detection method, apparatus, terminal, and storage medium. The method involves constructing an audio deepfake detection network model; obtaining a training set; performing decoupling training on the audio deepfake detection network model based on the training set to determine an initial training network model; performing meta-learning on the initial training network model based on the training set to determine a target training network model; obtaining the audio to be detected; inputting the audio to be detected into the target training network model to determine the audio category corresponding to the audio to be detected. Because this invention employs meta-learning in the training phase of the audio deepfake detection network model, forcing the audio deepfake detection network model to learn more generalized and universal knowledge, it can effectively solve the problem of poor detection performance caused by model overfitting when facing unknown vocoders and natural scenes in existing technologies.

[0142] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for detecting deepfake audio, characterized in that, The method includes: Construct a network model for detecting deep audio forgery; Obtain a training set, and perform decoupling phase training on the audio deep forgery detection network model based on the training set to determine the initial training network model; Meta-learning is performed on the initial trained network model based on the training set to determine the target trained network model; Acquire the audio to be detected, input the audio to be detected into the target training network model, and determine the audio category corresponding to the audio to be detected; The audio deepfake detection network model includes a first feature extractor, a second feature extractor, an adversarial training decoupling learning module, and a reconstruction decoupling module. The adversarial training decoupling learning module includes a vocoder-specific encoder and a vocoder-independent encoder. The audio deepfake detection network model is trained in a decoupling phase based on the training set to determine the initial training network model, including: The first feature extractor is used to extract features from the training set to determine the first global feature. The first global feature is encoded by the vocoder-specific encoder to determine the vocoder-specific feature; The first global feature is encoded by the vocoder-independent encoder to determine the vocoder-independent feature; The loss of the first decoupling stage is determined based on the vocoder-specific features and the vocoder-independent features. The training set is used to extract features through the second feature extractor to determine the second global feature; The reconstruction and decoupling module performs feature decoupling on the second global feature to determine the synthesized irrelevant features. The loss of the second decoupling stage is determined based on the vocoder-independent features and the synthesis-independent features; A target loss function is constructed based on the first decoupling stage loss and the second decoupling stage loss, and the parameters of the audio depth forgery detection network model are optimized based on the target loss function to determine the initial training network model.

2. The audio depth forgery detection method according to claim 1, characterized in that, The adversarial training decoupling learning module further includes a domain classifier, a spoofing classifier, and a discriminator with gradient inversion. It determines the loss for the first decoupling stage based on the vocoder-specific features and the vocoder-independent features, including: The domain classifier classifies the vocoder based on its specific features to determine the first vocoder category. The forgery classifier classifies forgeries based on vocoder-independent features to determine the authenticity category; The discriminator classifies the second vocoder category based on the vocoder-independent features. The loss of the first decoupling stage is determined based on the first vocoder category, the true / false category, and the second vocoder category.

3. The audio depth forgery detection method according to claim 1, characterized in that, The reconstruction decoupling module includes a speaker encoder and a content encoder. Based on the reconstruction decoupling module, the second global feature is decoupled to determine synthesized irrelevant features, including: The second global feature is encoded using the speaker encoder to determine the speaker feature; The second global feature is encoded using the content encoder to determine the content feature; The synthesis-irrelevant features are determined based on the speaker features and the content features.

4. The audio depth forgery detection method according to claim 3, characterized in that, The reconstruction and decoupling module further includes an audio generator, which determines the loss of the second decoupling stage based on the vocoder-independent features and the synthesis-independent features, including: The speaker encoder and the vocoder-independent features are subjected to adaptive instance normalization to determine the fused features; The audio generator reconstructs the audio signal based on the fusion features and the content features to determine the reconstructed audio signal. The loss of the second decoupling stage is determined based on the vocoder-independent features, the synthesis-independent features, and the reconstructed audio signal. The parameters of the second feature extractor, the content encoder, and the audio generator are frozen.

5. The audio depth forgery detection method according to claim 2, characterized in that, Meta-learning is performed on the initial trained network model based on the training set to determine the target trained network model, including: All parameters in the initial trained network model, except those in the vocoder-independent encoder and the spoofing classifier, are set to a frozen state to determine the parameter-frozen network model. Meta-learning is performed on the parameter-frozen network model based on the training set to determine the target training network model.

6. The audio depth forgery detection method according to claim 5, characterized in that, Meta-learning is performed on the parameter-frozen network model based on the training set to determine the target training network model, including: The training set is dynamically divided into a meta-training set and a meta-test set; The meta-training set is input into the parameter-freezing network model to determine the meta-training loss; The meta-test set is input into the parameter-frozen network model to determine the meta-test loss; The meta-learning loss function is determined based on the meta-training loss and the meta-testing loss. Based on the meta-learning loss function, the parameters in the vocoder-independent encoder and the spoofing classifier in the parameter-frozen network model are optimized to determine the target training network model.

7. An audio depth forgery detection device, characterized in that, The device includes: The model building module is used to build a network model for detecting deep audio forgery. The decoupled training module is used to acquire a training set, perform decoupled phase training on the audio deep forgery detection network model based on the training set, and determine the initial training network model. The meta-learning module is used to perform meta-learning on the initial trained network model based on the training set to determine the target trained network model; The audio detection module is used to acquire the audio to be detected, input the audio to be detected into the target training network model, and determine the audio category corresponding to the audio to be detected. The audio deepfake detection network model includes a first feature extractor, a second feature extractor, an adversarial training decoupling learning module, and a reconstruction decoupling module. The adversarial training decoupling learning module includes a vocoder-specific encoder and a vocoder-independent encoder. The audio deepfake detection network model is trained in a decoupling phase based on the training set to determine the initial training network model, including: The first feature extractor is used to extract features from the training set to determine the first global feature. The first global feature is encoded by the vocoder-specific encoder to determine the vocoder-specific feature; The first global feature is encoded by the vocoder-independent encoder to determine the vocoder-independent feature; The loss of the first decoupling stage is determined based on the vocoder-specific features and the vocoder-independent features. The training set is used to extract features through the second feature extractor to determine the second global feature; The reconstruction and decoupling module performs feature decoupling on the second global feature to determine the synthesized irrelevant features. The loss of the second decoupling stage is determined based on the vocoder-independent features and the synthesis-independent features; A target loss function is constructed based on the first decoupling stage loss and the second decoupling stage loss, and the parameters of the audio depth forgery detection network model are optimized based on the target loss function to determine the initial training network model.

8. A terminal, characterized in that, The terminal includes a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the audio depth forgery detection method as described in any one of claims 1-6; the processors are used to execute the programs.

9. A computer-readable storage medium storing a plurality of instructions thereon, characterized in that, The instructions are loaded and executed by the processor to implement the steps of the audio depth forgery detection method according to any one of claims 1-6.

Images