Audio deepfake detection method and device, computer device, and storage medium

Abstract

The application discloses an audio deepfake detection method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining prompt audio data and to-be-detected audio data; performing feature extraction on the prompt audio data and the to-be-detected audio data respectively to obtain prompt audio features and to-be-detected audio features; taking the prompt audio features and corresponding labels as context prompt information, and predicting the to-be-detected audio features by using a preset prediction model; and determining whether the to-be-detected sample data is a fake audio based on a prediction result. In the embodiment of the application, the generalization ability of an unknown fake method is improved by using a context learning ability improvement model, and in the model discrimination test sample, a certain prompt sample is given in a context learning manner, which helps the model to better understand the fake audio features and improves the detection accuracy.

Description

Technical Field

[0001] This application relates to the field of audio detection technology, and in particular to an audio deepfake detection method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the rapid development of deepfake technology, the quality of forged audio has reached a level that is difficult to distinguish. Furthermore, the proliferation of forgery methods poses a challenge to the generalization ability of existing forgery detection methods. Existing audio forgery detection techniques often rely on specific forgery methods or models, lacking strong generalization ability across different forgery techniques. Many detection methods depend on large-scale labeled data for training, leading to poor generalization in data-scarce scenarios. Existing models perform poorly when dealing with cross-domain forgery types or novel forgery methods. Summary of the Invention

[0003] Therefore, it is necessary to provide an audio deepfake detection method, apparatus, computer equipment, and storage medium to address the above-mentioned technical problems and solve at least one of the problems existing in the prior art.

[0004] Firstly, an audio deepfake detection method is provided, including:

[0005] Acquire the prompt audio data and the audio data to be detected;

[0006] Feature extraction is performed on the prompt audio data and the audio data to be detected respectively to obtain the prompt audio features and the audio features to be detected;

[0007] The audio features and corresponding tags are used as contextual cue information, and the audio features to be detected are predicted using a preset prediction model.

[0008] Based on the prediction results, it is determined whether the sample data to be detected is fake audio.

[0009] In one embodiment, the step of extracting prompt audio features and detection audio features based on the prompt sample data and the detection sample data includes:

[0010] The prompt audio data and the audio data to be detected are respectively converted into Mel spectrograms;

[0011] Based on the Mel spectrogram, advanced audio features are generated as both the cue audio features and the audio features to be detected.

[0012] In one embodiment, converting the prompt sample data and the sample data to be detected into a Mel spectrogram includes:

[0013] Perform Fourier transforms on the prompt sample data and the test sample data respectively to obtain the spectrum;

[0014] The Mel spectrum is obtained by filtering the spectrum using a Mel filter.

[0015] In one embodiment, performing Fourier transforms on the prompt sample data and the sample data to be detected respectively to obtain a spectrum includes:

[0016] The prompt audio data and the audio data to be detected are sampled and normalized respectively;

[0017] The sampled and normalized prompt audio data and the audio data to be detected are filtered separately.

[0018] The filtered prompt audio data and the audio data to be detected are respectively processed into frames;

[0019] Windowing is applied to each frame of prompt audio data and each frame of audio data to be detected, and Fourier transform is performed on the windowed prompt audio data and each frame of audio data to be detected to obtain the spectrogram.

[0020] In one embodiment, generating high-level audio features based on the Mel spectrogram includes:

[0021] The Mel spectrograms are input into the convolutional module and the Transformer module for feature encoding. Local features are extracted by the convolutional module, and global features are extracted by the Transformer module.

[0022] The local and global features are fused, normalized, and residually connected to generate the high-level audio features.

[0023] In one embodiment, the step of using the prompt audio features and corresponding tags as contextual prompt information to predict the audio features to be detected includes:

[0024] The audio features and corresponding labels are used as contextual cue information and are input into a non-autoregressive model along with the audio features to be detected. The non-autoregressive model is used to conditionally model the audio data to be detected in order to capture fake audio in the audio data to be tested.

[0025] In one embodiment, before predicting the audio features to be detected using a preset prediction model, the method further includes:

[0026] Construct a training audio sample dataset, which includes cue samples and training samples;

[0027] Multiple prompt samples are selected from the training audio sample dataset for feature extraction to obtain multiple prompt features. Initial training samples are selected from the training audio sample dataset for feature extraction to obtain training features.

[0028] Each of the aforementioned prompt features and its corresponding label is used as contextual prompt information, and the training features are predicted using an initial prediction model;

[0029] Based on the preset loss function, calculate the loss value of this prediction result;

[0030] If the loss value is greater than a preset threshold, the next training sample is selected from the training audio sample dataset for feature extraction and prediction until the preset convergence condition is met, and the preset prediction model is obtained.

[0031] Secondly, an audio depth forgery detection device is provided, comprising:

[0032] The audio data acquisition unit is used to acquire prompt audio data and audio data to be detected;

[0033] An audio feature extraction unit is used to extract features from the prompt audio data and the audio data to be detected, respectively, to obtain prompt audio features and audio features to be detected.

[0034] The prediction unit is used to predict the audio features to be detected by using the prompt audio features and the corresponding labels as contextual prompt information;

[0035] The fake audio detection unit is used to determine whether the sample data to be detected is fake audio based on the prediction result.

[0036] Thirdly, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and running on the processor, wherein the processor, when executing the computer-readable instructions, implements the audio depth spoofing detection method as described above.

[0037] Fourthly, a readable storage medium is provided that stores computer-readable instructions thereon, which, when executed by a processor, implement the audio depth forgery detection method described above.

[0038] The aforementioned audio deepfake detection method, apparatus, computer device, and storage medium, implemented as follows, include: acquiring prompt audio data and audio data to be detected; extracting features from the prompt audio data and the audio data to be detected, respectively, to obtain prompt audio features and audio features to be detected; using the prompt audio features and corresponding labels as contextual prompt information, and predicting the audio features to be detected using a preset prediction model; and determining whether the sample data to be detected is fake audio based on the prediction result. In this embodiment, context learning enhances the model's generalization ability to unknown forgery methods, and by providing certain prompt samples when the model judges test samples, it helps the model better understand forgery features and improves detection accuracy. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of a network architecture for an audio depth spoofing detection method according to an embodiment of this application;

[0041] Figure 2 This is a flowchart illustrating an audio depth forgery detection method according to one embodiment of this application;

[0042] Figure 3 This is a schematic diagram of a model architecture of the Wav2Vec2-BERT model in one embodiment of this application;

[0043] Figure 4 This is a schematic diagram of an audio depth forgery detection device according to an embodiment of this application;

[0044] Figure 5 This is a schematic diagram of a computer device according to one embodiment of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] In one embodiment, such as Figure 1As shown, an audio depth forgery detection method is provided, including the following steps:

[0047] In step S110, the prompt audio data and the audio data to be detected are acquired;

[0048] In this embodiment of the application, the prompt audio data and the audio data to be detected may be audio signals.

[0049] Optionally, audio signals from multiple languages ​​and styles (such as audiobooks and dialogues) and various forgery methods can be collected to construct a training sample dataset, ensuring the diversity of real and forged audio samples to improve the model's generalization ability. Existing publicly available datasets can also be used, such as ASVspoof2019, MLAAD, Wavefake, LibriSeVoc, and GigaSpeech. Among them, ASVspoof2019, MLAAD, Wavefake, and LibriSeVoc are authoritative datasets specifically for audio forgery detection, while GigaSpeech provides rich real audio samples, significantly improving the diversity of real samples and laying a solid foundation for training the forgery detection model. Then, at least one training sample can be randomly extracted from the training sample dataset as cue audio data. This audio data to be detected can be the audio signal entered by the user that needs to be checked for forged audio. Understandably, using multiple cue audio data to construct contextual information provides more comprehensive information and higher accuracy.

[0050] In step S120, feature extraction is performed on the prompt audio data and the audio data to be detected, respectively, to obtain prompt audio features and audio features to be detected;

[0051] In this embodiment of the application, a feature extraction model can be constructed, and the prompt audio data and the audio data to be detected can be input into the feature extraction model respectively, so as to extract the prompt audio features and the audio features to be detected through the feature extraction model.

[0052] See Figure 3Optionally, the feature extraction model can be a pre-trained Wav2Vec2-BERT model, which consists of three main modules: a Mel spectrogram extractor connected in sequence, and a Convolution-enhanced Transformer speech recognition model. This model includes convolutional modules and Transformer modules. Taking cue audio data as an example, the cue audio data is input into the Wav2Vec2-BERT model. First, the Mel spectrogram extractor converts the cue audio data into a Mel spectrogram to extract the temporal and frequency features of the audio. Then, the Convolution-enhanced Transformer speech recognition model encodes the features of the Mel spectrogram, capturing both local and global features by combining the convolutional and Transformer modules to generate high-level audio features, i.e., cue audio features. Similarly, the audio data to be detected can also undergo feature extraction in the above manner to obtain the audio features to be detected.

[0053] The Wav2Vec2-BERT model demonstrates exceptional generalization capabilities as a foundational speech model across various tasks, with a particular strength in deep audio forgery detection. Compared to traditional audio models, it exhibits greater stability across cross-language and cross-dataset scenarios. Its outstanding generalization ability is primarily attributed to its large-scale multilingual pre-training data and self-supervised learning design. Specifically, the Wav2Vec2-BERT model is built by pre-training on 4.5 million hours of unlabeled audio data, covering over 143 languages. This large-scale, diverse dataset provides the model with extensive opportunities to learn speech patterns, ensuring its ability to extract highly generalizable and discriminative features from speech signals. Its ability to capture subtle features of audio forgery is largely due to its powerful spectral analysis and temporal modeling capabilities. Forged audio often exhibits subtle anomalies in the spectrum, such as distortion of high-frequency signals or unnatural low-frequency energy distribution. Through its deep convolutional encoder, the Wav2Vec2-BERT model can detect these subtle inconsistencies. Furthermore, within the time domain, forged audio may exhibit signal discontinuity, waveform repetition, or abnormal time intervals. The Wav2Vec2-BERT model, utilizing the masking modeling mechanism in self-supervised learning, can efficiently capture these temporal forgery characteristics. More importantly, the BERT module of the Wav2Vec2-BERT model can extract deep semantic features, thereby helping to detect inconsistencies between speech content and background context. For example, in forged audio, the emotion and tone of the speech may contradict the context; this semantic conflict is a feature that forged audio is difficult to avoid. Combining the analysis of physical signals, temporal patterns, and semantic features, the Wav2Vec2-BERT model exhibits extremely high sensitivity to traces produced by different forgery techniques. By simply fine-tuning Wav2Vec2-BERT and utilizing a simple fully connected layer classifier, strong forgery detection performance can be achieved. Therefore, by extracting audio features based on the pre-trained Wav2Vec2-BERT model and fine-tuning it with a small learning rate, it can be better adapted to audio forgery detection tasks. By extracting audio features using the Wav2Vec2-BERT model, its sensitivity to audio forgery features can be maintained, and it still performs well under different datasets and forgery techniques.

[0054] It should be noted that the prompt audio data can include multiple prompt audio data, that is, multiple prompt audio data can be acquired at the same time, and feature extraction can be performed on each of them to obtain multiple prompt audio features, which are used as the prediction condition input for the audio to be detected in this case, so as to improve the prediction accuracy of the model.

[0055] In step S130, the prompt audio features and corresponding tags are used as contextual prompt information, and the audio features to be detected are predicted by a preset prediction model;

[0056] In this embodiment, the prompt audio data can be labeled manually, for example, as real or fake. After extracting the prompt audio features, the prompt audio features and their corresponding labels can be combined as contextual prompt information.

[0057] See Figure 2 Optionally, the preset prediction model can be a non-autoregressive (NAR) model based on the LLaMA architecture. The cue audio features and their corresponding ground truth labels together constitute contextual cue information, providing prior knowledge for the discrimination task of the sample to be detected. The NAR model, through a contextual modeling mechanism, tightly integrates the semantic information of the cue sample with the features of the test sample, thereby improving the ability to capture forged audio features, making it particularly suitable for identifying audio generated by complex forgery techniques. Since LLaMA itself, as an autoregressive (AR) generation model, is not suitable for deep audio forgery detection tasks, as this task does not involve the generation process but requires the discrimination and classification of audio signals, the LLaMA model structure is modified to a NAR model to avoid the computational and time complexity issues caused by the autoregressive generation process, while retaining the model's powerful ability in contextual modeling. The NAR architecture enables the model to process audio sequences efficiently in parallel, improving its ability to model long-term dependencies, making it more suitable for forged audio detection tasks.

[0058] In step S140, based on the prediction results, it is determined whether the sample data to be detected is fake audio.

[0059] In this embodiment, contextual clues and the features of the audio to be detected are input into a preset prediction model. The contextual clues are used as conditional inputs to provide prior information for judging the audio data to be detected. For example, <clue audio 1, tag 1, clue audio 2, tag 2, audio to be detected>. In this case, the preset prediction model can automatically generate the tag corresponding to the audio to be detected. This tag can be used as the prediction result output, such as real or fake, thereby determining whether the audio data to be detected is real or fake audio.

[0060] This application provides a method for detecting deepfake audio, comprising: acquiring prompt audio data and audio data to be detected; extracting features from the prompt audio data and the audio data to be detected respectively to obtain prompt audio features and audio features to be detected; using the prompt audio features and corresponding labels as contextual prompt information, and predicting the audio features to be detected using a preset prediction model; and determining whether the sample data to be detected is fake audio based on the prediction result. In this application embodiment, context learning enhances the model's generalization ability to unknown forgery methods, and by providing certain prompt samples when the model judges test samples, it helps the model better understand forgery features and improves detection accuracy.

[0061] In one embodiment of this application, the step of extracting the prompt audio features and the audio features to be detected based on the prompt sample data and the sample data to be detected includes:

[0062] The prompt audio data and the audio data to be detected are respectively converted into Mel spectrograms;

[0063] Based on the Mel spectrogram, advanced audio features are generated as both the cue audio features and the audio features to be detected.

[0064] Optionally, the feature extraction model can be a pre-trained Wav2Vec2-BERT model, which consists of three main modules: a Mel spectrum extractor and a convolution-enhanced Transformer speech recognition model, which includes convolutional modules and Transformer modules. The cue audio data and the audio data to be detected are respectively input into the Mel spectrum extractor, which converts the original audio waveform signal into a Mel spectrogram to extract the temporal and frequency features of the audio. Then, the convolution-enhanced Transformer speech recognition model encodes the Mel spectrogram to generate high-level audio features, which serve as the cue audio features and the audio features to be detected. These cue audio features and the audio features to be detected can provide high-quality input representations for subsequent audio detection tasks.

[0065] In one embodiment of this application, converting the prompt sample data and the sample data to be detected into a Mel spectrogram includes:

[0066] Perform Fourier transforms on the prompt sample data and the test sample data respectively to obtain the spectrum;

[0067] The Mel spectrum is obtained by filtering the spectrum using a Mel filter.

[0068] Optionally, the prompt audio data and the audio data to be detected can be audio waveform signals. First, the input audio waveform signal can be sampled and normalized. Then, a high-pass filter is used to remove low-frequency noise. The resulting audio signal is then segmented into frames to obtain audio frames within a preset time range, such as 20 milliseconds per frame, to ensure that the signal is approximately stable over a short period. Each audio frame is then windowed, for example, with a Hamming window, to reduce edge effects. A Fourier transform (FFT) is then performed on the windowed audio frames to convert the time-domain signal into a frequency-domain signal, thus obtaining a spectrogram. Based on the spectrogram, the square of the spectrum can be calculated to extract the energy distribution of each frequency band. A set of Mel filter banks is then used to weight and sum the spectrum to generate the energy distribution on the Mel frequency scale. Finally, the logarithm of these energy values ​​is taken to obtain the logarithmic Mel spectrum, i.e., the Mel spectrogram. The Mel spectrogram represents the frame order on the time axis and the intensity of the Mel frequency band on the frequency axis.

[0069] In one embodiment of this application, the step of performing Fourier transforms on the prompt sample data and the sample data to be detected respectively to obtain a spectrum includes:

[0070] The prompt audio data and the audio data to be detected are sampled and normalized respectively;

[0071] The sampled and normalized prompt audio data and the audio data to be detected are filtered separately.

[0072] The filtered prompt audio data and the audio data to be detected are respectively processed into frames;

[0073] Windowing is applied to each frame of prompt audio data and each frame of audio data to be detected, and Fourier transform is performed on the windowed prompt audio data and each frame of audio data to be detected to obtain the spectrogram.

[0074] Optionally, the prompt audio data and the audio data to be detected can be audio waveform signals. First, the input audio waveform signal can be sampled and normalized. Then, a high-pass filter is used to remove low-frequency noise. The resulting audio signal is then segmented into frames to obtain audio frames within a preset time range, such as 20 milliseconds per frame, to ensure that the signal is approximately stable over a short period. Each audio frame is then windowed, for example, using a Hamming window, to reduce edge effects. Finally, a Fourier transform (FFT) is performed on the windowed audio frames to convert the time-domain signal into a frequency-domain signal, thereby obtaining a spectrum.

[0075] In one embodiment of this application, generating high-level audio features based on the Mel spectrogram includes:

[0076] The Mel spectrograms are input into the convolutional module and the Transformer module for feature encoding. Local features are extracted by the convolutional module, and global features are extracted by the Transformer module.

[0077] The local and global features are fused, normalized, and residually connected to generate the high-level audio features.

[0078] Optionally, the feature extraction model can be a pre-trained Wav2Vec2-BERT model, which consists of three main modules: a Mel spectrogram extractor and a convolution-enhanced Transformer speech recognition model. This model includes a convolutional module and a Transformer module. The Mel spectrograms of the generated prompt audio data and the audio data to be detected are input into the convolutional and Transformer modules respectively for feature encoding. The convolutional module extracts local features of the audio signal through one-dimensional convolution, capturing spectral changes and transient information within a short time range. The Transformer module uses a multi-head self-attention mechanism to model global temporal dependencies, capturing long-term correlations by assigning weights to each time frame, and embedding temporal information through positional encoding to ensure temporal representation. By alternately stacking convolutional layers and the Transformer module, local and global features can be fused, and high-quality high-level audio feature representations can be generated using normalization and residual connections of fully connected layers. These features can comprehensively characterize the temporal and spectral properties of audio, providing a reliable input representation for the task of detecting fake audio.

[0079] In one embodiment of this application, the step of using the prompt audio features and corresponding tags as contextual prompt information to predict the audio features to be detected includes:

[0080] The audio features and corresponding labels are used as contextual cue information and input along with the audio features to be detected into a non-autoregressive model;

[0081] The non-autoregressive model uses the contextual cue information to conditionally model the audio data to be detected, in order to capture fake audio in the audio data to be tested.

[0082] Optionally, the non-autoregressive model can be a non-autoregressive (NAR) model based on an LLaMA architecture modification. The cue audio features extracted using Wav2Vec2-BERT and the audio features to be detected are input into the NAR model based on the LLaMA architecture modification for modeling. The cue audio features and their corresponding ground truth labels together constitute contextual cue information, providing prior knowledge for the discrimination task of the audio to be detected. Through a contextual modeling mechanism, the NAR model tightly integrates the semantic information of the cue samples with the features of the test samples, thereby improving the ability to capture forged audio features, making it particularly suitable for identifying audio generated by complex forgery techniques.

[0083] The specific prediction process of the non-autoregressive (NAR) model is as follows: The audio features of the prompt audio data, combined with their corresponding labels, and the audio features to be detected are input into the NAR model. The NAR model models the correlation between the prompt sample audio data and the audio data to be detected through an attention mechanism, calculating the semantic similarity and feature distribution relationship between them. Based on this, the features of the prompt sample are embedded into the context vector, forming a global semantic and feature distribution pattern. Subsequently, the features of the audio data to be detected are introduced. The NAR model uses contextual prompt information to conditionally model the audio features of the audio data to be detected, focusing on potential forgery features. During the modeling process, NAR captures possible forgery feature deviations between the test sample and the prompt sample through global semantic comparison and contextual semantic guidance, thereby completing the judgment of the forgery characteristics of the test sample. For example, in the case of <Callergy Audio 1, Label 1, Callergy Audio 2, Label 2, Audio to be Detected>, the preset prediction model can automatically generate the label corresponding to the audio to be detected. This label can be used as the prediction result output, such as real or fake, thus determining whether the audio data to be detected is real or fake audio.

[0084] Because cue audio data provides rich semantic and spoofing audio feature distribution patterns, NAR models can effectively utilize this contextual information to guide the model in capturing potential spoofing features in the audio data to be detected. This context-assisted approach is particularly suitable for handling task scenarios with covert spoofing features. Unlike the stepwise modeling approach of autoregressive models, the NAR architecture focuses more on overall modeling capabilities. By parallelizing the processing of features in cue sample audio data and the audio data to be detected, the NAR model can perform comprehensive modeling in both global semantic and temporal dimensions, making it more suitable for modeling long audio sequences and complex spoofing features in task scenarios. Especially for adversarial audio with different spoofing techniques, the NAR architecture can leverage the constraints provided by the contextual cue samples to enhance the model's generalization ability.

[0085] In one embodiment of this application, before predicting the audio features to be detected using a preset prediction model, the method further includes:

[0086] Construct a training audio sample dataset, which includes cue samples and training samples;

[0087] Multiple prompt samples are selected from the training audio sample dataset for feature extraction to obtain multiple prompt features. Initial training samples are selected from the training audio sample dataset for feature extraction to obtain training features.

[0088] Each of the aforementioned prompt features and its corresponding label is used as contextual prompt information, and the training features are predicted using an initial prediction model;

[0089] Based on the preset loss function, calculate the loss value of this prediction result;

[0090] If the loss value is greater than a preset threshold, the next training sample is selected from the training audio sample dataset for feature extraction and prediction until the preset convergence condition is met, and the preset prediction model is obtained.

[0091] Optionally, audio signals from multiple languages ​​and styles (such as audiobooks, dialogues, etc.) and various forgery methods can be collected to construct a training sample dataset, ensuring the diversity of real and forged audio samples to improve the model's generalization ability. Existing publicly available datasets can also be used, such as ASVspoof2019, MLAAD, Wavefake, LibriSeVoc, and GigaSpeech. Among them, ASVspoof2019, MLAAD, Wavefake, and LibriSeVoc are authoritative datasets specifically for audio forgery detection, while GigaSpeech provides rich real audio samples, significantly improving the diversity of real samples and laying a solid foundation for training the forgery detection model. Diverse data is crucial for improving the model's robustness and generalization ability. One of the challenges of audio forgery detection lies in the diversity and rapid changes in forgery methods. By introducing datasets with multiple languages, styles, and forgery methods, it is ensured that the model can maintain high performance in various forgery scenarios. Furthermore, the diversity of real audio samples (such as those from GigaSpeech) can further enhance the model's ability to distinguish between real and forged audio.

[0092] Then, a model can be built based on the NAR-based LLaMA model, focusing on addressing the need for parallel processing of long sequence features in audio forgery detection tasks. The model's feature extraction part uses a pre-trained Wav2Vec2-BERT model, which can efficiently extract spectral, temporal, and contextual features from audio signals, providing high-quality input for sequence modeling in the NAR model. Through this joint architecture, the model can accurately capture forgery features in audio while maintaining high computational efficiency.

[0093] After building the model, initial training samples can be randomly selected from the training sample dataset, along with multiple prompt samples (e.g., six prompt samples). These are then input into the pre-trained Wav2Vec2-BERT model for feature extraction, yielding multiple prompt audio features and training audio features. These features are then input into the NAR model, using the prompt audio features and their corresponding labels as contextual information to predict whether the initial training samples are real or fake audio. Further, a loss value can be calculated based on the labels of the training samples and the prediction results using a preset loss function, such as the cross-entropy loss function, combined with stochastic gradient descent (SGD) for optimization. If the loss value exceeds a preset threshold, the next training sample can be selected from the training sample dataset, and the above steps are repeated to generate a prediction result. This process is repeated until the preset convergence condition is met, such as the number of iterations reaching a preset number, the loss value falling below a preset threshold, or all training samples in the training sample set having been trained. At this point, the preset prediction model is obtained.

[0094] It should be noted that a test dataset can be established. After the model training is completed, it can be tested based on the test dataset. The model testing process is consistent with the training process. First, the Wav2Vec2-BERT model is used to extract features from the test samples. This ensures that the input features of the test samples are consistent with those of the training samples. Then, each test sample can be combined with several cue samples and input into the NAR model for prediction. The features and labels of the cue samples serve as contextual information to help the model determine whether the test sample is fake audio. The model testing phase uses multiple metrics to evaluate its performance, including accuracy (Acc%), false alarm rate (FAR%), false negative rate (FRR%), equal error rate (EER%), and F1 score. The threshold is determined through experimental results on the validation set. Experimental results show that the model exhibits significant robustness and high accuracy under various forgery techniques, languages, and styles, and can effectively cope with different types of audio forgery techniques.

[0095] In this embodiment, the Wav2Vec2-BERT module can extract low-level features (such as spectrum and temporal patterns) from the original audio and generate high-quality contextual embeddings. The pre-training process of this feature extractor allows the model to gain a deep understanding of multilingual and multi-style audio, thus providing a reliable feature foundation for forgery detection tasks. The contextual learning capability of the LLaMA model is the core of its generalization ability in forgery detection tasks. With the assistance of cue samples and labels, the model can utilize contextual information to capture the features of forged audio, thereby enhancing its adaptability to unknown forgery techniques. The contextual learning module not only improves the accuracy of forgery detection but also significantly enhances the model's robustness in different scenarios and language environments.

[0096] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0097] In one embodiment, an audio depth spoofing detection device is provided, which corresponds one-to-one with the audio depth spoofing detection method described in the above embodiments. For example... Figure 4 As shown, the audio deepfake detection device includes an audio data acquisition unit 10, an audio feature extraction unit 20, a prediction unit 30, and a fake audio judgment unit 40. Detailed descriptions of each functional module are as follows:

[0098] The audio data acquisition unit 10 is used to acquire prompt audio data and audio data to be detected.

[0099] The audio feature extraction unit 20 is used to extract features from the prompt audio data and the audio data to be detected, respectively, to obtain prompt audio features and audio features to be detected.

[0100] Prediction unit 30 is used to predict the audio features to be detected by using the prompt audio features and the corresponding tags as contextual prompt information;

[0101] The fake audio detection unit 40 is used to determine whether the sample data to be detected is fake audio based on the prediction result.

[0102] In one embodiment of this application, the audio feature extraction unit 20 is used for:

[0103] The prompt audio data and the audio data to be detected are respectively converted into Mel spectrograms;

[0104] Based on the Mel spectrogram, advanced audio features are generated as both the cue audio features and the audio features to be detected.

[0105] In one embodiment of this application, the audio feature extraction unit 20 is used for:

[0106] Perform Fourier transforms on the prompt sample data and the test sample data respectively to obtain the spectrum;

[0107] The Mel spectrum is obtained by filtering the spectrum using a Mel filter.

[0108] In one embodiment of this application, the audio feature extraction unit 20 is used for:

[0109] The prompt audio data and the audio data to be detected are sampled and normalized respectively;

[0110] The sampled and normalized prompt audio data and the audio data to be detected are filtered separately.

[0111] The filtered prompt audio data and the audio data to be detected are respectively processed into frames;

[0112] Windowing is applied to each frame of prompt audio data and each frame of audio data to be detected, and Fourier transform is performed on the windowed prompt audio data and each frame of audio data to be detected to obtain the spectrogram.

[0113] In one embodiment of this application, the audio feature extraction unit 20 is used for:

[0114] The Mel spectrograms are input into the convolutional module and the Transformer module for feature encoding. Local features are extracted by the convolutional module, and global features are extracted by the Transformer module.

[0115] The local and global features are fused, normalized, and residually connected to generate the high-level audio features.

[0116] In one embodiment of this application, the prediction unit 30 is further configured to:

[0117] The audio features and corresponding labels are used as contextual cue information and input along with the audio features to be detected into a non-autoregressive model;

[0118] The non-autoregressive model uses the contextual cue information to conditionally model the audio data to be detected, in order to capture fake audio in the audio data to be tested.

[0119] In one embodiment of this application, the apparatus further includes: a prediction model training unit, used for:

[0120] Construct a training audio sample dataset, which includes cue samples and training samples;

[0121] Multiple prompt samples are selected from the training audio sample dataset for feature extraction to obtain multiple prompt features. Initial training samples are selected from the training audio sample dataset for feature extraction to obtain training features.

[0122] Each of the aforementioned prompt features and its corresponding label is used as contextual prompt information, and the training features are predicted using an initial prediction model;

[0123] Based on the preset loss function, calculate the loss value of this prediction result;

[0124] If the loss value is greater than a preset threshold, the next training sample is selected from the training audio sample dataset for feature extraction and prediction until the preset convergence condition is met, and the preset prediction model is obtained.

[0125] In this embodiment, the Wav2Vec2-BERT module can extract low-level features (such as spectrum and temporal patterns) from the original audio and generate high-quality contextual embeddings. The pre-training process of this feature extractor allows the model to gain a deep understanding of audio from multiple languages ​​and styles, thus providing a reliable feature foundation for forgery detection tasks. The contextual learning capability of the LLaMA model is the core of its generalization ability in forgery detection tasks. With the assistance of cue samples and labels, the model can utilize contextual information to capture the features of forged audio, thereby enhancing its adaptability to unknown forgery techniques. The contextual learning module not only improves the accuracy of forgery detection but also significantly enhances the model's robustness in different scenarios and language environments.

[0126] Specific limitations regarding the audio deepfake detection device can be found in the limitations of the audio deepfake detection method described above, and will not be repeated here. Each module in the aforementioned audio deepfake detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0127] In one embodiment, a computer device is provided, which may be a terminal device, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a readable storage medium storing computer-readable instructions. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer-readable instructions implement an audio deepfake detection method. The readable storage medium provided in this embodiment includes both non-volatile and volatile readable storage media.

[0128] In this application embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, it implements the steps of the audio depth spoofing detection method described above.

[0129] In one embodiment of the application, a readable storage medium is provided, which stores computer-readable instructions that, when executed by a processor, implement the steps of the audio depth spoofing detection method described above.

[0130] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0132] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for detecting deepfake audio, characterized in that, The method includes: Acquire the prompt audio data and the audio data to be detected; Feature extraction is performed on the prompt audio data and the audio data to be detected to obtain prompt audio features and audio features to be detected, respectively. This includes: converting the prompt audio data and the audio data to be detected into Mel spectrograms; generating high-level audio features based on the Mel spectrograms as the prompt audio features and audio features to be detected, wherein the high-level audio features are obtained by: inputting the Mel spectrograms into a convolutional module and a Transformer module for feature encoding; extracting local features through the convolutional module and extracting global features through the Transformer module; fusing the local features and the global features, and performing normalization and residual connections to generate the high-level audio features. The audio features and corresponding tags are used as contextual cue information, and the audio features to be detected are predicted using a preset prediction model. Based on the prediction results, determine whether the audio data to be detected is fake audio; Before predicting the audio features to be detected using a preset prediction model, the method further includes: Construct a training audio sample dataset, which includes cue samples and training samples; Multiple prompt samples are selected from the training audio sample dataset for feature extraction to obtain multiple prompt features. Initial training samples are selected from the training audio sample dataset for feature extraction to obtain training features. Each of the aforementioned prompt features and its corresponding label is used as contextual prompt information, and the training features are predicted using an initial prediction model; Based on the preset loss function, calculate the loss value of this prediction result; If the loss value is greater than a preset threshold, the next training sample is selected from the training audio sample dataset for feature extraction and prediction until the preset convergence condition is met, and the preset prediction model is obtained.

2. The audio depth forgery detection method as described in claim 1, characterized in that, The step of converting the prompt audio data and the audio data to be detected into Mel spectrograms respectively includes: Perform Fourier transforms on the prompt audio data and the audio data to be detected respectively to obtain a spectrum. The Mel spectrum is obtained by filtering the spectrum using a Mel filter.

3. The audio depth forgery detection method as described in claim 2, characterized in that, The step of performing Fourier transforms on the prompt audio data and the audio data to be detected respectively to obtain a spectrum includes: The prompt audio data and the audio data to be detected are sampled and normalized respectively; The sampled and normalized prompt audio data and the audio data to be detected are filtered separately. The filtered prompt audio data and the audio data to be detected are respectively processed into frames; Windowing is applied to each frame of prompt audio data and each frame of audio data to be detected, and Fourier transform is performed on the windowed prompt audio data and each frame of audio data to be detected to obtain the spectrogram.

4. The audio deepfake detection method as described in claim 1, characterized in that, The step of using the prompt audio features and corresponding tags as contextual prompt information to predict the audio features to be detected includes: The audio features and corresponding tags are used as contextual cue information and are input into a non-autoregressive model along with the audio features to be detected. The non-autoregressive model is used to conditionally model the audio data to be detected in order to capture fake audio in the audio data to be detected.

5. An audio depth forgery detection device, characterized in that, The device includes: The audio data acquisition unit is used to acquire prompt audio data and audio data to be detected; An audio feature extraction unit is configured to extract features from the prompt audio data and the audio data to be detected, respectively, to obtain prompt audio features and audio features to be detected. The unit includes: converting the prompt audio data and the audio data to be detected into Mel spectrograms; generating high-level audio features based on the Mel spectrograms, which serve as the prompt audio features and audio features to be detected. The high-level audio features are obtained by: inputting the Mel spectrograms into a convolutional module and a Transformer module for feature encoding; extracting local features through the convolutional module and global features through the Transformer module; fusing the local features and global features, and performing normalization and residual connections to generate the high-level audio features. The prediction unit is used to use the prompt audio features and corresponding tags as contextual prompt information, and to predict the audio features to be detected using a preset prediction model; A fake audio detection unit is used to determine whether the audio data to be detected is fake audio based on the prediction result; The device further includes a prediction model training unit, used for: Construct a training audio sample dataset, which includes cue samples and training samples; Multiple prompt samples are selected from the training audio sample dataset for feature extraction to obtain multiple prompt features. Initial training samples are selected from the training audio sample dataset for feature extraction to obtain training features. Each of the aforementioned prompt features and its corresponding label is used as contextual prompt information, and the training features are predicted using an initial prediction model; Based on the preset loss function, calculate the loss value of this prediction result; If the loss value is greater than a preset threshold, the next training sample is selected from the training audio sample dataset for feature extraction and prediction until the preset convergence condition is met, and the preset prediction model is obtained.

6. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and running on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the audio depth spoofing detection method as described in any one of claims 1 to 4.

7. A readable storage medium having computer-readable instructions stored thereon, characterized in that, When the computer-readable instructions are executed by a processor, they implement the audio depth spoofing detection method as described in any one of claims 1 to 4.

Images