Audio depth forgery detection method and device, computer equipment and storage medium

Abstract

The invention discloses an audio deep forgery detection method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining prompt audio data and to-be-detected audio data; performing feature extraction on the prompt audio data and the to-be-detected audio data to obtain prompt audio features and to-be-detected audio features; taking the prompt audio feature and a corresponding label as context prompt information, and predicting the to-be-detected audio feature through a preset prediction model; and based on a prediction result, determining whether the to-be-detected sample data is a forged audio. In the embodiment of the invention, the generalization ability of the model to an unknown counterfeiting method is improved through the context learning ability, and a certain prompt sample is given through the context learning mode when the model discriminates the test sample, so that the model is helped to better understand the characteristics of the forged audio, and the detection accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of audio detection technology, and in particular to an audio deep fake detection method, apparatus, computer equipment and storage medium. Background Art

[0002] With the rapid development of deepfake technology, the quality of fake audio has reached a level that is difficult to distinguish. In addition, the endless number of fake methods has also brought generalization challenges to existing fake detection methods. Existing fake audio detection technologies often rely on specific fake methods or models and lack strong generalization capabilities for different fake technologies. A large number of detection methods rely on large-scale annotated data for training, which makes it difficult to generalize in scenarios where data is scarce. Existing models perform poorly when dealing with cross-domain fake types or new fake methods. Summary of the invention

[0003] Based on this, it is necessary to provide an audio deep fake detection method, device, computer equipment and storage medium to address the above-mentioned technical problems, so as to solve at least one problem existing in the above-mentioned prior art.

[0004] In a first aspect, a method for detecting deep fake audio is provided, comprising:

[0005] Obtain prompt audio data and audio data to be detected;

[0006] Extracting features from the prompt audio data and the audio data to be detected respectively to obtain prompt audio features and audio features to be detected;

[0007] The prompt audio feature and the corresponding label are used as context prompt information, and the audio feature to be detected is predicted by a preset prediction model;

[0008] Based on the prediction result, determine whether the sample data to be detected is forged audio.

[0009] In one embodiment, extracting the prompt audio feature and the audio feature to be detected based on the prompt sample data and the sample data to be detected respectively includes:

[0010] Convert the prompt audio data and the audio data to be detected into Mel-spectrograms respectively;

[0011] Based on the Mel-spectrogram, high-level audio features are generated to serve as the prompt audio features and the audio features to be detected.

[0012] In one embodiment, converting the prompt sample data and the sample data to be detected into a mel-spectrogram includes:

[0013] Performing Fourier transform on the prompt sample data and the sample data to be detected respectively to obtain a frequency spectrum;

[0014] The spectrum graph is filtered by a Mel filter to obtain the Mel spectrum graph.

[0015] In one embodiment, the Fourier transform is performed on the prompt sample data and the sample data to be detected to obtain a frequency spectrum, including:

[0016] Respectively sampling and normalizing the prompt audio data and the audio data to be detected;

[0017] Respectively filtering the prompt audio data and the audio data to be detected after sampling and normalization processing;

[0018] Respectively performing frame processing on the filtered prompt audio data and the audio data to be detected;

[0019] Each frame of prompt audio data and each frame of audio data to be detected are windowed respectively, and Fourier transform is performed on the windowed each frame of prompt audio data and each frame of audio data to be detected to obtain the spectrum diagram.

[0020] In one embodiment, generating high-level audio features based on the Mel-spectrogram includes:

[0021] The Mel-spectrogram is respectively input into a convolution module and a Transformer module for feature encoding, local features are extracted by the convolution module, and global features are extracted by the Transformer module;

[0022] The local features and the global features are fused, normalized and residually connected to generate the high-level audio features.

[0023] In one embodiment, the step of using the prompt audio feature and the corresponding label as context prompt information to predict the audio feature to be detected includes:

[0024] The prompt audio features and corresponding labels are used as context prompt information, and are input into a non-autoregressive model together with the audio features to be detected, so as to conditionally model the audio data to be detected through the non-autoregressive model to capture the forged audio in the audio data to be tested.

[0025] In one embodiment, before predicting the audio feature to be detected by using a preset prediction model, the method further includes:

[0026] Constructing a training audio sample data set, wherein the training audio sample data set includes prompt samples and training samples;

[0027] Selecting a plurality of the prompt samples from the training audio sample data set for feature extraction to obtain a plurality of prompt features, and selecting an initial training sample from the training audio sample data set for feature extraction to obtain a training feature;

[0028] Using each of the prompt features and the corresponding label as context prompt information, predicting the training features through an initial prediction model;

[0029] Based on the preset loss function, calculate the loss value of this prediction result;

[0030] If the loss value is greater than a preset threshold, the next training sample is selected from the training audio sample data set for feature extraction and prediction until the preset convergence condition is met, thereby obtaining the preset prediction model.

[0031] In a second aspect, an audio deep fake detection device is provided, comprising:

[0032] An audio data acquisition unit, used to acquire prompt audio data and audio data to be detected;

[0033] An audio feature extraction unit, used to extract features from the prompt audio data and the audio data to be detected, respectively, to obtain prompt audio features and audio features to be detected;

[0034] A prediction unit, configured to use the prompt audio feature and the corresponding label as context prompt information to predict the audio feature to be detected;

[0035] The forged audio judgment unit is used to determine whether the sample data to be detected is forged audio based on the prediction result.

[0036] In a third aspect, a computer device is provided, comprising a memory, a processor, and computer-readable instructions stored on the memory and running on the processor, wherein the processor implements the audio deep fake detection method as described above when executing the computer-readable instructions.

[0037] In a fourth aspect, a readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the audio deep fake detection method as described above is implemented.

[0038] The above-mentioned audio deep fake detection method, device, computer equipment and storage medium, and its method implementation include: obtaining prompt audio data and audio data to be detected; performing feature extraction on the prompt audio data and the audio data to be detected respectively to obtain prompt audio features and audio features to be detected; using the prompt audio features and corresponding labels as context prompt information, and predicting the audio features to be detected through a preset prediction model; based on the prediction results, determining whether the sample data to be detected is forged audio. In the embodiment of the present application, the generalization ability of the model for unknown forgery methods is improved through context learning capabilities, and through context learning, when the model discriminates test samples, certain prompt samples are given to help the model better understand the forgery features and improve the accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 This is a schematic diagram of a network architecture of an audio deep fake detection method in an embodiment of the present application;

[0041] Figure 2 This is a flow chart of a method for detecting deep audio forgery in one embodiment of the present application;

[0042] Figure 3 This is a schematic diagram of a model architecture of a Wav2Vec2-BERT model in one embodiment of the present application;

[0043] Figure 4 This is a structural diagram of an audio deep fake detection device according to an embodiment of the present application;

[0044] Figure 5 It is a schematic diagram of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] In one embodiment, if Figure 1As shown, a method for detecting deep audio forgery is provided, comprising the following steps:

[0047] In step S110, the prompt audio data and the audio data to be detected are obtained;

[0048] In the embodiment of the present application, the prompt audio data and the audio data to be detected may be audio signals.

[0049] Optionally, audio signals of multiple languages, multiple styles (such as audio books, conversations, etc.) and multiple forgery methods can be collected, and a training sample data set can be constructed to ensure the diversity of real audio samples and forged samples to improve the generalization ability of the model. Existing public data sets such as ASVspoof2019, MLAAD, Wavefake, LibriSeVoc and GigaSpeech can also be used. Among them, ASVspoof2019, MLAAD, Wavefake and LibriSeVoc are authoritative data sets specifically for audio forgery detection, while GigaSpeech provides a wealth of real audio samples, significantly improving the diversity of real samples, and laying a solid foundation for the training of forgery detection models. Then, at least one training sample can be randomly extracted from the training sample data set as prompt audio data. The audio data to be detected can be an audio signal entered by the user that needs to be detected for the presence of forged audio. It can be understood that the use of multiple prompt audio data to construct context information is more comprehensive and more accurate.

[0050] In step S120, feature extraction is performed on the prompt audio data and the audio data to be detected, respectively, to obtain prompt audio features and audio features to be detected;

[0051] In an embodiment of the present application, a feature extraction model may be constructed, and the prompt audio data and the audio data to be detected may be respectively input into the feature extraction model, so as to extract the prompt audio features and the audio features to be detected through the feature extraction model.

[0052] See also Figure 3, Optionally, the feature extraction model can be a pre-trained Wav2Vec2-BERT model, which consists of three main modules. It includes a Mel spectrum extractor and a Transformer speech recognition model (Conformer Speech Encoder) based on convolution enhancement, which is connected in sequence. The Transformer speech recognition model based on convolution enhancement includes a convolution module and a Transformer module. Taking the prompt audio data as an example, the prompt audio data is input into the Wav2Vec2-BERT model. First, the prompt audio data is converted into a Mel spectrum graph by a Mel spectrum extractor to extract the time and frequency features of the audio. Then, the Mel spectrum graph is feature encoded based on the convolution enhanced Transformer speech recognition model, and local and global features are captured by combining the convolution module and the Transformer module to generate high-level audio features, i.e., prompt audio features. Similarly, the audio data to be detected can also be feature extracted in the above manner to obtain the audio features to be detected.

[0053] The Wav2Vec2-BERT model demonstrates excellent generalization capabilities as a speech-based model on a variety of tasks, especially in the audio deepfake detection task. Compared with traditional audio models, it performs more stably in cross-language and cross-dataset scenarios, and its excellent generalization capabilities are mainly due to the design of large-scale multilingual pre-training data and self-supervised learning. Specifically, the Wav2Vec2-BERT model is built by pre-training on 4.5 million hours of unlabeled audio data, covering more than 143 languages. This large and diverse dataset provides the model with a wide range of speech pattern learning opportunities and ensures that it can extract highly general and discriminative features from speech signals. It can capture subtle features of audio forgeries, mainly due to its powerful spectral analysis and time series modeling capabilities. Forged audio often shows subtle anomalies in the spectrum, such as distortion of high-frequency signals or unnatural distribution of low-frequency energy. Through its deep convolutional encoder, the Wav2Vec2-BERT model is able to detect these subtle inconsistencies. In addition, in the time domain, forged audio may have intermittent signals, repeated waveforms, or abnormal time intervals. The Wav2Vec2-BERT model uses the mask modeling mechanism in self-supervised learning to efficiently capture these temporal forgery characteristics. More importantly, the BERT module of the Wav2Vec2-BERT model can extract deep semantic features to help detect inconsistencies between speech content and background context. For example, in forged audio, the emotion and tone of the voice may conflict with the context. This semantic conflict is a feature that is difficult to avoid in forged audio. Combining the analysis of physical signals, temporal patterns, and semantic features, the Wav2Vec2-BERT model is highly sensitive to traces produced by different forgery techniques. By simply fine-tuning Wav2Vec2-BERT and using a simple fully connected layer classifier, a strong forgery detection performance can be achieved. Therefore, audio features are extracted based on the pre-trained Wav2Vec2-BERT model and fine-tuned with a smaller learning rate to make it more suitable for audio forgery detection tasks. By extracting audio features through the Wav2Vec2-BERT model, it can maintain its sensitivity to audio forgery features and still perform well under different data sets and forgery techniques.

[0054] It should be noted that the prompt audio data may include multiple prompt audio data, that is, multiple prompt audio data can be obtained at the same time, and feature extraction can be performed separately to obtain multiple prompt audio features, which are used as prediction condition input for the audio to be detected this time to improve the model prediction accuracy.

[0055] In step S130, the prompt audio feature and the corresponding label are used as context prompt information, and the audio feature to be detected is predicted by a preset prediction model;

[0056] In the embodiment of the present application, the prompt audio data can be manually labeled to set corresponding labels, such as true or false. After the prompt audio features are extracted, the prompt audio features and their corresponding labels can be combined as context prompt information.

[0057] See also Figure 2 Optionally, the preset prediction model can be a non-autoregressive (NAR) model based on the LLaMA architecture. The prompt audio features and their corresponding true labels together constitute contextual prompt information, providing prior knowledge for the discrimination task of the sample to be detected. The NAR model uses a context modeling mechanism to closely combine the semantic information of the prompt sample with the features of the test sample, thereby improving the ability to capture forged audio features, and is particularly suitable for identifying audio generated by complex forgery techniques. Since LLaMA itself is an autoregressive (AR) generation model, it is not suitable for audio deep fake detection tasks, because this task does not involve a generation process, but requires discrimination and classification of audio signals. Therefore, by changing the LLaMA model structure to a NAR model, the computational and time complexity problems brought about by the autoregressive generation process are avoided, while retaining the model's powerful ability in context modeling. The NAR architecture enables the model to process audio sequences efficiently and in parallel, improves the ability to model long-term dependencies, and is more suitable for forged audio detection tasks.

[0058] In step S140, based on the prediction result, it is determined whether the sample data to be detected is forged audio.

[0059] In the embodiment of the present application, the contextual prompt information and the audio features to be detected are input into the preset prediction model, and the contextual prompt information is input as a condition to provide prior information for the judgment of the audio data to be detected. For example, <prompt audio 1, label 1, prompt audio 2, label 2, audio to be detected>, at this time, the preset prediction model can automatically generate a label corresponding to the audio to be detected, and the label can be output as a prediction result, such as true or false, so as to determine whether the audio data to be detected is true audio or false audio.

[0060] In an embodiment of the present application, a method for detecting deep fake audio is provided, including: obtaining prompt audio data and audio data to be detected; performing feature extraction on the prompt audio data and the audio data to be detected respectively to obtain prompt audio features and audio features to be detected; using the prompt audio features and corresponding labels as context prompt information, and predicting the audio features to be detected through a preset prediction model; based on the prediction results, determining whether the sample data to be detected is fake audio. In an embodiment of the present application, the generalization ability of the model for unknown fake methods is improved through context learning capabilities, and through context learning, when the model discriminates test samples, certain prompt samples are given to help the model better understand fake features and improve detection accuracy.

[0061] In an implementation of the present application, the extracting of the prompt audio features and the audio features to be detected based on the prompt sample data and the sample data to be detected respectively includes:

[0062] Convert the prompt audio data and the audio data to be detected into Mel-spectrograms respectively;

[0063] Based on the Mel-spectrogram, high-level audio features are generated to serve as the prompt audio features and the audio features to be detected.

[0064] Optionally, the feature extraction model can be a pre-trained Wav2Vec2-BERT model, which consists of three main modules. It includes a Mel spectrum extractor and a Transformer speech recognition model (Conformer Speech Encoder) connected in sequence, and the Transformer speech recognition model based on convolution enhancement includes a convolution module and a Transformer module. The prompt audio data and the audio data to be detected are respectively input into the Mel spectrum extractor, and the original audio waveform signal is converted into a Mel spectrum graph by the Mel spectrum extractor to extract the time features and frequency features of the audio. Then, the Mel spectrum graph is feature encoded by the Transformer speech recognition model based on convolution enhancement to generate high-level audio features as prompt audio features and audio features to be detected. The prompt audio features and the audio features to be detected can be high-quality input representations for subsequent audio detection tasks.

[0065] In one embodiment of the present application, converting the prompt sample data and the sample data to be detected into a Mel-spectrogram includes:

[0066] Performing Fourier transform on the prompt sample data and the sample data to be detected respectively to obtain a frequency spectrum;

[0067] The spectrum graph is filtered by a Mel filter to obtain the Mel spectrum graph.

[0068] Optionally, the prompt audio data and the audio data to be detected may be audio waveform signals. First, the input audio waveform signal may be sampled and normalized, and then low-frequency noise may be removed by a high-pass filter, and the obtained audio signal may be framed to obtain an audio frame of a preset time range, for example, each frame of audio may be 20 milliseconds. To ensure that the signal is approximately stable in a short time. Each frame of audio is windowed, such as a Hamming window, to reduce edge effects, and the windowed audio frame is Fourier transformed (FFT) to convert the time domain signal into a frequency domain signal, thereby obtaining a spectrogram. Then, based on the spectrogram, the square of the spectrum may be calculated to extract the energy distribution of each frequency band. The spectrum is then weighted and summed by a set of Mel filter banks to generate an energy distribution on the Mel frequency scale, and finally the logarithm of these energy values ​​is taken to obtain a logarithmic Mel spectrum, i.e., a Mel spectrogram. The Mel spectrogram represents the order of frames on the time axis and the intensity of the Mel frequency band on the frequency axis.

[0069] In one embodiment of the present application, the Fourier transform is performed on the prompt sample data and the sample data to be detected to obtain a spectrum diagram, including:

[0070] Respectively sampling and normalizing the prompt audio data and the audio data to be detected;

[0071] Respectively filtering the prompt audio data and the audio data to be detected after sampling and normalization processing;

[0072] Respectively performing frame processing on the filtered prompt audio data and the audio data to be detected;

[0073] Each frame of prompt audio data and each frame of audio data to be detected are windowed respectively, and Fourier transform is performed on the windowed each frame of prompt audio data and each frame of audio data to be detected to obtain the spectrum diagram.

[0074] Optionally, the prompt audio data and the audio data to be detected may be audio waveform signals. First, the input audio waveform signal may be sampled and normalized, and then low-frequency noise may be removed by a high-pass filter. The obtained audio signal may be framed to obtain audio frames within a preset time range, for example, each frame of audio may be 20 milliseconds. This ensures that the signal is approximately stable in a short time. Each frame of audio is windowed, such as a Hamming window, to reduce edge effects, and the windowed audio frame is Fourier transformed (FFT) to convert the time domain signal into a frequency domain signal, thereby obtaining a spectrogram.

[0075] In an embodiment of the present application, generating high-level audio features based on the Mel-spectrogram includes:

[0076] The Mel-spectrogram is respectively input into a convolution module and a Transformer module for feature encoding, local features are extracted by the convolution module, and global features are extracted by the Transformer module;

[0077] The local features and the global features are fused, normalized and residually connected to generate the high-level audio features.

[0078] Optionally, the feature extraction model can be a pre-trained Wav2Vec2-BERT model, which consists of three main modules. It includes a Mel spectrum extractor and a Transformer speech recognition model (Conformer Speech Encoder) based on convolution enhancement connected in sequence, and the Transformer speech recognition model based on convolution enhancement includes a convolution module and a Transformer module. The Mel spectrum map corresponding to the generated prompt audio data and the Mel spectrum map corresponding to the audio data to be detected are respectively input into the convolution module and the Transformer module for feature encoding. The convolution module extracts the local features of the audio signal through one-dimensional convolution to capture the spectrum changes and transient information in a short time range. The Transformer module uses a multi-head self-attention mechanism to model global time dependencies, captures long-term correlations by assigning weights to each time frame, and embeds time information in combination with position encoding to ensure temporal expression. By alternately stacking convolution layers and Transformer modules, local features and global features can be fused, and high-quality high-level audio feature representations can be generated using normalization and residual connections of the fully connected layer. These features can comprehensively characterize the temporal and spectral characteristics of audio, providing a reliable input representation for the forged audio detection task.

[0079] In an embodiment of the present application, the step of using the prompt audio feature and the corresponding label as context prompt information to predict the audio feature to be detected includes:

[0080] The prompt audio feature and the corresponding label are used as context prompt information, and are input into a non-autoregressive model together with the audio feature to be detected;

[0081] The non-autoregressive model performs conditional modeling on the audio data to be detected through the contextual prompt information to capture the forged audio in the audio data to be tested.

[0082] Optionally, the non-autoregressive model can be a non-autoregressive (NAR) model based on the LLaMA architecture. The prompt audio features and the audio features to be detected extracted by Wav2Vec2-BERT are input into the non-autoregressive (NAR) model based on the LLaMA architecture for modeling. The prompt audio features and their corresponding true labels together constitute the contextual prompt information, providing prior knowledge for the discrimination task of the audio to be detected. The NAR model uses the context modeling mechanism to closely combine the semantic information of the prompt sample with the characteristics of the test sample, thereby improving the ability to capture forged audio features, and is particularly suitable for identifying audio generated by complex forgery techniques.

[0083] The specific prediction process of the non-autoregressive (NAR) model is as follows: the audio features of the prompt audio data are combined with their corresponding labels and the audio features to be detected are input into the NAR model. The NAR model models the correlation between the prompt sample audio data and the audio data to be detected through the attention mechanism, and calculates the semantic similarity and feature distribution relationship between the two. On this basis, the features of the prompt sample are embedded in the context vector to form a global semantic and feature distribution pattern. Subsequently, the features of the audio data to be detected are introduced, and the NAR model uses the context prompt information to conditionally model the audio features of the audio data to be detected, focusing on potential forged features. In the modeling process, NAR captures the possible forged feature deviations between the test sample and the prompt sample through global semantic comparison and context semantic guidance, thereby completing the judgment of the forged characteristics of the test sample. For example, <prompt audio 1, label 1, prompt audio 2, label 2, audio to be detected>, at this time, the preset prediction model can automatically generate the label corresponding to the audio to be detected, and the label can be output as a prediction result, such as real or forged, so as to determine whether the audio data to be detected is real audio or forged audio.

[0084] Since the prompt audio data provides rich semantics and distribution patterns of forged audio features, the NAR model can effectively utilize this contextual information to guide the model to capture potential forged features in the audio data to be detected. This context-assisted approach is particularly suitable for task scenarios with hidden forged features. Unlike the way the autoregressive model gradually models the sequence, the NAR architecture focuses more on overall modeling capabilities. By parallelizing the features of the prompt sample audio data and the audio data to be detected, the NAR model can perform comprehensive modeling in the global semantic and time dimensions, which makes it more suitable for modeling long audio sequences and complex forged features in task scenarios. Especially for adversarial audio with different forgery techniques, the NAR architecture can use the constraints provided by the contextual prompt samples to enhance the generalization ability of the model.

[0085] In an embodiment of the present application, before predicting the audio feature to be detected by using a preset prediction model, the method further includes:

[0086] Constructing a training audio sample data set, wherein the training audio sample data set includes prompt samples and training samples;

[0087] Selecting a plurality of the prompt samples from the training audio sample data set for feature extraction to obtain a plurality of prompt features, and selecting an initial training sample from the training audio sample data set for feature extraction to obtain a training feature;

[0088] Using each of the prompt features and the corresponding label as context prompt information, predicting the training features through an initial prediction model;

[0089] Based on the preset loss function, calculate the loss value of this prediction result;

[0090] If the loss value is greater than a preset threshold, the next training sample is selected from the training audio sample data set for feature extraction and prediction until the preset convergence condition is met, thereby obtaining the preset prediction model.

[0091] Optionally, audio signals of multiple languages, multiple styles (such as audiobooks, conversations, etc.) and multiple forgery methods can be collected, and training sample datasets can be constructed to ensure the diversity of real audio samples and forged samples to improve the generalization ability of the model. Existing public datasets such as ASVspoof2019, MLAAD, Wavefake, LibriSeVoc, and GigaSpeech can also be used. Among them, ASVspoof2019, MLAAD, Wavefake, and LibriSeVoc are authoritative datasets specifically for audio forgery detection, while GigaSpeech provides a wealth of real audio samples, significantly improving the diversity of real samples and laying a solid foundation for the training of forgery detection models. Diverse data is crucial to improving the robustness and generalization ability of the model. One of the challenges of audio forgery detection is the diversity and rapid changes of forgery methods. By introducing datasets of multiple languages, multiple styles, and multiple forgery methods, it is ensured that the model can maintain high performance in a variety of different forgery scenarios. In addition, the diversity of real audio samples (such as from GigaSpeech) can further enhance the model's ability to distinguish between real and forged audio.

[0092] Then, the model can be built based on the LLaMA model based on the NAR architecture, focusing on solving the need for parallel processing of long sequence features in the audio forgery detection task. The feature extraction part of the model uses the pre-trained Wav2Vec2-BERT model, which can efficiently extract the spectrum, timing and context features in the audio signal, providing high-quality input for the sequence modeling of the NAR model. Through this joint architecture, the model can accurately capture the forgery features in the audio and maintain high computational efficiency.

[0093] After the model is established, the initial training sample can be randomly selected from the training sample data set, and multiple prompt samples, such as 6 prompt samples, can be randomly selected and respectively input into the pre-trained Wav2Vec2-BERT model for feature extraction to obtain multiple prompt audio features and training audio features, and then the multiple prompt audio features and training audio features are input into the NAR model, and the prompt audio features and their corresponding labels are used as contextual prompt information to predict the initial training sample to determine whether it is real audio or forged audio. Further, the loss value can be calculated according to the label of the training sample and the prediction result through a preset loss function, such as a cross entropy loss function, combined with the stochastic gradient descent method (SGD) for optimization. That is, if the loss value is greater than the preset threshold, the next training sample can be selected from the training sample data set, and the prediction result can be generated again through the above steps, and the above steps are repeated until the preset convergence condition is met, such as the number of iterations reaches the preset number, or the loss value is less than the preset threshold, or all training samples in the training sample set are trained, then the preset prediction model can be obtained.

[0094] It should be noted that a test data set can be established. After the model training is completed, the model can be tested based on the test data set. The model testing process is consistent with the training process. The test sample is first feature extracted through the Wav2Vec2-BERT model. This ensures that the input features of the test sample are consistent with the training sample. Then, each test sample can be combined with several prompt samples and input into the NAR model for prediction. The features and labels of the prompt samples are used as context information to assist the model in determining whether the test sample is forged audio. The model testing phase uses multiple indicators to evaluate its performance, including accuracy (Acc%), false alarm rate (FAR%), missed alarm rate (FRR%), equal error rate (EER%) and F1 score. The threshold is determined by the experimental results of the validation set. The experimental results show that the model shows significant robustness and high accuracy under a variety of forgery techniques, languages ​​and styles, and can effectively deal with different types of audio forgery techniques.

[0095] In an embodiment of the present application, the Wav2Vec2-BERT module can be used to extract low-level features (such as spectrum, timing patterns) from the original audio and generate high-quality contextual embedding representations. The pre-training process of this feature extractor enables the model to have a deep understanding of multi-language and multi-style audio, thereby providing a reliable feature basis for the forgery detection task. The contextual learning ability of the LLaMA model is the core of the generalization ability in the forgery detection task. With the assistance of prompt samples and labels, the model can use contextual information to capture the characteristics of forged audio, thereby enhancing the adaptability to unknown forgery techniques. The contextual learning module not only improves the accuracy of forgery detection, but also significantly enhances the robustness of the model in different scenarios and language environments.

[0096] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0097] In one embodiment, an audio deep fake detection device is provided, and the audio deep fake detection device corresponds to the audio deep fake detection method in the above embodiment. Figure 4 As shown, the audio deep forgery detection device includes an audio data acquisition unit 10, an audio feature extraction unit 20, a prediction unit 30 and a forged audio judgment unit 40. The functional modules are described in detail as follows:

[0098] An audio data acquisition unit 10, used to acquire prompt audio data and audio data to be detected;

[0099] The audio feature extraction unit 20 is used to extract features from the prompt audio data and the audio data to be detected, respectively, to obtain prompt audio features and audio features to be detected;

[0100] A prediction unit 30, configured to use the prompt audio feature and the corresponding label as context prompt information to predict the audio feature to be detected;

[0101] The forged audio determination unit 40 is used to determine whether the sample data to be detected is forged audio based on the prediction result.

[0102] In one embodiment of the present application, the audio feature extraction unit 20 is used to:

[0103] Convert the prompt audio data and the audio data to be detected into Mel-spectrograms respectively;

[0104] Based on the Mel-spectrogram, high-level audio features are generated to serve as the prompt audio features and the audio features to be detected.

[0105] In one embodiment of the present application, the audio feature extraction unit 20 is used to:

[0106] Performing Fourier transform on the prompt sample data and the sample data to be detected respectively to obtain a frequency spectrum;

[0107] The spectrum graph is filtered by a Mel filter to obtain the Mel spectrum graph.

[0108] In one embodiment of the present application, the audio feature extraction unit 20 is used to:

[0109] Respectively sampling and normalizing the prompt audio data and the audio data to be detected;

[0110] Respectively filtering the prompt audio data and the audio data to be detected after sampling and normalization processing;

[0111] Respectively performing frame processing on the filtered prompt audio data and the audio data to be detected;

[0112] Each frame of prompt audio data and each frame of audio data to be detected are windowed respectively, and Fourier transform is performed on the windowed each frame of prompt audio data and each frame of audio data to be detected to obtain the spectrum diagram.

[0113] In one embodiment of the present application, the audio feature extraction unit 20 is used to:

[0114] The Mel-spectrogram is respectively input into a convolution module and a Transformer module for feature encoding, local features are extracted by the convolution module, and global features are extracted by the Transformer module;

[0115] The local features and the global features are fused, normalized and residually connected to generate the high-level audio features.

[0116] In an embodiment of the present application, the prediction unit 30 is further configured to:

[0117] The prompt audio feature and the corresponding label are used as context prompt information, and are input into a non-autoregressive model together with the audio feature to be detected;

[0118] The non-autoregressive model performs conditional modeling on the audio data to be detected through the contextual prompt information to capture the forged audio in the audio data to be tested.

[0119] In one embodiment of the present application, the device further includes: a prediction model training unit, which is used to:

[0120] Constructing a training audio sample data set, wherein the training audio sample data set includes prompt samples and training samples;

[0121] Selecting a plurality of the prompt samples from the training audio sample data set for feature extraction to obtain a plurality of prompt features, and selecting an initial training sample from the training audio sample data set for feature extraction to obtain a training feature;

[0122] Using each of the prompt features and the corresponding label as context prompt information, predicting the training features through an initial prediction model;

[0123] Based on the preset loss function, calculate the loss value of this prediction result;

[0124] If the loss value is greater than a preset threshold, the next training sample is selected from the training audio sample data set for feature extraction and prediction until the preset convergence condition is met, thereby obtaining the preset prediction model.

[0125] In an embodiment of the present application, the Wav2Vec2-BERT module can be used to extract low-level features (such as spectrum, timing patterns) from the original audio and generate high-quality contextual embedding representations. The pre-training process of this feature extractor enables the model to have a deep understanding of multi-language and multi-style audio, thereby providing a reliable feature basis for the forgery detection task. The contextual learning ability of the LLaMA model is the core of the generalization ability in the forgery detection task. With the assistance of prompt samples and labels, the model can use contextual information to capture the characteristics of forged audio, thereby enhancing the adaptability to unknown forgery techniques. The contextual learning module not only improves the accuracy of forgery detection, but also significantly enhances the robustness of the model in different scenarios and language environments.

[0126] For the specific definition of the audio deep fake detection device, please refer to the definition of the audio deep fake detection method above, which will not be repeated here. Each module in the above-mentioned audio deep fake detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0127] In one embodiment, a computer device is provided. The computer device may be a terminal device, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium. The readable storage medium stores computer-readable instructions. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer-readable instructions are executed by the processor, an audio deep fake detection method is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0128] In an embodiment of the present application, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, and when the processor executes the computer-readable instructions, the steps of the audio deep fake detection method as described above are implemented.

[0129] In an embodiment of the application, a readable storage medium is provided, which stores computer-readable instructions. When the computer-readable instructions are executed by a processor, the steps of the audio deep fake detection method as described above are implemented.

[0130] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0131] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0132] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for detecting deep fake audio, characterized in that: The method comprises: Obtain prompt audio data and audio data to be detected; Extracting features from the prompt audio data and the audio data to be detected respectively to obtain prompt audio features and audio features to be detected; The prompt audio feature and the corresponding label are used as context prompt information, and the audio feature to be detected is predicted by a preset prediction model; Based on the prediction result, determine whether the sample data to be detected is forged audio.

2. The audio deep fake detection method according to claim 1, characterized in that: The extracting features of the prompt audio data and the audio data to be detected respectively to obtain prompt audio features and audio features to be detected includes: Convert the prompt audio data and the audio data to be detected into Mel-spectrograms respectively; Based on the Mel-spectrogram, high-level audio features are generated to serve as the prompt audio features and the audio features to be detected.

3. The audio deep fake detection method according to claim 2, characterized in that: The converting the prompt sample data and the sample data to be detected into a Mel frequency spectrum graph comprises: Performing Fourier transform on the prompt sample data and the sample data to be detected respectively to obtain a frequency spectrum; The spectrum graph is filtered by a Mel filter to obtain the Mel spectrum graph.

4. The audio deep fake detection method according to claim 3, characterized in that: The Fourier transform is performed on the prompt sample data and the sample data to be detected respectively to obtain a spectrum diagram, including: Respectively sampling and normalizing the prompt audio data and the audio data to be detected; Respectively filtering the prompt audio data and the audio data to be detected after sampling and normalization processing; Respectively performing frame processing on the filtered prompt audio data and the audio data to be detected; Each frame of prompt audio data and each frame of audio data to be detected are windowed respectively, and Fourier transform is performed on the windowed each frame of prompt audio data and each frame of audio data to be detected to obtain the spectrum diagram.

5. The audio deep fake detection method according to claim 3, characterized in that: The step of generating high-level audio features based on the Mel-spectrogram includes: The Mel-spectrogram is respectively input into a convolution module and a Transformer module for feature encoding, local features are extracted by the convolution module, and global features are extracted by the Transformer module; The local features and the global features are fused, normalized and residually connected to generate the high-level audio features.

6. The audio deep fake detection method according to claim 1, characterized in that: The step of using the prompt audio feature and the corresponding label as context prompt information to predict the audio feature to be detected includes: The prompt audio features and corresponding labels are used as context prompt information, and are input into a non-autoregressive model together with the audio features to be detected, so as to conditionally model the audio data to be detected through the non-autoregressive model to capture the forged audio in the audio data to be tested.

7. The audio deep fake detection method according to claim 1, characterized in that: Before predicting the audio feature to be detected by using a preset prediction model, the method further includes: Constructing a training audio sample data set, wherein the training audio sample data set includes prompt samples and training samples; Selecting a plurality of the prompt samples from the training audio sample data set for feature extraction to obtain a plurality of prompt features, and selecting an initial training sample from the training audio sample data set for feature extraction to obtain a training feature; Using each of the prompt features and the corresponding label as context prompt information, predicting the training features through an initial prediction model; Based on the preset loss function, calculate the loss value of this prediction result; If the loss value is greater than a preset threshold, the next training sample is selected from the training audio sample data set for feature extraction and prediction until the preset convergence condition is met, thereby obtaining the preset prediction model.

8. An audio deep fake detection device, characterized in that: The device comprises: An audio data acquisition unit, used to acquire prompt audio data and audio data to be detected; An audio feature extraction unit, used to extract features from the prompt audio data and the audio data to be detected, respectively, to obtain prompt audio features and audio features to be detected; A prediction unit, configured to use the prompt audio feature and the corresponding label as context prompt information to predict the audio feature to be detected; The forged audio judgment unit is used to determine whether the sample data to be detected is forged audio based on the prediction result.

9. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executed on the processor, characterized in that: When the processor executes the computer-readable instructions, the audio deep fake detection method as described in any one of claims 1 to 7 is implemented.

10. A readable storage medium having computer readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the audio deep fake detection method as described in any one of claims 1 to 7 is implemented.

Images