Deep fake audio detection and protection method based on large-scale pre-trained model Whisper

Abstract

The present invention discloses a deep fake audio detection and protection method based on a large-scale pre-trained model Whisper, comprising the steps of step S1: inputting the audio to be detected into the pre-trained model Whisper, and using the transcribed text of the entire audio as a prompt during the fine-tuning process of each audio segment, so that the model Whisper can utilize global text information; step S2: pre-processing the audio data to adapt to the model Whisper during training and decoding; step S3: evaluating the performance of the model Whisper during training by designing a detection cross-entropy loss function. The deep fake audio detection and protection method based on the large-scale pre-trained model Whisper disclosed by the present invention realizes audio authenticity identification through transfer learning. Different from existing speech classification or recognition tasks, the complete transcribed text of the audio is embedded in the decoder as prompt information through an innovative fine-tuning strategy, which overcomes the problem of single reliance on traditional acoustic features.

Description

Technical Field

[0001] The present invention belongs to the technical field of forged audio detection and protection, and specifically relates to a deep forged audio detection and protection method based on a large-scale pre-trained model Whisper. Background Art

[0002] With the rapid development of deep learning technology, its application in machine hearing is becoming increasingly widespread. Numerous efficient and accurate models have emerged, surpassing traditional techniques and significantly improving the efficiency of media content production and creation. This is particularly true in the field of audio synthesis, such as text-to-speech (TTS) and voice conversion (VC). These technologies can generate highly realistic target speaker voices, even capable of deceiving the human ear and human-computer interaction devices.

[0003] However, the misuse of audio deepfake technology poses significant risks. Deepfake audio can potentially harm both the human and machine auditory systems, posing a serious threat to livelihoods, the economy, politics, and social security worldwide. Therefore, there is an urgent need to develop automated and highly accurate deepfake audio detection methods to effectively address this threat. Summary of the Invention

[0004] The main purpose of this invention is to provide a deep fake audio detection and protection method based on the large-scale pre-trained Whisper model. The large-scale pre-trained Whisper model based on the Transformer architecture realizes audio authenticity identification through transfer learning. Unlike existing speech classification or recognition tasks (such as those based on Hubert, Wav2vec, etc.), this invention not only utilizes the powerful multilingual speech recognition capabilities of the Whisper model, but also uses an innovative fine-tuning strategy to embed the complete transcript of the audio as a prompt into the decoder, overcoming the problem of traditional single reliance on acoustic features.

[0005] To achieve the above objectives, the present invention provides a deep fake audio detection and protection method based on the large-scale pre-trained Whisper model, comprising the following steps:

[0006] Step S1: Input the audio to be tested into the pre-trained Whisper model (hereinafter referred to as Model Whisper), and use the transcript of the entire audio as a prompt during the fine-tuning process of each audio segment, so that Model Whisper can utilize global text information;

[0007] Step S2: Preprocess the audio data to adapt to the Whisper model during training and decoding (Whisper requires that audio be padded or trimmed to segments no longer than a preset time (30 seconds). For long-form transcription tasks, Whisper provides a continuous transcription interface that leverages contextual information from preceding segments. However, during fine-tuning for transfer learning, audio segments must still meet the 30-second limit).

[0008] Step S3: In terms of loss function design, based on the default cross-entropy loss function, a detection cross-entropy loss function is designed to evaluate the performance of the model Whisper during the training process (with special attention to the accuracy of predicting the content between specific tags identified as <|transcribe|> and <|endoftext|>).

[0009] As a further preferred technical solution of the above technical solution, in step S1:

[0010] Define the audio as x and the full text transcribed by the Whisper model as transcript x ,Under this framework, the prediction process of model Whisper for each segment is expressed as:

[0011] P(y|x,transcript x )→result;

[0012] Where y represents the input token sequence of the decoder in the model, and result is the classification result of the model Whisper. During the training process, the prompt of the transcribed text should be used as a reference rather than the target of prediction. Therefore, in the input and target of the decoder, from | <startofprev>The position of the prompt at the beginning of the | mark remains unchanged, and the model Whisper directly outputs the same content according to the input. When encountering | <startoftranscript>|Mark, start predicting subsequent content;

[0013] (This invention plans to use the official Whisper-large model to transcribe the audio to obtain its corresponding text information.) During the testing phase, not only the prefix but also the full text needs to be provided as a prompt to ensure that the Whisper model refers to the complete text information when generating the output.

[0014] As a further preferred technical solution of the above technical solution, in step S2:

[0015] For long audio that exceeds the preset time, the Whisper model is used to split it into smaller decoding segments and record the end time of each segment (such as T0, T1, ..., T k ), then, a greedy merging strategy is used to sequentially merge the decoded segments into merged segments, making their lengths (as close as possible) but not exceeding the preset time, and the remaining decoded segments that cannot be merged are discarded;

[0016] For short audio that does not exceed the preset time, it is extended to the specified length through repetition and splicing to meet the input requirements of the Whisper model; finally, the preprocessed audio (segment) data is used for training and testing of the Whisper model.

[0017] As a further preferred technical solution of the above technical solution, in step S3, the loss calculation is performed in steps to ensure that the Whisper model is penalized for prediction errors in key segments of the sequence, as shown below:

[0018] The total loss L is the global loss L global and local intermediate loss L inter The weighted combination of the global loss L global Covering the entire sequence, it is represented as;

[0019] L global =∑ i,j L CE (y i,j ,y′ i,j );

[0020] Among them, L CE represents the cross entropy loss function, y is the predicted output of the model, y′ is the real label, and y i,j and y′ i,j Represent the predicted value and true value at position (i, j) respectively; then, assume that the position of the i-th <|transcribe|> tag is T i , and the position of the i-th <|endoftext|> tag is E i , local intermediate loss L inter Focusing on the subsequence between <|transcribe|> and <|endoftext|>, which is the prediction result of the audio authenticity, it is expressed as:

[0021]

[0022] The final loss value is calculated as L = L global +α*L inter , where α is a weight factor used to increase the influence of local intermediate losses (which means that during training, the model will not only try to minimize the error in the entire sequence (by minimizing L global ), with special attention paid to reducing the error in the subsequence between the <|transcribe|> and <|endoftext|> tags (by minimizing L inter and give it higher weight)).

[0023] As a further preferred technical solution of the above technical solution, during the training phase, the audio sample to be tested will first reach a random decision point, where it will randomly determine whether to perform speech enhancement processing on it, thereby improving the diversity of the audio and allowing the Whisper model to be trained in an environment closer to reality (thus enhancing the robustness of the model and laying the foundation for subsequent accurate detection);

[0024] Subsequently, the processed (or unprocessed) audio data is input into the Whisper model. At this time, the encoder part of the Whisper model has been frozen to ensure that its pre-trained parameters remain stable during the training process, ensuring that the Whisper model can robustly extract the features of the audio data. Then, through the designed loss function, the loss value is calculated based on the predicted results and the true label, which is used to update the parameters of the Whisper model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the structure of Whisper model of the present invention.

[0026] Figure 2 It is a schematic diagram of the transfer learning training process of the Whisper model applied in the present invention. DETAILED DESCRIPTION

[0027] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are for illustrative purposes only, and those skilled in the art will readily appreciate other obvious variations. The basic principles of the present invention defined in the following description may be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the present invention.

[0028] In a preferred embodiment of the present invention, those skilled in the art should note that the large-scale pre-training model Whisper and the like involved in the present invention can be regarded as prior art.

[0029] The codec-based speech generation framework consists of three core modules: (1) Audio Codec: encodes speech features into discrete tokens. (2) Large Language Model (LLM): maps input text tokens to audio tokens. (3) Vocoder: reconstructs audio waveforms from audio tokens. During training, the LLM achieves speech generation by learning the relationship between text embeddings and audio tokens, while the audio codec takes on the key task of preserving key information in the audio through compression and quantization. A general audio codec usually consists of three parts: encoder, codec, and decoder, which are responsible for audio feature extraction, compression, and recovery, respectively. During the inference phase, the LLM maps text input to audio tokens, and the vocoder converts them into high-quality speech. This LLM-based speech synthesis architecture greatly improves the authenticity and naturalness of the synthesized speech.

[0030] However, it remains uncertain whether existing anti-spoofing techniques can effectively counter deepfake audio generated by codec-based systems. Recent research has shown that anti-spoofing models trained on vocoders often fail to detect deepfake audio generated by these advanced codec systems. The unique properties of codec-based deepfake audio exacerbate these challenges, creating three key obstacles for current Audio Deepfake Detection (ADD) systems:

[0031] (1) The surge in the number of forged audios: The rapid development of codec systems has significantly lowered the technical threshold for voice forgery, making it easier to generate forged audio on a large scale.

[0032] (2) Stronger deception capabilities: Codec-based deep fake audio exhibits higher authenticity and subtlety, making it more difficult to distinguish from real audio.

[0033] (3) Enhanced audio adaptability: Emerging systems have shown strong adaptability in different languages, environments, and audio types, further increasing the difficulty of detection.

[0034] Furthermore, there are currently no systematic studies on the identification of deepfake audio generated by decoder systems, which highlights the limitations of existing anti-forgery technologies. Therefore, it is urgent and important to develop new anti-spoofing systems with higher generalization and cross-type detection capabilities.

[0035] To address the above issues, the present invention proposes an innovative detection solution, the core of which includes:

[0036] (1) A transfer learning strategy based on the large-scale pre-trained model Whisper enables the model to efficiently adapt to different types of deep fake speech;

[0037] (2) Introducing additional pre-trained audio features and combining them with a multi-audio feature fusion mechanism to capture subtle differences in forged audio.

[0038] The Whisper model has been trained on 680,000 hours of supervised audio data and is capable of performing tasks such as multilingual speech recognition, speech translation, and voice activity monitoring. It has achieved outstanding results on multiple public datasets. Furthermore, research has found that the audio features provided by the Whisper model are highly robust and well-suited for tasks such as audio classification. Furthermore, many researchers have used transfer learning methods to effectively apply the Whisper model to tasks such as speech understanding. This opens up the possibility of leveraging Whisper's strengths in a variety of speech processing applications.

[0039] Transfer learning is a technique that can transfer knowledge learned in one domain to another, and is mostly used to adapt pre-trained models to different downstream tasks. Due to the limited datasets for the task of distinguishing the authenticity of audio, it is very valuable to use models trained on large-scale datasets for other tasks and fine-tune and transfer them. For example, many previous studies have explored the use of unsupervised pre-trained models such as Wav2vec and Hubert to obtain audio features, and then input these audio features into downstream models for specific tasks. However, these methods require adding additional classifiers after the pre-trained model to address the problem of inconsistent shapes of audio features in different domains. In contrast, when using large-scale models with supervised learning methods such as Whisper for transfer learning, the labels can be directly converted to text and used as targets in the fine-tuning phase, effectively eliminating the need for additional layers and classifiers.

[0040] Preferred embodiment.

[0041] The present invention discloses a deep fake audio detection and protection method based on a large-scale pre-trained model Whisper, comprising the following steps:

[0042] Step S1: (as Figure 1 As shown, the research scheme of the present invention chooses to freeze the encoder layer of Whisper while fine-tuning the decoder layer of Whisper. The target of each segment is a text sequence containing the prefix "CodecFake Task:" followed by the labels "spoofing" and "genuine", where each label corresponds to a different single tag. During the testing phase, the present invention still uses "CodecFake Task:" as the decoding prefix to predict subsequent labels. In order to effectively utilize the global voice information of the entire audio, it is necessary to extract the transcribed text of the entire audio as a prompt and add the prompt information to the training and testing process to prove the effectiveness of the transcription prompt method. Although it is feasible for the Whisper model to use only audio as input data, it essentially limits the audio to a narrow window of up to 30 seconds. To address this limitation, this study proposes an innovative method) to input the audio to be tested into the pre-trained model Whisper (hereinafter referred to as model Whisper), and use the transcribed text of the entire audio as a prompt during the fine-tuning process of each audio segment, so that the model Whisper can utilize global text information;

[0043] Step S2: Preprocess the audio data to adapt to the Whisper model during training and decoding (Whisper requires that audio be padded or trimmed to segments no longer than a preset time (30 seconds). For long-form transcription tasks, Whisper provides a continuous transcription interface that leverages contextual information from preceding segments. However, during fine-tuning for transfer learning, audio segments must still meet the 30-second limit).

[0044] Step S3: In terms of loss function design, based on the default cross-entropy loss function, a detection cross-entropy loss function is designed to evaluate the performance of the model Whisper during the training process (with special attention to the accuracy of predicting the content between specific tags identified as <|transcribe|> and <|endoftext|>).

[0045] Specifically, in step S1:

[0046] Define the audio as x and the full text transcribed by the Whisper model as transcript x ,Under this framework, the prediction process of model Whisper for each segment is expressed as:

[0047] P(y|x,transcript x )→result;

[0048] Where y represents the input token sequence of the decoder in the model, and result is the classification result of the model Whisper. During the training process, the prompt of the transcribed text should be used as a reference rather than the target of prediction. Therefore, in the input and target of the decoder, from | <startofprev>The position of the prompt at the beginning of the | mark remains unchanged, and the model Whisper directly outputs the same content according to the input. When encountering | <startoftranscript>|Mark, start predicting subsequent content;

[0049] (This invention plans to use the official Whisper-large model to transcribe the audio to obtain its corresponding text information.) During the testing phase, not only the prefix but also the full text needs to be provided as a prompt to ensure that the Whisper model refers to the complete text information when generating the output.

[0050] More specifically, in step S2:

[0051] For long audio that exceeds the preset time, the Whisper model is used to split it into smaller decoding segments and record the end time of each segment (such as T0, T1, ..., T k ), then, a greedy merging strategy is used to sequentially merge the decoded segments into merged segments, making their lengths (as close as possible) but not exceeding the preset time, and the remaining decoded segments that cannot be merged are discarded;

[0052] For short audio that does not exceed the preset time, it is extended to the specified length through repetition and splicing to meet the input requirements of the Whisper model; finally, the preprocessed audio (segment) data is used for training and testing of the Whisper model.

[0053] Furthermore, in step S3, the loss calculation is performed in steps to ensure that the model Whisper is penalized for prediction errors within the key segments of the sequence, as shown below:

[0054] The total loss L is the global loss L global and local intermediate loss L inter The weighted combination of the global loss L global Covering the entire sequence, it is represented as;

[0055] L global =∑ i,j L CE (y i,j ,y′ i,j );

[0056] Among them, L CE represents the cross entropy loss function, y is the predicted output of the model, y′ is the real label, and y i,j and y′ i,j Represent the predicted value and true value at position (i, j) respectively; then, assume that the position of the i-th <|transcribe|> tag is T i , and the position of the i-th <|endoftext|> tag is E i , local intermediate loss L inter Focusing on the subsequence between <|transcribe|> and <|endoftext|>, which is the prediction result of the audio authenticity, it is expressed as:

[0057]

[0058] The final loss value is calculated as L = L global +α*L inter , where α is a weight factor used to increase the influence of local intermediate losses (which means that during training, the model will not only try to minimize the error in the entire sequence (by minimizing L global ), with special attention paid to reducing the error in the subsequence between the <|transcribe|> and <|endoftext|> tags (by minimizing L inter and give it higher weight)).

[0059] Furthermore, the complete training process of the model is as follows Figure 2 During the training phase, the audio sample to be tested first reaches a random decision point, where it is randomly determined whether to perform speech enhancement processing on it. This improves the diversity of the audio and enables the Whisper model to be trained in a more realistic environment (thus enhancing the model's robustness and laying the foundation for subsequent accurate detection).

[0060] Subsequently, the processed (or unprocessed) audio data is input into the Whisper model. At this time, the encoder part of the Whisper model has been frozen to ensure that its pre-trained parameters remain stable during the training process, ensuring that the Whisper model can robustly extract the features of the audio data. Then, through the designed loss function, the loss value is calculated based on the predicted results and the true label, which is used to update the parameters of the Whisper model.

[0061] It is important to note that in this process, if the audio transcript is used as a prompt for the model to help it better understand the audio content and make accurate judgments, the model will be named "Whisper-Trans-Prompt", which focuses on extracting key information directly from the audio signal itself for training and reasoning. On the contrary, if there is no need for the prompt of the audio transcript, the model is called "Whisper-Trans", which focuses on extracting key information directly from the audio signal itself for training and reasoning. These two different settings laid the foundation for subsequent comparative experiments to verify the importance of introducing audio transcripts as prompts. Regardless of which method is used, the model can ultimately output prediction results efficiently, clearly identifying the audio sample as "Spoofing" (fake) or "Genuine" (real).

[0062] For the present invention:

[0063] The large-scale pre-trained model Whisper based on the Transformer architecture is used to achieve audio authenticity identification through transfer learning. Unlike existing speech classification or recognition tasks (such as tasks based on Hubert, Wav2vec, etc.), the present invention not only utilizes the powerful multilingual speech recognition capabilities of the Whisper model, but also embeds the complete transcription of the audio as a prompt into the decoder through an innovative fine-tuning strategy, overcoming the problem of traditional single reliance on acoustic features. In models such as Hubert and Wav2vec, feature extraction mainly relies on acoustic or time-frequency features, which makes it difficult to fully capture the context and semantic information in the speech, especially when processing long speech, which shows problems of feature sparsity and information loss.

[0064] In comparison, Whisper effectively aligns acoustic and semantic information by embedding the transcript of the speech, making it more robust in long audio processing and complex forged audio scenarios. In addition, the present invention breaks through the input length limit of 30-second audio clips of the Whisper model by effectively utilizing global speech information, and fine-tunes the global speech information through the sequence-to-sequence (Seq2Seq) paradigm, in the hope of improving the accuracy and generalization ability of the model in the task of audio authenticity identification. Compared with existing methods, this strategy performs well in multilingual environments and ensures the capture and detection of forged audio details through dual alignment of semantics and acoustics in various forgery types.

[0065] ii. LoRA empowers the Whisper model, achieving high-efficiency and low-cost breakthroughs in forged audio detection.

[0066] This paper applies the Low Rank Adaptive Tuning (LoRA) method to the Whisper model to address the computational cost and resource consumption issues in forged audio detection. Unlike existing transfer learning methods (such as Wav2vec and Hubert's full parameter tuning strategy), the LoRA method injects a low-rank factorization matrix into the linear weights of the Transformer attention layer and freezes all parameters of the Whisper model encoder, fine-tuning only a small number of gating weights. This significantly reduces the number of parameters that need to be updated, significantly reducing computational cost and video memory requirements.

[0067] In traditional methods such as Wav2vec and Hubert, fine-tuning usually requires updating most of the model parameters, which can easily lead to high video memory usage and low training efficiency, especially when training on large-scale audio datasets. The cost is high. However, the present invention uses the LoRA fine-tuning strategy, which only requires a graphics card with 20GB of video memory to efficiently train the Whisper base version with 74 million parameters, significantly improving resource utilization efficiency. In preliminary experiments, the LoRA fine-tuning strategy not only significantly reduced the training time and computing resource consumption of the model, but also demonstrated performance similar to or even better than the full fine-tuning method in the task of forged audio detection. This low-cost, high-efficiency optimization strategy provides a new solution for audio authenticity identification, and can provide strong support for large-scale industrial applications and cross-scenario deployment.

[0068] Compared with existing methods, the advantages of the Whisper and LoRA combination solution proposed in this invention are:

[0069] Small number of parameter updates and low computational cost: Compared with the traditional full-parameter fine-tuning strategy, the LoRA strategy only requires fine-tuning a small number of weight parameters, reducing hardware requirements and computational burden.

[0070] Strong ability to integrate acoustic and semantic information: By using speech transcription text as prompt information, the Whisper model demonstrates stronger robustness and generalization capabilities in long speech and complex forgery scenarios.

[0071] Wide applicability: In the detection of multiple languages, multiple scenarios, and future unknown forgery types, it has stronger generalization capabilities and lower costs, laying a technical foundation for large-scale deployment.

[0072] It is worth mentioning that the technical features such as the large-scale pre-training model Whisper involved in the patent application of this invention should be regarded as prior art. The specific structure, working principle and possible control method and spatial layout method of these technical features can be selected by conventional choices in the field and should not be regarded as the inventive point of the patent of this invention. The patent of this invention will not be further elaborated.

[0073] For those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned embodiments, or to make equivalent replacements for some of the technical features therein. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.< / startoftranscript> < / startofprev> < / startoftranscript> < / startofprev>

Claims

1. A deep fake audio detection and protection method based on the large-scale pre-trained Whisper model, characterized by: The following steps are involved: Step S1: Input the audio to be tested into the pre-trained Whisper model, and use the transcript of the entire audio as a prompt during the fine-tuning process of each audio segment, so that the Whisper model can utilize global text information; Step S2: During the training and decoding process, the audio data is preprocessed to adapt the model Whisper; Step S3: In terms of loss function design, based on the default cross entropy loss function, a detection cross entropy loss function is designed to evaluate the performance of the Whisper model during the training process; In step S1: Define audio as , the full text of Whisper model transcription is defined as , the prediction process of model Whisper for each segment is expressed as: in, represents the input token sequence to the decoder in the model, is the classification result of the Whisper model. During the training process, the prompt of the transcribed text should be used as a reference rather than a prediction target. Therefore, in the input and target of the decoder, The position of the prompt that marks the beginning of the mark remains unchanged. Model Whisper directly outputs the same content according to the input. Mark and start predicting subsequent content; During the testing phase, it is necessary to provide not only the prefix but also the complete text as a prompt to ensure that the Whisper model refers to the complete text information when generating output; In step S2: For long audio that exceeds the preset time, the Whisper model is used to split it into smaller decoded segments and record the end time of each segment. Subsequently, a greedy merging strategy is used to sequentially merge the decoded segments into merged segments so that their length is close to but not longer than the preset time. The remaining decoded segments that cannot be merged are discarded. For short audio clips that do not exceed the preset time, they are extended to the specified length through repetition and splicing to meet the input requirements of the Whisper model. The preprocessed audio data is finally used for training and testing the Whisper model. In step S3, the loss calculation is performed in steps to ensure that the model Whisper is penalized for prediction errors within the key segments of the sequence, as shown below: Total loss is the global loss and local intermediate losses A weighted combination of global losses Covering the entire sequence, it is represented as; in, represents the cross entropy loss function, is the predicted output of the model, is a real label, and Respectively indicate the position The predicted value and true value at ; then, assuming that indivual The marked position is , and the indivual Mark the location , local intermediate loss Focus on and The subsequence between , that is, the prediction result of the audio authenticity, is expressed as: The final loss value is calculated as ,in is a weighting factor used to increase the impact of local intermediate loss; During the training phase, the audio sample to be tested first reaches a random decision point, where it is randomly determined whether to perform speech enhancement processing on it. This improves the diversity of the audio and enables the Whisper model to be trained in a more realistic environment. The processed audio data is then fed into the Whisper model. At this point, the encoder portion of the Whisper model is frozen to ensure that its pre-trained parameters remain stable during training, enabling the Whisper model to robustly extract features from the audio data. A loss function is then used to calculate the loss value based on the predicted results and the true labels, which is then used to update the Whisper model's parameters. A low-rank adaptive fine-tuning method is applied to the Whisper model. This method injects a low-rank decomposition matrix into the linear weights of the Transformer attention layer, freezes all parameters of the Whisper model encoder, and only fine-tunes a small number of gating weights. This greatly reduces the number of parameters that need to be updated, significantly reducing computational costs and memory requirements.

Images