A Fake Audio Detection Method and System Based on Data-Guided Feature Distillation
By using a data-guided feature distillation method, which utilizes features from the teacher model to guide the student model, the robustness of fake audio detection in low-quality scenarios is solved, achieving higher detection accuracy and generalization ability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ANHUI UNIV
- Filing Date
- 2023-04-25
- Publication Date
- 2026-07-17
AI Technical Summary
Existing methods for detecting fake audio lack robustness in low-quality scenarios and are difficult to effectively identify fake audio.
A fake audio detection method based on data-guided feature distillation is adopted. Log power spectrum features are extracted by performing short-time Fourier transform on the original audio. Data augmentation is used to pre-train the teacher model, and the feature differences between the teacher and student models are calculated. The prediction of the student model is guided by KL divergence and mean square error function to prevent overfitting and improve the detection accuracy.
It significantly improves the accuracy of fake audio detection and robustness in low-quality scenarios, with the student model outperforming the teacher model on the original dataset.
Smart Images
Figure CN116434756B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of fake audio detection, and in particular to a fake audio detection method and system based on data-guided feature distillation. Background Technology
[0002] With the rise of Deepfake technology, fake news, scams, and other malicious events have become increasingly common. Speech playback, speech conversion, and speech synthesis seriously threaten the security of speaker verification systems, increasing the difficulty of fake speech detection (FSD). Currently, research on FSD mainly focuses on two aspects: its generalization ability to unknown attacks and its robustness in low-quality scenarios.
[0003] Previous work has primarily focused on generalization capabilities against unknown attacks. To improve the generalization ability of FSD, researchers have proposed numerous methods around front-end features and back-end classifiers, mainly categorized into end-to-end structures based on raw waveforms and non-end-to-end structures based on artificial features. These two structures each have their advantages in terms of generalization ability and model size. However, the performance of these methods deteriorates sharply when faced with datasets containing communication interference.
[0004] Therefore, there is an urgent need to provide a new method for detecting fake audio to solve the robustness problem in the aforementioned low-quality scenarios. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to provide a fake audio detection method and system based on data-guided feature distillation, which can significantly improve the accuracy of fake audio detection.
[0006] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is to provide a fake audio detection method based on data-guided feature distillation, comprising the following steps:
[0007] S1: Perform a short-time Fourier transform on the original audio to extract the logarithmic power spectrum, and use its F0 subband as the input feature of the network;
[0008] S2: Data augmentation is performed on the original data, and a teacher model is pre-trained. The teacher model learns on the data-augmented training set data and outputs prediction results and features at different depths.
[0009] S3: Use the model trained with the original data as the student model, calculate the loss between the predicted value and the label, and calculate the difference between the features of the teacher model and the student model at the same depth in feature learning, so that the student model can learn the feature distribution of the teacher model based on data augmentation.
[0010] S4: The teacher model provides further guidance on the prediction dimension and balances the loss to prevent the student model from overfitting the original data;
[0011] S5: Set the training rounds to train the student model, select the student model with the smallest loss on the validation set for testing, and use the output of the deepest network as the prediction result.
[0012] In a preferred embodiment of the present invention, step S1 specifically includes the following steps:
[0013] S101: Converting time-domain speech signals into time-frequency domain speech signals using Short-Time Fourier Transform (STFT):
[0014] X r [t, f] + i*X i [t, f] = STFT(X[k]) (1)
[0015] Where x[k] represents the speech signal in the time domain, and k is the time index of the speech signal. and These are the corresponding real and imaginary parts of the STFT, t is the index of the time frame, and f is the index of the frequency unit;
[0016] S102: The following operations are performed on the real and imaginary parts of the STFT to obtain the logarithmic power spectrum characteristics:
[0017]
[0018] Where log represents the logarithm operation, LPS full This is the full-band characteristic of the required logarithmic power spectrum;
[0019] S103: Apply the 0-400Hz frequency band of the logarithmic power spectrum as the required F0 subband:
[0020] LPS F0 =LPS 0-400HZ (3)
[0021] In a preferred embodiment of the present invention, in step S2, the teacher model is pre-trained on the training set data enhanced by Rawboost. During the training process, feature weights of different depths need to be retained, i.e.:
[0022]
[0023] Where NN represents a deep neural network model, DA represents data augmented by the Rawboost method, and predict... t This represents the prediction results of the teacher model. This represents the features of the teacher model trained at different network depths.
[0024] In a preferred embodiment of the present invention, in step S3, the loss of the feature mapping of each layer of network features between the teacher model and the student model is calculated using the MSE function:
[0025]
[0026] Where MSE represents the mean square error function. and These are the characteristics of each layer of the student network and the teacher network. The result is the loss function for each layer of features in both the teacher and student models.
[0027] In a preferred embodiment of the present invention, step S3 further includes calculating the loss between the predicted value and the label using the A-softmax function:
[0028]
[0029] Where A_softmax represents the A_softmax function, and predicts and label represent the predicted values and labels of the student model, respectively. This represents the loss between the student model's predicted values and the labels.
[0030] In a preferred embodiment of the present invention, step S4 specifically includes:
[0031] S401: Provide guidance on the prediction dimension by using KL divergence to calculate the difference between teacher and student predictions:
[0032]
[0033] Where KL represents the Kullback-Leibler Divergence function, predict s This is the prediction result from the student model. t This is the prediction result of the teacher model; The loss of the teacher model guides the student model in the prediction dimension;
[0034] S402: Balance the three losses—feature loss, prediction loss, and label loss—by setting two hyperparameters, α and β.
[0035]
[0036] α and β are the loss balancing coefficients, respectively. This is the final loss.
[0037] In a preferred embodiment of the present invention, the teacher model and the student model use Res2Net as the backbone network and introduce two attention mechanisms: global attention and local attention.
[0038] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is: to provide a fake audio detection system based on data-guided feature distillation, characterized in that it includes:
[0039] The speech feature input module is used to perform a short-time Fourier transform on the original audio, extract the logarithmic power spectrum, and use its F0 subband as the input feature of the network.
[0040] The pre-trained teacher module is used to augment the original data and pre-train the teacher model. The teacher model learns on the augmented training data and outputs prediction results and features at different depths.
[0041] The feature distillation module is used to calculate the differences between the features at each level of the teacher model and the student model at the same depth of feature learning, so that the student model learns the data-augmented feature distribution of the teacher model.
[0042] The predictive distillation module is used to guide the prediction of the student model, prevent the student model from overfitting, and combine the loss generated by the feature distillation module with that generated by this module to balance the feature differences between each level of the network.
[0043] The prediction output module is used to set the training rounds to train the student model, select the student model with the smallest loss on the validation set for testing, and output the deepest network as the prediction result.
[0044] In a preferred embodiment of the present invention, the feature distillation module allows the shallow and deep features of the teacher model to be used to guide the generation of shallow and deep features of the student model, and uses the mean squared error function to calculate the difference between the features of each layer of the teacher model and the student model, so that the student model learns the feature distribution derived by the teacher model based on data augmentation.
[0045] In a preferred embodiment of the present invention, the specific steps of the predictive distillation module guiding the prediction of the student model to prevent overfitting, and combining the losses generated by each module to balance the feature differences between different levels of the network include:
[0046] (1) Provide guidance on the prediction dimension by using KL divergence to calculate the difference between teacher and student predictions:
[0047]
[0048] Where KL represents the Kullback-Leibler Divergence function, predict s This is the prediction result from the student model. t This is the prediction result of the teacher model; The loss of the teacher model guides the student model in the prediction dimension;
[0049] (2) The feature loss, prediction loss, and label loss are balanced by setting two hyperparameters α and β:
[0050]
[0051] α and β are the loss balancing coefficients, This is the final loss.
[0052] The beneficial effects of this invention are:
[0053] The method described in this invention is the first to use data-guided feature distillation for fake audio detection. Through the distillation method, the student model learns features of different depths from the teacher model, and the teacher model guides the prediction of the student model in the prediction dimension, preventing overfitting. This allows the student model to achieve results that exceed the performance of the teacher model on the original dataset, thereby improving the accuracy of fake audio detection and its robustness in low-quality scenarios. Attached Figure Description
[0054] Figure 1 This is a flowchart of the fake audio detection method based on data-guided feature distillation of the present invention;
[0055] Figure 2 This is a schematic diagram of the fake audio detection method based on data-guided feature distillation and the network framework in the system.
[0056] Figure 3 This is a block diagram of the fake audio detection system based on data-guided feature distillation. Detailed Implementation
[0057] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.
[0058] Please see Figure 1 and Figure 2 The embodiments of the present invention include:
[0059] A method for detecting fake audio based on data-guided feature distillation includes the following steps:
[0060] S1: Perform a short-time Fourier transform on the original audio to extract the logarithmic power spectrum, and use its F0 subband as the input feature of the network; specific steps include:
[0061] S101: Converting time-domain speech signals into time-frequency domain speech signals using Short-Time Fourier Transform (STFT):
[0062] X r [t, f] + i*X i [t, f] = sTFT(X[k]) (1)
[0063] Where x[k] represents the speech signal in the time domain, and k is the time index of the speech signal. and These are the corresponding real and imaginary parts of the STFT, t is the index of the time frame, and f is the index of the frequency unit;
[0064] S102: The following operations are performed on the real and imaginary parts of the STFT to obtain the logarithmic power spectrum characteristics:
[0065]
[0066] Where log represents the logarithm operation, LPS full This is the full-band characteristic of the required logarithmic power spectrum;
[0067] S103: Apply the 0-400Hz frequency band of the logarithmic power spectrum as the required F0 subband:
[0068] LPS F0 =LPS 0-4oonz (3)
[0069] S2: Data augmentation is performed on the original data, and a teacher model is pre-trained. The teacher model learns on the data-augmented training set data and outputs prediction results and features at different depths.
[0070] The network is divided into a teacher network and a student network. The teacher model is trained using a dataset enhanced with Rawboost, while the student model is trained using the original dataset.
[0071] The teacher model is pre-trained on a training set enhanced with Rawboost, and feature weights of varying depths need to be retained during training. In other words, the teacher model should output not only predictions but also features of different depths.
[0072]
[0073] Where NN represents a deep neural network model, DA represents data augmented by the Rawboost method, and predict...t This represents the prediction results of the teacher model. This represents the features of the teacher model trained at different network depths, such as Figure 2 The features output by each module block of the teacher model.
[0074] S3: Use the model trained with the original data as the student model, calculate the loss between the predicted value and the label, and calculate the difference between the features of the teacher model and the student model at the same depth in feature learning, so that the student model can learn the feature distribution of the teacher model based on data augmentation.
[0075] Preferably, the A-softmax function is used to calculate the loss between the predicted value and the label:
[0076]
[0077] Where A_softmax represents the A_softmax function, predict s `label` and `predict` represent the predicted value and label of the student model, respectively. This represents the loss between the student model's predicted values and the labels. The teacher model also uses the A-softmax function to calculate the loss between the predicted values and the labels, which was already completed in the pre-training process of step S2.
[0078] The data features obtained from training at different network depths differ. Therefore, both the shallow and deep features of the teacher model are used to guide the generation of shallow and deep features in the student model. Preferably, the mean squared error function is used to calculate the differences in features at each layer between the teacher and student models, allowing the student model to learn the feature distribution derived from the teacher model based on data augmentation.
[0079] Specifically, the MSE function is used to calculate the loss of the feature mapping of each layer of network features between the teacher model and the student model:
[0080]
[0081] Where MSE represents the Mean Squared Error function. and These are the characteristics of each layer of the student network and the teacher network. The result is the loss function for each layer of features in both the teacher and student models.
[0082] In addition, such as Figure 2 As shown, the A-softmax function is used to calculate the loss between the predicted value and the label, exploring the potential features of the dataset.
[0083] S4: The teacher model further guides the prediction dimension and balances the loss to prevent the student model from overfitting the original data; specific steps include:
[0084] S401: Provide guidance on the prediction dimension by using KL divergence to calculate the difference between teacher and student predictions:
[0085]
[0086] Where KL represents the Kullback-Leibler Divergence function, predict s This is the prediction result from the student model. t This is the prediction result of the teacher model; The loss of the teacher model guides the student model in the prediction dimension;
[0087] S402: Balance the three losses—feature loss, prediction loss, and label loss—by setting two hyperparameters, α and β.
[0088]
[0089] α and β are the loss balancing coefficients, respectively. This is the final loss.
[0090] S5: Set the training rounds to train the student model, select the student model with the smallest loss on the validation set for testing, and use the output of the deepest network as the prediction result.
[0091] It should be noted that this invention uses Res2Net as the backbone network and introduces two attention mechanisms: global attention (squeeze and excitation, SE) and local attention (efficient channel attention, ECA), collectively referred to as SE2NET and ECA2NET. These two network architectures are used to verify the effectiveness of distillation. Furthermore, Adam is used as the optimizer during training, with weight decay set to 0.0001. Figure 2 The entire network comprises a teacher network and a student network, using Res2Net as the backbone network. The output of each block represents the feature outputs of students and teachers at different depths. First, the teacher network is pre-trained on a Rawboost-enhanced training set. The student model is trained on the original training set data, using the A-softmax function to calculate the loss between the predicted value and the label (Label loss). The mean squared error (MSE) function is used to calculate the loss between the features learned at each level of the teacher and student models (Feature loss), allowing the student model to acquire the feature distribution of the teacher model. Then, the prediction loss function is calculated for the prediction results of both the teacher and student models to prevent overfitting in the student's predictions. Finally, hyperparameters are set to balance the three loss functions. The student model is trained with a set number of training epochs. The student model with the smallest loss calculated by A-softmax between the predicted result and the label on the validation set is selected for testing. The output of the deepest network is used as the prediction result.
[0092] In the embodiments of the present invention, see Figure 3 Furthermore, a fake audio detection system based on data-guided feature distillation is provided, comprising:
[0093] The speech feature input module is used to perform a short-time Fourier transform on the original audio, extract the logarithmic power spectrum, and use its F0 subband as the input feature of the network.
[0094] The pre-trained teacher module is used to augment the original data and pre-train the teacher model. The teacher model learns on the augmented training data and outputs prediction results and features at different depths.
[0095] The feature distillation module is used to calculate the differences between the features at each level of the teacher model and the student model at the same depth of feature learning, so that the student model learns the data-augmented feature distribution of the teacher model.
[0096] The predictive distillation module is used to guide the prediction of the student model, prevent the student model from overfitting, and combine the loss generated by the feature distillation module with that generated by this module to balance the feature differences between each level of the network.
[0097] The prediction output module is used to set the training rounds to train the model, select the model with the smallest loss on the validation set for testing, and output the deepest network as the prediction result.
[0098] The feature distillation module allows the shallow and deep features of the teacher model to guide the generation of shallow and deep features of the student model, and uses the mean squared error function to calculate the difference between the features of each layer of the teacher model and the student model, so that the student model can learn the feature distribution derived by the teacher model based on data augmentation.
[0099] The predictive distillation module guides the student model's predictions, prevents overfitting, and, by incorporating the losses from each module, balances the feature differences between network levels. Specific steps include:
[0100] (1) Provide guidance on the prediction dimension by using KL divergence to calculate the difference between teacher and student predictions:
[0101]
[0102] Where KL represents the Kullback-Leibler Divergence function, predict s This is the prediction result from the student model. t This is the prediction result of the teacher model; The loss of the teacher model guides the student model in the prediction dimension;
[0103] (2) The feature loss, prediction loss, and label loss are balanced by setting two hyperparameters α and β:
[0104]
[0105] α and β are the loss balancing coefficients, respectively. This is the final loss.
[0106] This invention was tested on the public datasets ASVspoof 2019LA, ASVspoof 2021LA, and DF. Equal error rate (EER) and minimum normalized concatenated detection cost function (min-tDCF) were used to quantitatively evaluate the fake audio detection results.
[0107] Table 1
[0108]
[0109] Table 2
[0110]
[0111]
[0112] Table 1 shows the experimental results based on different architectures of ECA2Net and SE2Net, and Table 2 shows the experimental results compared with methods in other papers. The EER and t-DCF on the raw speech, DA, and KD systems of the ASVspoof2019 LA dataset are shown in Table 1, where "DA" represents data augmentation and "KD" represents knowledge distillation. Table 1 shows that the KD system outperforms the DA system, demonstrating the effectiveness of our proposed method. On the ASVspoof2021 LA dataset, the performance using raw speech is significantly lower than on the ASVspoof2019 LA dataset, possibly due to noise causing the loss of important information. On all three datasets, the KD results are lower than the DA results because the student model has both the ability to transfer knowledge from the teacher model to generalize and the possibility of losing important information when trained with raw data. Table 2 shows that our proposed method outperforms other methods on the same datasets.
[0113] These results demonstrate the effectiveness of the method proposed in this invention. Furthermore, it is shown that using enhanced data to guide the student model can improve the robustness and generalization of FSD.
[0114] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.
Claims
1. A method for detecting fake audio based on data-guided feature distillation, characterized in that, Includes the following steps: S1: Perform a short-time Fourier transform on the original audio to extract the logarithmic power spectrum, and use its F0 subband as the input feature of the network; S2: Data augmentation is performed on the original data, and a teacher model is pre-trained. The teacher model learns on the data-augmented training set data and outputs prediction results and features at different depths. The teacher model is pre-trained on the training set data enhanced by Rawboost. During training, feature weights at different depths need to be retained, i.e.: , in, Represents a deep neural network model. This indicates data enhanced using the Rawboost method. This represents the prediction results of the teacher model. This represents the features of the teacher model trained at different network depths; S3: Use the model trained with the original data as the student model, calculate the loss between the predicted value and the label, and calculate the difference between the features of the teacher model and the student model at the same depth in feature learning, so that the student model can learn the feature distribution of the teacher model based on data augmentation. S4: The teacher model provides further guidance on the prediction dimension and balances the loss to prevent the student model from overfitting the original data; S5: Set the training rounds to train the student model, select the student model with the smallest loss on the validation set for testing, and use the output of the deepest network as the prediction result.
2. The fake audio detection method based on data-guided feature distillation according to claim 1, characterized in that, The specific steps of step S1 include: S101: Converting time-domain speech signals into time-frequency domain speech signals using Short-Time Fourier Transform (STFT): ， Where x[k] represents the speech signal in the time domain, and k is the time index of the speech signal. and These are the corresponding real and imaginary parts of the STFT, t is the index of the time frame, and f is the index of the frequency unit; S102: The following operations are performed on the real and imaginary parts of the STFT to obtain the logarithmic power spectrum characteristics: ， Where log represents the logarithm operation. This is the full-band characteristic of the required logarithmic power spectrum; S103: Apply the 0-400Hz frequency band of the logarithmic power spectrum as the required F0 subband: 。 3. The fake audio detection method based on data-guided feature distillation according to claim 1, characterized in that, In step S3, the A-softmax function is used to calculate the loss between the predicted value and the label: ， Where A_softmax indicates that it is the A_softmax function. and These represent the predicted value and label of the student model, respectively. This represents the loss between the student model's predicted values and the labels.
4. The fake audio detection method based on data-guided feature distillation according to claim 1, characterized in that, In step S3, the MSE function is used to calculate the loss of the feature mapping of each layer of network features between the teacher model and the student model: ， Where MSE represents the mean square error function. and These are the characteristics of each layer of the student network and the teacher network. The result is the loss function for each layer of features in both the teacher and student models.
5. The fake audio detection method based on data-guided feature distillation according to claim 1, characterized in that, The specific steps of step S4 include: S401: Provide guidance on the prediction dimension by using KL divergence to calculate the difference between teacher and student predictions: ， Where KL represents the Kullback-Leibler Divergence function. These are the prediction results from the student model. This is the prediction result of the teacher model; The loss of the teacher model guides the student model in the prediction dimension; S402: By setting two hyperparameters and To balance the three losses: feature loss, prediction loss, and label loss: ， and These are the loss balancing coefficients. This is the final loss.
6. The fake audio detection method based on data-guided feature distillation according to claim 1, characterized in that, The teacher and student models use Res2Net as the backbone network and introduce two attention mechanisms: global attention and local attention.
7. A fake audio detection system based on data-guided feature distillation, characterized in that, include: The speech feature input module is used to perform a short-time Fourier transform on the original audio, extract the logarithmic power spectrum, and use its F0 subband as the input feature of the network. The pre-trained teacher module is used to augment the original data and pre-train a teacher model. This teacher model learns and outputs predictions and features of different depths on the augmented training set. The teacher model is then pre-trained on the Rawboost-enhanced training set, retaining the feature weights of different depths during training. ， in, Represents a deep neural network model. This indicates data enhanced using the Rawboost method. This represents the prediction results of the teacher model. This represents the features of the teacher model trained at different network depths; The feature distillation module is used to calculate the differences between the features at each level of the teacher model and the student model at the same depth of feature learning, so that the student model learns the data-augmented feature distribution of the teacher model. The predictive distillation module is used to guide the prediction of the student model, prevent the student model from overfitting, and combine the loss generated by the feature distillation module with that generated by this module to balance the feature differences between each level of the network. The prediction output module is used to set the training rounds to train the student model, select the student model with the smallest loss on the validation set for testing, and output the deepest network as the prediction result.
8. The fake audio detection system based on data-guided feature distillation according to claim 7, characterized in that, The feature distillation module allows the shallow and deep features of the teacher model to guide the generation of shallow and deep features of the student model, and uses the mean squared error function to calculate the difference between the features of each layer of the teacher model and the student model, so that the student model can learn the feature distribution derived by the teacher model based on data augmentation.
9. The fake audio detection system based on data-guided feature distillation according to claim 7, characterized in that, The predictive distillation module guides the student model's predictions, prevents overfitting, and, by incorporating the losses from each module, balances the feature differences between network levels. Specific steps include: (1) Provide guidance on the prediction dimension by using KL divergence to calculate the difference between teacher and student predictions: ， Where KL represents the Kullback-Leibler Divergence function. These are the prediction results from the student model. This is the prediction result of the teacher model; The loss of the teacher model guides the student model in the prediction dimension; (2) By setting two hyperparameters and To balance the three losses: feature loss, prediction loss, and label loss: ， and These are the loss balancing coefficients. This is the final loss.