Deep forged voice detection method based on multi-teacher knowledge distillation

Abstract

The invention discloses a deep forged voice detection method based on multi-teacher knowledge distillation, and belongs to the technical field of voice processing and safety detection. According to the technical scheme, two teacher models are used for respectively focusing on detection tasks of synthetic speech and converted speech, wherein one teacher model focuses on synthetic speech detection by utilizing emotional characteristics, and the other teacher model focuses on converted speech detection by adopting a double-branch network structure; through a multi-teacher knowledge distillation mechanism, the special knowledge learned by the two teacher models is transmitted to the student model. The method has the beneficial effects that the method has remarkable advantages in the aspects of detection range, feature utilization, model lightweight, generalization ability, actual deployment efficiency and the like, and a reliable technical scheme is provided for precise recognition and safety protection of deeply-forged voice.

Description

Technical Field

[0001] This invention relates to the field of speech processing and security detection technology, and in particular to a deepfake speech detection method based on multi-teacher knowledge distillation, which is used to accurately identify deepfake speech such as synthesized speech and converted speech, and to ensure the authenticity and security of speech content. Background Technology

[0002] With the development of speech synthesis and conversion technologies, deepfake speech is increasingly prevalent in cyberspace, posing serious security risks to fields such as financial transactions, identity authentication, and media dissemination. Existing deepfake speech detection methods have several limitations: First, most detection models are designed for a single type of forged speech (such as synthesized speech only or converted speech only), making it difficult to effectively detect both synthesized and converted speech simultaneously. Second, sentiment analysis methods commonly used in synthesized speech detection often fail in converted speech detection scenarios, failing to capture the artifact features unique to converted speech. Third, some detection models have complex structures and consume significant computational resources, making them difficult to apply in resource-constrained scenarios such as mobile terminals and embedded devices. Therefore, there is an urgent need for a deepfake speech detection solution that can simultaneously detect both synthesized and converted speech, fully utilize features, and is lightweight. Summary of the Invention

[0003] To address the technical problems existing in the prior art, this invention provides a deep forged speech detection method based on multi-teacher knowledge distillation. This method uses two teacher models to focus on the detection tasks of synthesized speech and converted speech, respectively: one teacher model focuses on synthesized speech detection using emotional features, while the other teacher model uses a dual-branch network structure to focus on converted speech detection. Through the multi-teacher knowledge distillation mechanism, the proprietary knowledge learned by each teacher model is transferred to the student model. This collaborative approach enables the student model to more accurately capture the unique forged features of synthesized and converted speech, thereby improving the detection performance of these two types of forged speech and avoiding the feature confusion and performance degradation problems that may occur when a single model learns two types of features simultaneously.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A deep forgery speech detection method based on multi-teacher knowledge distillation, comprising the following steps:

[0006] S1. Constructing a TTS teacher model: A two-way time-aware multi-scale emotion recognition model is used as an emotion feature extractor, followed by a ResNet18 classification network and a single-classification loss function OC-Softmax, focusing on synthesized speech detection;

[0007] S2. Constructing the VC teacher model: A dual-branch network architecture is adopted, including the LFCC branch and the CQT branch. Each branch uses a ResNet18 network with a convolutional attention module. Transformed speech detection and forgery type classification are achieved through multi-task learning.

[0008] S3. Constructing a student model: An LFCC feature extraction layer is connected to a ResNet18 classification network, and an OC-Softmax loss function is applied after the fully connected layer for lightweight fake speech detection.

[0009] S4. Perform the independent training process for the teacher model:

[0010] TTS teacher model training: Using several synthesized speech samples and several real speech samples from the ASVspoof2019 LA training set, the model was optimized based on the difference in emotional features. The compactness of the distribution of emotional features in the synthesized speech was verified by t-SNE visualization.

[0011] VC teacher model training: Using several converted speech samples and several real speech samples from the ASVspoof2019 LA training set, combined with converted speech samples from the ASVspoof2015 training set, detection performance is improved by capturing transient anomalies and capturing complementary spectral distortion features.

[0012] S5. Optimize knowledge distillation and student model:

[0013] Data routing mechanism: For real speech, a random number R between 0 and 1 is generated. When R = 0, it is input into the TTS teacher model, and when R = 1, it is input into the VC teacher model. For fake speech, it is input into the corresponding teacher model according to the type label.

[0014] S6. Establish a loss function system:

[0015] TTS distillation loss: in Embed the sentiment features output by the TTS teacher model. Features corresponding to the student model;

[0016] VC distillation loss: Branch loss:

[0017] Student classification loss: OC-Softmax loss function The parameters are set as α = 20, m0 = 0.9, m1 = 0.2;

[0018] Overall optimization goals: The Adam optimizer (β1 = 0.9, β2 = 0.999, ∈ = 10) was used. -8The weight decays by 0.0005, the learning rate is 0.0003, and the weight decays by 0.5 times every 10 rounds, with 100 training rounds.

[0019] Furthermore, the TIM-Net architecture of the TTS teacher model includes:

[0020] Bidirectional feature processing module: Extracts 39-dimensional MFCC features from the input speech, performs time reversal to generate reversed feature sequence, and inputs the forward and reversed features into the forward and reverse branches respectively;

[0021] Time-aware blocks: Each branch contains multiple time-aware blocks, each block consisting of dilated causal convolutions, batch normalization layers, and ReLU activation functions. Dilated convolutions are used to expand the temporal receptive field, and causal convolutions ensure the isolation of non-causal information.

[0022] Multi-scale feature fusion module: fuses the outputs of each block through global temporal pooling. Final emotional characteristic F emo It is generated by weighted summation of the outputs of each block, with weight w. i Learning through backpropagation.

[0023] Furthermore, the dual-branch network implementation of the VC teacher model is as follows:

[0024] LFCC branch: Extract 60-dimensional LFCC features and input them into a 4-layer ResNet18 network. Each residual block contains a convolutional attention module, and channel attention and spatial attention optimization are performed sequentially to detect vocal tract abnormalities in fake speech.

[0025] CQT branch: Extract 100-dimensional CQT features with parameters fmin=3.5 and B=48, input to a 4-layer ResNet18-CBAM network, and use variable time-frequency resolution to capture fundamental frequency and harmonic distortion;

[0026] Multi-task learning: Simultaneously outputs true / false classification results and fake type classification results, with the loss function being branch-weighted cross-entropy, and the weights being equal.

[0027] Furthermore, the lightweight design of the student model includes:

[0028] Feature extraction layer: 60-dimensional LFCC features are used, and the length is fixed to 750 frames through frame segmentation and padding / cropping;

[0029] Classification network: 4-layer ResNet18 network, each residual block consists of a one-dimensional convolutional layer, a batch normalization layer, and a ReLU activation function;

[0030] Generalization enhancement: OC-Softmax improves the detection capability against unknown forgery algorithms by compressing the distribution of real speech features and expanding the distribution of forged speech features.

[0031] The present invention also includes a deep forgery speech detection system based on multi-teacher knowledge distillation, comprising:

[0032] TTS Teacher Model Module:

[0033] TIM-Net sentiment feature extraction unit: implements MFCC forward and reverse processing and time-aware block computation;

[0034] ResNet18-OC classification unit: outputs synthesized speech detection results based on emotion features; VC teacher model module:

[0035] LFCC-CQT dual-branch feature extraction unit: processes linear frequency and constant Q transform features in parallel;

[0036] Multi-task classification unit: ResNet18 network integrating CBAM, outputting true / false and type classification results;

[0037] Student Model Module:

[0038] LFCC Fast Extraction Unit: Real-time feature calculation based on short-time Fourier transform;

[0039] Lightweight ResNet18 classification unit: supports end-to-end detection using a single-classification loss function; Knowledge distillation execution module:

[0040] Data routing engine: dynamically assigns data to the corresponding teacher model based on voice tags;

[0041] Loss Fusion Calculator: Real-time calculation of MSE distillation loss and OC-Softmax classification loss to drive student model parameter updates.

[0042] The beneficial effects of this invention are:

[0043] Compared with existing technologies, the deep forgery speech detection method based on multi-teacher knowledge distillation described in this invention has the following technical features and beneficial effects:

[0044] (1) Multi-type forged speech detection capability: By introducing two teacher models with distinct functions (TTS teacher model focuses on synthesized speech detection, and VC teacher model focuses on converted speech detection), and using knowledge distillation technology to transfer the proprietary knowledge of the two to the lightweight student model, this invention can simultaneously and efficiently detect synthesized speech and converted speech, solving the problem that existing methods only target a single type of forgery, resulting in a limited detection range.

[0045] (2) Feature extraction and classification performance optimization: The TTS teacher model adopts a bidirectional time-aware multi-scale emotion recognition network (TIM-Net) and combines emotion feature analysis to effectively capture artifact features caused by the lack of multimodal information in synthesized speech; the VC teacher model adopts a dual-branch network (LFCC+CQT complementary features) and combines a convolutional attention module (CBAM) with multi-task learning to comprehensively extract artifact features such as vocal tract abnormalities and spectral distortion in converted speech; the student model uses the OC-Softmax single-classification loss function to compress the distribution of real speech features and expand the distribution of fake speech features, significantly improving the generalization ability to unknown fake algorithms and alleviating the problem of differences in training and testing data distribution.

[0046] (3) Lightweight and efficient deployment: The student model only needs to extract LFCC features and adopts a simplified residual network structure, which greatly reduces the computational complexity while ensuring detection accuracy. It is suitable for resource-constrained scenarios such as mobile terminals and embedded devices.

[0047] (4) Robustness across datasets and algorithms: The ASVspoof2015 and 2019 datasets are integrated in the training of the VC teacher model to enhance data diversity and improve the model’s adaptability to unknown transformation algorithms. Experiments show that on the ASVspoof2019LA test set, the overall equal error rate (EER) of the student model is ≤1.84%, and the EER is reduced by ≥16% in cross-database tests (such as ASVspoof2015), which verifies the strong generalization of the method.

[0048] (5) Experimental performance is significantly improved:

[0049] The TTS teacher model achieves an EER of ≤0.38% for synthesized speech detection, while the VC teacher model achieves an EER of ≤0.88% for converted speech detection. The student model outperforms traditional single-model and binary classification methods in overall performance. Through the knowledge distillation mechanism, the student model inherits the advantages of the dual-teacher model, maintaining high detection accuracy while reducing the number of parameters, thus achieving a balance between efficiency and performance.

[0050] This invention has significant advantages in terms of detection range, feature utilization, model lightweighting, generalization ability, and actual deployment efficiency, providing a reliable technical solution for accurate recognition and security protection of deepfake speech. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] in:

[0053] Figure 1 This is a diagram illustrating the overall framework of a forged speech detection method based on multi-teacher knowledge distillation.

[0054] Figure 2 Flowchart for training a student model for a forged speech detection method based on multi-teacher knowledge distillation;

[0055] Figure 3 The residual block structure diagram of the student model in the forgery detection method based on multi-teacher knowledge distillation;

[0056] Figure 4 This is a schematic diagram of OC-Softmax;

[0057] Figure 5 Flowchart for testing a forged speech detection method based on multi-teacher knowledge distillation;

[0058] Figure 6 A structural diagram of the emotion feature extraction model;

[0059] Figure 7 A diagram of the time-aware block structure;

[0060] Figure 8 Here is a diagram of the VC teacher model structure;

[0061] Figure 9 This is a residual block structure diagram of the VC teacher model. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The following description, in conjunction with the accompanying drawings... Figure 1-9 The deepfake speech detection method based on multi-teacher knowledge distillation is further explained.

[0063] Example 1

[0064] The overall framework of the forged speech detection method based on multi-teacher knowledge distillation is as follows: Figure 1As shown, the model consists of three parts: a TTS teacher model, a VC teacher model, and a student model. The training process is as follows: First, the TTS teacher model is trained using synthesized and real speech from the ASVspoof2019LA training set, enabling it to learn unique artifact features in synthesized speech. Next, the VC teacher model is trained using converted speech and real speech from the ASVspoof2019 LA training set, allowing it to learn unique artifact features in converted speech. Since the converted speech in the ASVspoof2019 LA training set is relatively small, accounting for only one-third of the fabricated speech in the dataset (a total of 7600 converted speech samples), to enhance the diversity of the training data, this method additionally includes converted speech from the ASVspoof2015 training set when training the VC teacher model. Finally, the student model is trained using the two pre-trained teacher models and Mean Squared Error (MSE) loss distillation.

[0065] The training process of the student model is as follows: Figure 2 As shown, during the training process of the student model, the input fake speech includes two types: synthesized speech and converted speech. First, the authenticity of the input speech is determined based on its label. If the input is real speech, a random number R is generated, and the real speech is input into either the TTS teacher model or the VC teacher model for distillation training based on the generated R. If the input is fake speech, it is determined whether it is synthesized or converted speech based on the fake speech type label. If it is synthesized speech, it is input into the corresponding TTS teacher model for distillation training; if it is converted speech, it is input into the corresponding VC teacher model for distillation training.

[0066] Student Model

[0067] 1. Student Model

[0068] The overall structure of the student model is shown in the overall framework of this method. Figure 1 Speech features extracted from the student model

[0069] LFCC is one of the most widely used traditional hand-crafted features in the field of fake speech detection, characterized by uniform frequency resolution.

[0070] Its high computational efficiency and sensitivity to high-frequency information make it well-suited for speech forgery detection tasks. (Student model)

[0071] The classification network of this type contains four structurally consistent residual blocks, the specific composition of which is as follows: Figure 3 As shown.

[0072] Since the feature distributions of synthesized speech and converted speech are different, and the forgery algorithms in the training data and test data are also different, this method introduces a one-class softmax (OC-Softmax) loss function after the fully connected layer of the student model to enhance the model's generalization ability to different types of forgery algorithms and unknown forgery algorithms.

[0073] The main idea of ​​a single-class loss function is to construct a decision boundary that can distinguish between target class samples and non-target class samples by learning the feature distribution of the target class. Most research on fake speech detection usually assumes that real speech and fake speech in the training and test data have similar feature distributions. Therefore, many detection models use binary classification loss functions to learn the classification boundary between real and fake speech. This assumption is reasonable for real speech data, but not for fake speech. Because with the continuous development of speech forgery technology, new forgery algorithms are emerging one after another, and the forgery algorithms in the current forgery detection training dataset cannot keep up with the forgery technology used in actual applications. Therefore, there is a large difference between the forgery feature distribution in the training data and the test data, which makes the single-class loss function more suitable for fake speech detection tasks. The OC-Softmax method used in this paper treats real speech as the target class and fake speech as the non-target class. It optimizes the feature space by introducing two different boundary margins. For the target class, OC-Softmax uses a smaller boundary margin to compress its feature distribution, making the feature space of real speech samples more compact. For the non-target class, a larger boundary margin is used to expand its feature distribution, making the feature space of fake speech samples more dispersed. The specific calculation formula of OC-Softmax is as follows:

[0074]

[0075] Here, w0 is a weight vector representing the optimization direction of the target class features. xi and yi represent the feature vector and label of the i-th sample, respectively, yi∈{0,1}, with the target class label being 0 and the non-target class label being 1. w^ and x^ represent the normalized w and x, respectively. α is a scaling factor. m0 and m1 represent two boundaries (m0,m1∈[-1,1] and m0>m1), used to constrain the angle θi between the weight vector w0 and the sample xi. When yi=0, m0 makes θi less than arccos m0, while when yi=1, m1 makes θi greater than arccos m1. Figure 4 As shown, a smaller arccos m0 can concentrate target class samples around the weight vector w0, while a relatively larger arccos m1 helps to separate non-target class samples from w0.

[0076] 2. TTS Knowledge Distillation Module

[0077] The TTS knowledge distillation module utilizes the TTS teacher model to teach the student model unique artifact information in synthesized speech, thereby improving the student model's performance in detecting synthesized speech. Text-to-speech (TTS) speech synthesis technology takes plain text as input, and the emotional expression of text may differ across different contexts. Furthermore, emotional expression typically relies on multimodal information such as facial expressions and body language, while speech synthesis algorithms depend solely on text.

[0078] Input difficulties and the inability to capture this information make it challenging for speech synthesis technology to accurately generate natural emotional information. This paper, through playback tests, finds that synthesized speech in publicly available datasets for fake speech detection often sounds neutral and lacks emotional variation. Therefore, our TTS teacher model uses emotional features as the key feature to distinguish between real and fake speech.

[0079] The TTS teacher model employs a bidirectional time-aware multi-scale emotion recognition model (TIM-Net) as the speech emotion feature extractor. The model structure is as follows: Figure 6 As shown, since judging speech emotion requires combining global contextual information of the speech, the TIM-Net model is designed with a bidirectional architecture, which processes the forward and reverse signals of the input speech through two branches to fuse the information of the preceding and following sequences of the speech.

[0080] The TIM-Net model first extracts the Mel-frequency cepstral coefficients (MFCCs) of the speech, and then performs a time-reversal operation on the MFCC features. Since the speech signal is divided into several short frames through framing before MFCC feature extraction, MFCC features are typically extracted frame-by-frame. The time-reversal of MFCC features involves rearranging the features of each frame in reverse order. Next, the TIM-Net model inputs the forward-order and reverse-order MFCC features into the forward and reverse branches, respectively, which consist of multiple time-aware blocks. The structure of the time-aware block is as follows: Figure 7 As shown, it utilizes dilated causal convolution. Dilated convolution expands the receptive field of the convolution, while causal convolution ensures that the convolution can only access current and past data, avoiding interference from future data in the current prediction. The model combines the output features of each corresponding forward and backward time-aware block to fuse the bidirectional semantic dependencies of speech, thereby enriching the contextual representation of the features. Its formula is as follows:

[0081]

[0082] Where i represents the sequence number of the time-aware block, and G represents the average value of the global time pooling operation over the time dimension. This represents the output features of the positive time-aware block. This represents the output features of the inverse time-aware block. Finally, the TIM-Net model performs a weighted summation of g1 to gn to fuse features from multiple time scales, thereby better adapting to changes in speech emotion. The formula for combining g1 to gn is as follows:

[0083]

[0084] Where n represents the number of time-aware blocks, and wi is a learnable parameter representing the dynamic weight of gi.

[0085] This method uses the SAVEE English dataset, which contains seven emotions (anger, disgust, fear, happiness, neutrality, sadness, and surprise), to train a TIM-Net model as its emotion feature extractor, used to extract emotion features from both real and synthesized speech. Here, the method uses the final fused features of this model as the extracted emotion features.

[0086] This method connects a ResNet18 classification network after the sentiment feature extractor, aiming to enable ResNet18 to learn more discriminative deep features from sentiment characteristics. The structure of the ResNet18 residual blocks is consistent with that of the student model. Furthermore, similar to the student model, the TTS teacher model adds a single-classification loss function after the fully connected layers of the classification network to help the model further improve its generalization ability in forgery detection. The forgery data used in the training process of the TTS teacher model is only synthesized speech from the ASVSpoof2019 training set.

[0087] The TTS knowledge distillation module transfers the detection knowledge of synthesized speech to the student model through the aforementioned TTS teacher model. This knowledge transfer process can be achieved using mean squared error loss (MSE), the formula of which is shown below:

[0088]

[0089] Where N represents the number of synthesized speech samples in the training set. and and represent the feature embeddings generated after inputting the i-th speech sample into the TTS teacher model and student model, respectively.

[0090] 3. VC Knowledge Distillation Module

[0091] This method selects a detection model (Dual-Branch Network) different from the TTS teacher model as the VC teacher model for converted speech. The structure of this model is as follows: Figure 8As shown, this model employs a dual-branch network structure, leveraging the complementarity of different features to enhance the detection performance of forged speech. The model first extracts the linear frequency cepstral coefficients (LFCC) and constant-Q transform (CQT) of the input speech as inputs to both branches. LFCC, based on short-time Fourier transform, is suitable for capturing transient anomalies in speech signals and is robust to noise and channel distortion. Simultaneously, it primarily reflects the vocal tract characteristics of speech through cepstral analysis, making it suitable for detecting vocal tract anomalies in forged speech. CQT, on the other hand, has variable time-frequency resolution, with high frequency resolution in the low-frequency region, suitable for analyzing pitch and harmonics, and high time resolution in the high-frequency region, suitable for capturing rapidly changing signals. It is sensitive to the fundamental frequency and harmonic structure, directly reflecting spectral features, making it suitable for detecting spectral distortion and traces of human manipulation in forged speech. LFCC and CQT complement each other from multiple dimensions, comprehensively capturing the forgery features of forged speech and improving forgery detection performance. The classification network structure of the two branches of the Dual-Branch Network model is consistent, both using the ResNet18 network, where the residual block structure is as follows... Figure 9 As shown, compared to the residual blocks of a regular ResNet18 network, this residual block incorporates a Convolutional Block Attention Module (CBAM) to enhance the model's ability to represent features. CBAM consists of a channel attention module and a spatial attention module connected in series, simultaneously enhancing the model's ability to extract important features in both channel and spatial dimensions while suppressing interference from irrelevant or noisy information.

[0092] The Dual-BranchNetwork model incorporates multi-task learning during training, including a fake speech detection task and a fake speech type classification task, which are trained jointly. The fake speech type classification task helps the model learn common fake speech features from different fake speech types, thereby improving the model's adversarial capability against unknown fake attacks. The model uses the Cross Entropy Loss function in both the real / fake speech classification module and the fake speech type detection module, as shown in the following formula:

[0093]

[0094] Where N represents the number of speech samples, and K represents the number of categories of speech samples. and Let y represent the predicted probabilities of the LFCC branch and the CQT branch predicting that the i-th sample belongs to the k-th class, respectively. i,k Let y represent the sample label, where y is defined as the label of the i-th sample. i,k=1. To enable the VC teacher model to learn the unique artifact information of converted speech, this method selects only converted speech as spoof data to train the Dual-Branch Network model. Since the number of converted speech samples in the ASVspoof2019 LA training set is relatively small, this method additionally includes converted speech samples from the ASVspoof2015 training set when training the VC teacher model to increase the diversity of the training data.

[0095] The VC knowledge distillation module uses distillation loss to transfer the converted speech detection knowledge learned by the two branches of the VC teacher model to the student model. The specific loss function can be expressed as:

[0096]

[0097] in, and Let represent the distillation losses of the LFCC and CQT branches of the VC teacher model and the student model, respectively, and let N represent the number of transcribed speech samples in the training set. and These represent the feature representations generated after inputting the i-th speech sample into the LFCC branch and CQT branch of the student model and VC teacher model, respectively.

[0098] The total loss of the VC knowledge distillation module is due to and The weighted composition, where λ and γ represent the weights of the distillation losses in the two branches, respectively.

[0099] Example 2

[0100] environment

[0101] The experimental environment for this method is based on the Ubuntu operating system, using PyTorch version 1.9.1 as the deep learning framework and Python version 3.9.16 as the programming language. A GPU server is used in this experiment to provide efficient computing and training capabilities.

[0102] This method utilizes three speech features: 39-dimensional MFCC, 60-dimensional LFCC, and 100-dimensional CQT. During feature extraction, the pre-emphasis parameter is set to 0.97, the frame length is 20 milliseconds, and the frame shift is 10 milliseconds. For CQT feature extraction, the parameter fmin is set to 3.5, and B is set to 48. During experiments, to enable batch processing of data, each feature was either padded or randomly cropped to a fixed length of 750 frames. The OC-Softmax function in both the student and TTS teacher models is optimized using a stochastic gradient descent (SGD) optimizer, with parameters set to α = 20, m0 = 0.9, and m1 = 0.2.

[0103] The student model and both teacher models used the Adam optimizer, with parameters set to β1 = 0.9, β2 = 0.999, and ∈ = 10. -8 The weight decay was set to 0.0005. The learning rate was set to 0.0003, decaying by a factor of 0.5 every 10 epochs. The batch size was set to 64, and each model was trained for 100 epochs. In the VC knowledge distillation module, to balance the contributions of the two branches in knowledge transfer, the parameters λ and γ were both set to 0.5. In the overall loss function, this method first fixed the VC knowledge distillation loss weight parameter β to 1, and then adjusted the value of the TTS knowledge distillation loss weight parameter α based on the training effect. In the experiments, it was found that the overall training effect of the model was optimal when α was set to 1.

[0104] Model building phase

[0105] 1. Constructing a TTS teacher model (synthetic speech detection)

[0106] Emotion feature extraction module:

[0107] A two-way time-aware multi-scale emotion recognition model (TIM-Net) is adopted, with the following structure:

[0108] (1) Input speech preprocessing: Extract 39-dimensional me spectral coefficients (MFCC), with frame parameters of 20ms frame length and 10ms frame shift.

[0109] (2) Time reversal operation: The MFCC features are rearranged in reverse order by frame to generate forward and reverse feature sequences.

[0110] (3) Bidirectional branch processing: Forward and reverse features are respectively input into forward and reverse branches composed of multiple "time-aware blocks".

[0111] (4) Temporally Aware Block: Contains dilated causal convolutions (dilation rate increases layer by layer, such as 1, 2, 4, 8), batch normalization layers (BatchNorm1d), and ReLU activation functions, used to capture long-term contextual dependencies of speech.

[0112] (5) Feature fusion: Forward output of each time-aware block and reverse output After summation, g is generated through global average pooling (AdaptiveAvgPool1d(1)). i The final emotional characteristic is Where w i The weights are learnable, and n is the number of blocks (which can be set to 4 layers).

[0113] (6) Classification module: Connect to ResNet18 network, the residual block structure is the same as the student model, and the OC-Softmax loss function is connected after the fully connected layer with parameters set as (α=20, m0=0.9, m1=0.2).

[0114] 2. Construct a VC teacher model (converted speech detection)

[0115] Dual-branch feature extraction module:

[0116] (1) LFCC branch: Extract 60-dimensional linear frequency cepstral coefficients (LFCC) and input them into a ResNet18 network with convolutional attention module (CBAM) to capture transient anomalies and vocal tract features.

[0117] (2) CQT branch: Extract 100-dimensional constant Q transform (CQT) features (parameter f) min =3.5, B=48), input a ResNet18 network with CBAM to capture spectral distortion and fundamental frequency anomalies.

[0118] (3) Multi-task classification module: Each branch outputs true / false classification (2 classes) and fake type classification (≥2 classes), and the loss function is the branch weighted cross-entropy (with equal weights).

[0119] 3. Construct a student model (lightweight detection model)

[0120] (1) Feature extraction layer: Extract 60-dimensional LFCC features and fill / crop them to a fixed length of 750 frames.

[0121] (2) Classification network: 4-layer ResNet18 network, each residual block contains one-dimensional convolution (kernel size 3), batch normalization, ReLU, and the fully connected layer is followed by the OC-Softmax loss function with the same parameters as the TTS teacher model.

[0122] II. Teacher Model Training Phase

[0123] 1. TTS Teacher Model Training

[0124] Dataset: ASVspoof2019 LA training set (15200 synthesized speech samples and 2580 real speech samples).

[0125] Optimize configuration:

[0126] Optimizer: Adam, parameters β1 = 0.9, β2 = 0.999, ∈ = 10 -8 Weight decay of 0.0005.

[0127] Learning rate: Initially 0.0003, decays by 0.5 times every 10 rounds, batch size 64, training 100 rounds.

[0128] Objective: To minimize the OC-Softmax loss and separate the emotional feature distributions of synthesized speech from those of real speech (verified by t-SNE visualization).

[0129] 2. VC Teacher Model Training

[0130] Dataset: ASVspoof2019 LA training set (7600 converted speech samples, 2580 real speech samples) + ASVspoof2015 training set (converted speech samples).

[0131] Optimize configuration:

[0132] Optimizer: Adam, parameters as above.

[0133] Loss function: Joint optimization of bi-branch cross-entropy loss, with each branch weight being 0.5.

[0134] Objective: To improve the performance of converted speech detection by complementing LFCC and CQT features, with an EER target of ≤0.88%.

[0135] III. Knowledge Distillation and Student Model Training Stage

[0136] 1. Data routing mechanism

[0137] Real speech: Generate random numbers (R∈{0,1}, R=0) to input into the TTS teacher model, and R=1 to input into the VC teacher model.

[0138] Spoofing speech: Based on the type label (synthesis / conversion), the input is directed to the corresponding teacher model (synthesis → TTS teacher, conversion → VC teacher).

[0139] 2. Calculation of knowledge distillation loss

[0140] TTS distillation loss:

[0141] in, Embed the sentiment features output by the TTS teacher model. The features corresponding to the student model.

[0142] VC distillation loss:

[0143]

[0144] For the branch features of the VC teacher model, ) represents the features of the student model.

[0145] 3. Student Model Optimization

[0146] Optimizer: Adam, with the same parameters as the teacher model.

[0147] Training process:

[0148] After feature extraction, the input speech is routed to the corresponding teacher model via data routing. Distillation loss and classification loss are calculated, and student model parameters are updated via backpropagation. The detection performance of synthesized speech (EER target ≤ 0.38%) and converted speech (EER target ≤ 4.60%) is monitored in each round.

[0149] IV. Testing Phase

[0150] Deployment model: Use only the trained student model and input unknown types of speech.

[0151] Testing process:

[0152] Extract LFCC features and input them into the ResNet18 network.

[0153] The classification results are output by OC-Softmax to determine whether the speech is real or fake (synthesized / converted).

[0154] Performance metrics: On the ASVspoof2019 LA test set, the overall EER is ≤1.84%, and the EER is reduced by ≥16% in cross-library experiments (such as ASVspoof2015).

[0155] Example 3

[0156] A deep forgery speech detection method based on multi-teacher knowledge distillation includes the following steps:

[0157] Constructing a multimodal teacher-student model system:

[0158] Constructing a TTS teacher model: A bidirectional time-aware multi-scale emotion recognition model (TIM-Net) is used as the emotion feature extractor, followed by a ResNet18 classification network and a single-classification loss function OC-Softmax, focusing on synthesized speech detection;

[0159] Constructing the VC teacher model: A dual-branch network architecture is adopted, including an LFCC branch and a CQT branch. Each branch uses a ResNet18 network with a convolutional attention module (CBAM). Transformed speech detection and forgery type classification are achieved through multi-task learning.

[0160] To construct the student model, an LFCC feature extraction layer is connected to a ResNet18 classification network, and an OC-Softmax loss function is applied after the fully connected layer for lightweight fake speech detection.

[0161] Independent training process for teacher models:

[0162] TTS teacher model training: Using 15,200 synthesized speech (TTS generated) and 2,580 real speech in the ASVspoof 2019LA training set, the model was optimized based on the difference in emotional features. The compactness of the distribution of emotional features of synthesized speech was verified by t-SNE visualization.

[0163] VC Teacher Model Training: Using 7600 converted speech samples and 2580 real speech samples from the ASVspoof2019 LA training set, combined with converted speech samples from the ASVspoof2015 training set, the detection performance is improved by complementing the features of LFCC (capturing transient anomalies) and CQT (capturing spectral distortion).

[0164] Knowledge distillation and student model optimization:

[0165] Data routing mechanism: For real speech, a random number R between 0 and 1 is generated. When R = 0, it is input into the TTS teacher model, and when R = 1, it is input into the VC teacher model. For fake speech, it is input into the corresponding teacher model according to the type label (synthesis / conversion).

[0166] Loss function system:

[0167] 1. TTS distillation loss: in Embed the sentiment features output by the TTS teacher model. Features corresponding to the student model;

[0168] 2. VC distillation loss: Branch loss:

[0169] Student classification loss: OC-Softmax loss function The parameters are set as α = 20, m0 = 0.9, m1 = 0.2;

[0170] 3. Overall optimization goals: The Adam optimizer was used (β1=0.9, β2=0.999, ∈=10). -8 The weight decays by 0.0005, the learning rate is 0.0003, and the weight decays by 0.5 times every 10 rounds. The training is conducted for 100 rounds.

[0171] The TIM-Net architecture of the TTS teacher model includes:

[0172] Bidirectional feature processing: 39-dimensional MFCC features are extracted from the input speech, and time reversal is performed to generate a reversed feature sequence. The forward and reversed features are input into the forward and reverse branches, respectively.

[0173] Time-aware blocks: Each branch contains multiple time-aware blocks. Each block consists of dilated causal convolutions (with the dilation rate doubling layer by layer), batch normalization layers, and ReLU activation functions. The dilated convolutions are used to expand the temporal receptive field, and the causal convolutions ensure the isolation of non-causal information.

[0174] Multi-scale feature fusion: Fusion of outputs from various blocks through global temporal pooling. Final emotional characteristic F emo It is generated by weighted summation of the outputs of each block, with weight w. i Learning through backpropagation.

[0175] The dual-branch network implementation of the VC teacher model is as follows:

[0176] LFCC branch: Extract 60-dimensional LFCC features and input them into a 4-layer ResNet18 network. Each residual block contains a convolutional attention module (CBAM), and channel attention and spatial attention optimization are performed sequentially to detect vocal tract abnormalities in fake speech.

[0177] CQT branch: Extract 100-dimensional CQT features (parameters fmin=3.5, B=48), input them into a 4-layer ResNet18-CBAM network, and use variable time-frequency resolution to capture fundamental frequency and harmonic distortion;

[0178] Multi-task learning: Simultaneously output true / false classification results (2 classes) and fake type classification results (≥2 classes), with the loss function being branch-weighted cross-entropy and equal weights.

[0179] The lightweight design of the student model includes:

[0180] Feature extraction layer: Using 60-dimensional LFCC features, the computational efficiency is improved by 30% compared to MFCC by framing (frame length 20ms, frame shift 10ms) and padding / cropping to a fixed length of 750 frames.

[0181] Classification network: 4-layer ResNet18 network, each residual block consists of a one-dimensional convolutional layer (kernel size 3), a batch normalization layer, and a ReLU activation function, with the number of parameters reduced by 50% compared to the original ResNet18;

[0182] Generalization enhancement: OC-Softmax improves the detection capability against unknown forgery algorithms by compressing the distribution of real speech features (boundary margin (m_{0} = 0.9)) and expanding the distribution of forged speech features (boundary margin m1 = 0.2). Cross-database experiments show that on the ASVspoof2015, FoR-2seconds, and Wavfake datasets, the EER is reduced by 16.2%, 20.22%, and the ACC is improved by 29.66%.

[0183] A deepfake speech detection system based on multi-teacher knowledge distillation includes:

[0184] TTS Teacher Model Module:

[0185] TIM-Net sentiment feature extraction unit: implements MFCC forward and reverse processing and time-aware block computation;

[0186] ResNet18-OC classification unit: Outputs synthesized speech detection results based on emotion features.

[0187] VC Teacher Model Module:

[0188] LFCC-CQT dual-branch feature extraction unit: processes linear frequency and constant Q transform features in parallel;

[0189] Multi-task classification unit: A ResNet18 network integrating CBAM, outputting true / false and type classification results.

[0190] Student Model Module:

[0191] LFCC Fast Extraction Unit: Real-time feature calculation based on short-time Fourier transform;

[0192] Lightweight ResNet18 classification unit: supports end-to-end detection with single-class loss function.

[0193] Knowledge distillation execution module:

[0194] Data routing engine: Dynamically assigns speech to the corresponding teacher model based on speech tags (real / synthetic / converted);

[0195] Loss Fusion Calculator: Real-time calculation of MSE distillation loss and OC-Softmax classification loss to drive student model parameter updates.

[0196] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A deep forgery speech detection method based on multi-teacher knowledge distillation, characterized in that, The steps are as follows: S1. Constructing a TTS teacher model: A two-way time-aware multi-scale emotion recognition model is used as an emotion feature extractor, followed by a ResNet18 classification network and a single-classification loss function OC-Softmax, focusing on synthesized speech detection; S2. Constructing the VC teacher model: A dual-branch network architecture is adopted, including the LFCC branch and the CQT branch. Each branch uses a ResNet18 network with a convolutional attention module. Transformed speech detection and forgery type classification are achieved through multi-task learning. S3. Constructing a student model: An LFCC feature extraction layer is connected to a ResNet18 classification network, and an OC-Softmax loss function is applied after the fully connected layer for lightweight fake speech detection. S4. Perform the independent training process for the teacher model: TTS teacher model training: Using several synthesized speech samples and several real speech samples from the ASVspoof2019 LA training set, the model was optimized based on the difference in emotional features. The compactness of the distribution of emotional features in the synthesized speech was verified by t-SNE visualization. VC teacher model training: Using several converted speech samples and several real speech samples from the ASVspoof2019 LA training set, combined with converted speech samples from the ASVspoof2015 training set, detection performance is improved by capturing transient anomalies and capturing complementary spectral distortion features. S5. Optimize knowledge distillation and student model: Data routing mechanism: For real speech, a random number R between 0 and 1 is generated. When R = 0, it is input into the TTS teacher model, and when R = 1, it is input into the VC teacher model. For fake speech, it is input into the corresponding teacher model according to the type label. S6. Establish a loss function system: TTS distillation loss: in Embed the sentiment features output by the TTS teacher model. Features corresponding to the student model; VC distillation loss: Branch loss lose: Student classification loss: OC-Softmax loss function The parameters are set as α = 20, m0 = 0.9, m1 = 0.2; Overall optimization goals: The Adam optimizer (β1 = 0.9, β2 = 0.999, ∈ = 10) was used. -8 The weight decays by 0.0005, the learning rate is 0.0003, and the weight decays by 0.5 times every 10 rounds, with 100 training rounds.

2. The deep forgery speech detection method based on multi-teacher knowledge distillation according to claim 1, characterized in that, The TIM-Net architecture of the TTS teacher model includes: Bidirectional feature processing module: Extracts 39-dimensional MFCC features from the input speech, performs time reversal to generate reversed feature sequence, and inputs the forward and reversed features into the forward and reverse branches respectively; Time-aware blocks: Each branch contains multiple time-aware blocks, each block consisting of dilated causal convolutions, batch normalization layers, and ReLU activation functions. Dilated convolutions are used to expand the temporal receptive field, and causal convolutions ensure the isolation of non-causal information. Multi-scale feature fusion module: fuses the outputs of each block through global temporal pooling. Final emotional characteristic F emo It is generated by weighted summation of the outputs of each block, with weight w. i Learning through backpropagation.

3. The deep forgery speech detection method based on multi-teacher knowledge distillation according to claim 1, characterized in that, The dual-branch network implementation of the VC teacher model is as follows: LFCC branch: Extract 60-dimensional LFCC features and input them into a 4-layer ResNet18 network. Each residual block contains a convolutional attention module, and channel attention and spatial attention optimization are performed sequentially to detect vocal tract abnormalities in fake speech. CQT branch: Extract 100-dimensional CQT features with parameters fmin=3.5 and B=48, input to a 4-layer ResNet18-CBAM network, and use variable time-frequency resolution to capture fundamental frequency and harmonic distortion; Multi-task learning: Simultaneously outputs true / false classification results and fake type classification results, with the loss function being branch-weighted cross-entropy, and the weights being equal.

4. The deep forgery speech detection method based on multi-teacher knowledge distillation according to claim 1, characterized in that, The lightweight design of the student model includes: Feature extraction layer: 60-dimensional LFCC features are used, and the length is fixed to 750 frames through frame segmentation and padding / cropping; Classification network: 4-layer ResNet18 network, each residual block consists of a one-dimensional convolutional layer, a batch normalization layer, and a ReLU activation function; Generalization enhancement: OC-Softmax improves the detection capability against unknown forgery algorithms by compressing the distribution of real speech features and expanding the distribution of forged speech features.

5. A deepfake speech detection system based on multi-teacher knowledge distillation, characterized in that, include: TTS Teacher Model Module: TIM-Net sentiment feature extraction unit: implements MFCC forward and reverse processing and time-aware block computation; ResNet18-OC classification unit: outputs synthesized speech detection results based on emotion features; VC teacher model module: LFCC-CQT dual-branch feature extraction unit: processes linear frequency and constant Q transform features in parallel; Multi-task classification unit: ResNet18 network integrating CBAM, outputting true / false and type classification results; Student Model Module: LFCC Fast Extraction Unit: Real-time feature calculation based on short-time Fourier transform; Lightweight ResNet18 classification unit: supports end-to-end detection using a single-classification loss function; Knowledge distillation execution module: Data routing engine: dynamically assigns data to the corresponding teacher model based on voice tags; Loss Fusion Calculator: Real-time calculation of MSE distillation loss and OC-Softmax classification loss to drive student model parameter updates.

Images