Voice conversion authentic identification method and system based on knowledge distillation alignment

Abstract

The invention discloses a voice conversion authentic identification method and system based on knowledge distillation alignment, and is applied to the technical field of voice authentic identification. The method comprises the following steps that a double-branch model used for voice authentic identification is constructed, pure voice serves as input of a teacher branch, noisy voice serves as input of a student branch, and the teacher branch and the student branch share a feature extraction network of the same structure; applying a speech enhancement technology at the front end of a student branch to generate an enhanced speech signal; deep features are extracted, and alignment of the deep features in a hidden space is restrained through a knowledge distillation loss function; carrying out dynamic weight fusion on the aligned deep features through a fusion weight; and training a classifier, performing joint optimization in combination with classification loss and knowledge distillation loss, and outputting a voice authentic identification result. According to the method, distribution alignment of pure and noise features is realized through knowledge distillation, forged traces in voice conversion are effectively recognized, and high detection precision is still kept in complex noise and unknown attack scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of speech authentication, and more particularly to a speech conversion authentication method and system based on knowledge distillation and alignment. Background Art

[0002] In recent years, with the rapid development of deep learning, voice conversion (VC) technology has made significant progress, particularly in cross-speaker voice conversion using non-parallel corpora. Modern VC methods can achieve high-fidelity timbre transfer, generating speech that sounds highly similar to the target speaker and, in some scenarios, is difficult for humans or automated systems to distinguish. This high-quality forged speech poses a potential threat to identity verification systems, voice security, and the dissemination of information authenticity. Therefore, effective detection methods are urgently needed to identify these synthesized or converted speech. Currently, most research on VC forgery detection focuses on improving model performance in noise-free environments. For example, the ASVspoof series of challenges provides standardized, clean datasets, which has promoted the development of various feature representation methods and classification architectures, including discriminative subband analysis, end-to-end waveform modeling, and convolutional neural network (CNN)-based classifiers. However, these methods often exhibit significant performance degradation when faced with additive noise, which is common in real-world applications. Even the introduction of slight ambient noise can significantly reduce the detection accuracy of detectors trained solely on clean speech. To address this issue, some studies have attempted to use data augmentation, multi-condition training, or adversarial training strategies to enhance the robustness of the model. Although these methods have improved the model's adaptability to known noise types to a certain extent, they still have significant limitations when dealing with unknown types of noise, and adversarial training may cause instability in the training process, affecting model convergence. In addition, there are also works exploring the introduction of front-end speech enhancement modules into the detection process, attempting to improve acoustic conditions through noise reduction. However, due to the estimation error of noise characteristics during the speech enhancement process, this method is prone to introduce new signal distortion, which may interfere with subsequent detection tasks and even lead to performance degradation that is more serious than the original noise. Therefore, how to provide a speech conversion authentication method and system based on knowledge distillation alignment is a problem that those skilled in the art urgently need to solve. Summary of the Invention

[0003] In view of this, the present invention provides a speech conversion authentication method and system based on knowledge distillation alignment. The teacher model guides the student model to learn clean feature distribution, and combines dynamic feature fusion to suppress forgery traces, thereby significantly improving the robustness of detection and solving the problem of insufficient generalization ability of existing speech authentication technology in noisy environments and unknown attacks.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] A method for detecting fake speech conversion based on knowledge distillation and alignment, comprising the following steps:

[0006] S1. Build a dual-branch model for speech authentication, including a teacher branch and a student branch. The teacher branch takes clean speech as input, and the student branch takes noisy speech as input. The teacher and student branches share the same feature extraction network structure.

[0007] S2. Apply speech enhancement technology at the front end of the student branch to generate an enhanced speech signal by processing the amplitude spectrum of the noisy speech signal, thereby preliminarily removing or reducing the background noise in the input speech and improving the speech quality;

[0008] S3. Extract the deep features of the teacher branch and the student branch respectively, and use the knowledge distillation loss function to constrain the alignment of the deep features of the student branch and the deep features of the teacher branch in the latent space;

[0009] S4, adaptively generate fusion weights based on the channel attention mechanism, and dynamically fuse the aligned deep features through the fusion weights;

[0010] S5. Train the classifier based on the fused features, perform joint optimization based on the classification loss and knowledge distillation loss, and output the speech authentication result.

[0011] Optionally, S2 is:

[0012] The input noisy speech signal S(t) is converted to the frequency domain through short-time Fourier transform to obtain the amplitude spectrum S(f, t) and phase spectrum Φ(f, t). The amplitude spectrum is processed using the trained speech enhancement network SE(·) to generate the enhanced amplitude spectrum. Enhanced speech signal The enhanced amplitude spectrum is obtained by inverse short-time Fourier transform And the original phase spectrum Φ(f, t) is reconstructed to obtain:

[0013]

[0014] Among them, iSTFT represents the inverse short-time Fourier transform process. During the training process, the mean square error loss function is used to optimize the speech enhancement network, which is defined as:

[0015]

[0016] Where F and T represent the indices in the frequency and time dimensions respectively, and X(f, t) is the amplitude spectrum of the clean speech.

[0017] Optionally, the loss function in S3 is:

[0018] L SSD =(1-α)L SL +αL KD +L TL

[0019] Where, L SSD is the knowledge distillation loss function, α is the loss trade-off parameter, L SL Hard loss for the student branch, L KD is Soft loss, L TL Hard loss for the teacher branch;

[0020] Soft loss refers to the knowledge distillation loss, which is used to measure the loss of similarity of softened logits between the student branch and the teacher branch. KL divergence is used to calculate the difference between the soft labels of the student branch and the teacher branch:

[0021] L KD =τ 2 KLD(softmax(y′) s ), softmax(y′ t ))

[0022] Where τ is a hyperparameter, KLD represents KL divergence, and y′ s is the soft label of the student branch, y′ t Soft label for the teacher branch;

[0023] Hard loss refers to the loss between the model prediction and the true label:

[0024] L SL =A-softmax(y s , L)

[0025] L TL =A-softmax(y t , L)

[0026] Where Asoftmax is an angle-based softmax variant used to measure the difference between the last layer output of the teacher model and the student model and the corresponding true label, L is the true label, y s Logits, y, output from the last layer of the student branch t Logits are the output of the last layer of the teacher branch.

[0027] Optionally, S3 adopts an adversarial training strategy by adding a gradient reversal layer to the student branch, which encourages the student branch to learn a representation that is invariant to noise during the feature extraction stage.

[0028] Optionally, S4 is:

[0029] X fusion =conv1(X concat )

[0030] ω=σ(conv2(X concat ))

[0031]

[0032] M=σ(conv(X N ))

[0033]

[0034] Where, X concat is achieved by concatenating the denoising features X in the channel dimension E and noise characteristics X N Obtained, X fusion To fuse features, conv1 and conv2 represent different 3×3 convolutional layers, which are used to generate fusion features and fusion weights ω. σ represents the sigmoid activation function, which is used to generate weight values ​​between 0 and 1. and They represent the enhanced features and noise features after the initial interaction, respectively, where the fusion weight w is used to weight the features; in the spatial information fusion stage, the noise feature X is firstly obtained by convolution operation and sigmoid activation function. N Generate a spatial mask matrix M, and then use the mask matrix M to and Fusion, get the final interactive feature X Inter .

[0035] Optionally, the joint optimization objective function in S5 is:

[0036] L c =L SSD +L SE

[0037] Where, L c To jointly optimize the objective function, L SSD is the knowledge distillation loss function, L SE is the mean square error loss function.

[0038] A speech conversion authentication system based on knowledge distillation and alignment, which implements the above-mentioned speech conversion authentication method based on knowledge distillation and alignment, includes:

[0039] The speech data enhancement module performs enhancement processing on the input noisy speech signal;

[0040] The branch feature extraction module includes a student branch and a teacher branch, which are used to extract deep features of noisy speech and clean speech respectively;

[0041] Branch feature alignment module, used to align the deep features of the student branch with the deep features of the teacher branch in the latent space;

[0042] The interactive fusion module performs dynamic weight fusion on the aligned deep features to obtain interactive features;

[0043] The voice authentication and classification module performs classification based on interactive features to obtain voice authentication results.

[0044] As can be seen from the above technical solution, compared with the existing technology, the present invention provides a method and system for speech conversion authentication based on knowledge distillation alignment, which has the following beneficial effects:

[0045] 1. This paper designs a parallel data flow consisting of a clean teacher branch and a noisy student branch, and proposes an interactive fusion module and a response-based teacher-student paradigm. This can guide learning of noisy data from two perspectives: data distribution and decision making, thereby effectively reducing noise interference and ensuring consistency with the clean data distribution.

[0046] 2. This invention applies speech enhancement technology to the front-end of the noisy student branch for preliminary noise reduction. An interactive fusion module adaptively combines denoised features with original noise features to generate noise-resilient feature representations. Knowledge distillation maps the decision space of the student model to the decision space of the teacher model, making the behavior of noisy speech as close as possible to that of clean speech.

[0047] 3. The present invention adopts a joint training method to optimize the entire structure so that the teacher-student network reaches the global optimum. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0049] Figure 1 This is a flow chart of the speech conversion authentication method based on knowledge distillation alignment of the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] The embodiment of the present invention discloses a method for detecting false voice conversion based on knowledge distillation alignment. Figure 1 As shown, the following steps are included:

[0052] S1. Build a dual-branch model for speech authentication, including a teacher branch and a student branch. The teacher branch takes clean speech as input, and the student branch takes noisy speech as input. The teacher and student branches share the same feature extraction network structure.

[0053] S2. Apply speech enhancement technology at the front end of the student branch to generate an enhanced speech signal by processing the amplitude spectrum of the noisy speech signal, thereby preliminarily removing or reducing the background noise in the input speech and improving the speech quality;

[0054] S3. Extract the deep features of the teacher branch and the student branch respectively, and use the knowledge distillation loss function to constrain the alignment of the deep features of the student branch and the deep features of the teacher branch in the latent space;

[0055] S4, adaptively generate fusion weights based on the channel attention mechanism, and dynamically fuse the aligned deep features through the fusion weights;

[0056] S5. Train the classifier based on the fused features, perform joint optimization based on the classification loss and knowledge distillation loss, and output the speech authentication result.

[0057] In one embodiment of the present invention, both the teacher and student branches use SENet34 as the feature extraction network. This network structure was chosen due to its outstanding performance on the ASVspoof2019 LA evaluation set. ResNet-34 is a classic deep residual network that addresses the vanishing gradient problem in deep network training by introducing residual blocks. SE-ResNet builds on this by adding a channel-wise attention mechanism, enabling the network to adaptively recalibrate feature responses across channels, further improving feature extraction.

[0058] In one embodiment of the present invention, the features input to these networks can be mel-spectrograms or original waveform signals; mel-spectrograms are a method of converting sound signals into two-dimensional image representations, which can capture the key features of the human auditory system's perception of sound; while directly using waveform signals retains all the original information of the audio signal and is suitable for more complex scenarios.

[0059] Furthermore, S2 is specifically:

[0060] The input noisy speech signal S(t) is converted to the frequency domain through short-time Fourier transform to obtain the amplitude spectrum S(f, t) and phase spectrum Φ(f, t). The amplitude spectrum is processed using the trained speech enhancement network SE(·) to generate the enhanced amplitude spectrum. Enhanced speech signal The enhanced amplitude spectrum is obtained by inverse short-time Fourier transform And the original phase spectrum Φ(f, t) is reconstructed to obtain:

[0061]

[0062] Among them, iSTFT represents the inverse short-time Fourier transform process. During the training process, the mean square error loss function is used to optimize the speech enhancement network, which is defined as:

[0063]

[0064] Where F and T represent the indices in the frequency and time dimensions respectively, and X(f, t) is the amplitude spectrum of the clean speech.

[0065] Furthermore, the loss function in S3 is specifically:

[0066] L SSD =(1-α)L SL +αL KD +L TL

[0067] Where, L SSD is the knowledge distillation loss function, α is the loss trade-off parameter, L SL Hard loss for the student branch, L KD is Soft loss, L TL Hard loss for the teacher branch;

[0068] Soft loss refers to the knowledge distillation loss, which is used to measure the loss of similarity of softened logits between the student branch and the teacher branch. KL divergence is used to calculate the difference between the soft labels of the student branch and the teacher branch:

[0069] L KD =τ 2 KLD(softmax(y′) s ), softmax(y′ t ))

[0070] Where τ is a hyperparameter, KLD represents KL divergence, and y′ sis the soft label of the student branch, y′ t Soft label for the teacher branch

[0071] The knowledge distillation loss function is calculated between soft labels, which are obtained by dividing the logits output by a hyperparameter called temperature τ. Higher values ​​of τ lead to smoother probability distributions, thereby promoting more knowledge transfer from the teacher model to the student model, but this may also reduce the accuracy of the student model.

[0072] Hardloss refers to the loss between the model prediction and the true label:

[0073] L SL =A-softmax(y s , L)

[0074] L TL =A-softmax(y t , L)

[0075] Where Asoftmax is an angle-based variant of s0ftmax, which is used to measure the difference between the last layer output of the teacher model and the student model and the corresponding true label, L is the true label, y s Logits, y, output from the last layer of the student branch t Logits are the output of the last layer of the teacher branch.

[0076] In the embodiment of the present invention, the calculation formula of Asoftmax is as follows:

[0077]

[0078] Where x i is the feature vector of the i-th sample after it passes through the network to the penultimate layer, y i For the real category, define in m is a hyperparameter that controls the angle range, usually m=1, θ j，i It is the angle between the feature vector output by the penultimate layer of the model and the weight vector of the last layer.

[0079] Furthermore, S3 adopts an adversarial training strategy by adding a gradient reversal layer (GRL) to the student branch, which encourages the student branch to learn a representation that is invariant to noise during the feature extraction stage.

[0080] GRL maintains normal data flow during forward propagation, but reverses the direction of the gradient during backward propagation and multiplies it by a negative coefficient. This allows the model to learn to ignore interfering features caused by noise while learning to distinguish between real and fake speech. This helps the student model better adapt to various noise conditions and maintain high detection accuracy even in unknown noise environments.

[0081] Furthermore, S4 is specifically:

[0082] X fusion =conv1(X concat )

[0083] ω=σ(conv2(X concat ))

[0084]

[0085] M=σ(conv(X N ))

[0086]

[0087] Where, X concat is achieved by concatenating the denoising features X in the channel dimension E and noise characteristics X N Obtained, X fusion To fuse features, conv1 and conv2 represent different 3×3 convolutional layers, which are used to generate fusion features and fusion weights ω. σ represents the sigmoid activation function, which is used to generate weight values ​​between 0 and 1. and They represent the enhanced features and noise features after the initial interaction, respectively, where the fusion weight w is used to weight the features; in the spatial information fusion stage, the noise feature X is firstly obtained by convolution operation and sigmoid activation function. N Generate a spatial mask matrix M, and then use the mask matrix M to and Fusion, get the final interactive feature X Inter .

[0088] In the embodiment of the present invention, the dynamic weight fusion of S4 is completed by the interactive fusion module, which mainly includes two parts: channel interaction and spatial fusion. First, the denoising feature X E and noise characteristics X N Generate multi-channel frequency-time features X through convolution operations and pooling concat , then concatenate in the channel dimension, and then generate the fusion feature X through two different 3×3 convolution layers fusionand fusion weight ω; then the fusion weight ω is used to adaptively adjust the enhanced features and noise characteristics The ratio of; finally, the final interaction feature X is obtained by spatial information fusion Inter In this step, the spatial mask matrix M is generated based on the noise characteristics, and the importance of each point is determined through maximum pooling and average pooling operations; the spatial mask matrix M determines the importance of each frequency-time point and adaptively adjusts the fusion ratio according to the noise characteristics, thereby effectively reducing noise interference and balancing distortion problems.

[0089] Furthermore, the joint optimization objective function in S5 is:

[0090] L c =L SSD +L SE

[0091] Where, L c To jointly optimize the objective function, L SSD is the knowledge distillation loss function, L SE is the mean square error loss function.

[0092] In one embodiment of the present invention, if the fused features are determined by the classifier to be real speech, the result is directly output; if it is detected as forged speech, the model is jointly optimized by combining the classification loss and the knowledge distillation loss, and the modeling ability of the noise-invariant representation is enhanced through the gradient reversal layer, and finally a high-confidence authentication result is output.

[0093] In one embodiment of the present invention, the feature extraction network uses a 161-dimensional amplitude spectrum as input feature, and the time frame is set to 600. In order to extract the logarithmic amplitude spectrum features, the Blackman window length and jump length of the STFT are set to 1728 and 130 respectively. The classifier is trained based on the fusion feature, and the classification loss is combined with the knowledge distillation loss for joint optimization, and the loss trade-off parameter α is set to 0.05. This embodiment mainly uses ASVspoof 2019 LA and ASVspoof 2015 as clean datasets, and simulates complex noise environments by adding noise sources such as NOISEX-92 and 100 Nonspeech Sounds. The SNR range is set to 0dB to 20dB. Finally, tests under different signal-to-noise ratio conditions show that this embodiment reduces the average equal error rate (EER) by 32% compared with the baseline model, and the EER in cross-dataset tests (such as ASVspoof 2015) is also reduced by about 28%. These results demonstrate the effectiveness of the forgery detection method disclosed in this embodiment in improving the robustness and generalization ability of the synthetic speech detection system.

[0094] and Figure 1Corresponding to the method shown, an embodiment of the present invention further discloses a voice conversion authentication system based on knowledge distillation alignment, which executes the above-mentioned voice conversion authentication method based on knowledge distillation alignment, including:

[0095] The speech data enhancement module performs enhancement processing on the input noisy speech signal;

[0096] The branch feature extraction module includes a student branch and a teacher branch, which are used to extract deep features of noisy speech and clean speech respectively;

[0097] Branch feature alignment module, used to align the deep features of the student branch with the deep features of the teacher branch in the latent space;

[0098] The interactive fusion module performs dynamic weight fusion on the aligned deep features to obtain interactive features;

[0099] The voice authentication and classification module performs classification based on interactive features to obtain voice authentication results.

[0100] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0101] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech conversion authentication method based on knowledge distillation and alignment, characterized by: The following steps are involved: S1. Build a dual-branch model for speech authentication, including a teacher branch and a student branch. The teacher branch takes clean speech as input, and the student branch takes noisy speech as input. The teacher and student branches share the same feature extraction network structure. S2. Apply speech enhancement technology at the front end of the student branch to generate an enhanced speech signal by processing the amplitude spectrum of the noisy speech signal, thereby preliminarily removing or reducing the background noise in the input speech and improving the speech quality; S3. Extract the deep features of the teacher branch and the student branch respectively, and use the knowledge distillation loss function to constrain the alignment of the deep features of the student branch and the deep features of the teacher branch in the latent space; S4, adaptively generate fusion weights based on the channel attention mechanism, and dynamically fuse the aligned deep features through the fusion weights; S5. Train the classifier based on the fused features, perform joint optimization based on the classification loss and knowledge distillation loss, and output the speech authentication result.

2. The method for speech conversion authentication based on knowledge distillation alignment according to claim 1, characterized in that: S2 is specifically: The input noisy speech signal S(t) is converted to the frequency domain through short-time Fourier transform to obtain the amplitude spectrum S(f, t) and phase spectrum Φ(f, t). The amplitude spectrum is processed using the trained speech enhancement network SE(·) to generate the enhanced amplitude spectrum. Enhanced speech signal The enhanced amplitude spectrum is obtained by inverse short-time Fourier transform And the original phase spectrum Φ(f,t) is reconstructed to obtain: Among them, iSTFT represents the inverse short-time Fourier transform process. During the training process, the mean square error loss function is used to optimize the speech enhancement network, which is defined as: Where F and T represent the indices in the frequency and time dimensions respectively, and X(f,t) is the amplitude spectrum of the clean speech.

3. The method for speech conversion authentication based on knowledge distillation alignment according to claim 1, characterized in that: The loss function in S3 is specifically: L SSD =(1-α)L SL +αL KD +L TL Where, L SSD is the knowledge distillation loss function, α is the loss trade-off parameter, L SL Hard loss for the student branch, L KD is Softloss, L TL Branch Hardloss for teachers; Softloss refers to the knowledge distillation loss, which is used to measure the loss of similarity of softened logits between the student branch and the teacher branch. KL divergence is used to calculate the difference between the soft labels of the student branch and the teacher branch: L KD =τ 2 ·KLD(softmax(y' s ),softmax(y' t )) Where τ is a hyperparameter, KLD represents KL divergence, and y' s Soft label for the student branch, y' t Soft label for the teacher branch; Hard loss refers to the loss between the model prediction and the true label: L SL =Asoftmax(y s ,L) L TL =Asoftmax(y t ,L) Where Asoftmax is an angle-based softmax variant used to measure the difference between the last layer output of the teacher model and the student model and the corresponding true label, L is the true label, y s Logits, y, output from the last layer of the student branch t Logits are the output of the last layer of the teacher branch.

4. The method for speech conversion authentication based on knowledge distillation alignment according to claim 1, characterized in that: S3 adopts an adversarial training strategy. By adding a gradient reversal layer to the student branch, it encourages the student branch to learn a representation that is invariant to noise during the feature extraction stage.

5. The method for speech conversion authentication based on knowledge distillation alignment according to claim 1, characterized in that: S4 is specifically: X fusion =conv1(X concat ) ω=σ(conv2(X concat )) M=σ(conv(X N )) Where, X concat is achieved by concatenating the denoising features X in the channel dimension E and noise characteristics X N Obtained, X fusion To fuse features, conv1 and conv2 represent different 3×3 convolutional layers, which are used to generate fusion features and fusion weights ω. σ represents the sigmoid activation function, which is used to generate weight values ​​between 0 and 1. and Represent the enhanced features and noise features after preliminary interaction, where the fusion The weight w is used to weight the features; in the spatial information fusion stage, the noise feature X is firstly obtained by convolution operation and sigmoid activation function. N Generate a spatial mask matrix M, and then use the mask matrix M to and Fusion, get the final interaction feature X Inter .

6. The method for speech conversion authentication based on knowledge distillation alignment according to claim 1, characterized in that: The joint optimization objective function in S5 is: L c =L SSD +L SE Where, L c To jointly optimize the objective function, L SSD is the knowledge distillation loss function, L SE is the mean square error loss function.

7. A speech conversion authentication system based on knowledge distillation and alignment, characterized by: Executing the method for speech conversion authentication based on knowledge distillation alignment according to any one of claims 1 to 6, comprising: The speech data enhancement module performs enhancement processing on the input noisy speech signal; The branch feature extraction module includes a student branch and a teacher branch, which are used to extract deep features of noisy speech and clean speech respectively; Branch feature alignment module, used to align the deep features of the student branch with the deep features of the teacher branch in the latent space; The interactive fusion module performs dynamic weight fusion on the aligned deep features to obtain interactive features; The voice authentication and classification module performs classification based on interactive features to obtain voice authentication results.

Images