A robust speaker recognition method based on spectrogram denoising and adversarial learning

Abstract

The present invention provides a robust speaker recognition method based on spectrogram denoising and adversarial learning. First, a spectrogram dataset of clean speech and a noisy spectrogram dataset after the clean speech is noisy are collected; a U-net with a multi-level encoding and decoding structure is trained using a mean square error loss function to remove noise interference from the mel-spectrogram of the noisy speech signal to obtain an enhanced mel-spectrogram; a conditional generative adversarial network based on a time-delay neural network (TDNN-CGAN) is trained using a least squares loss function, a time-delay neural network (TDNN) is used as a generator in the TDNN-CGAN to extract deep features of the enhanced mel-spectrogram, and a multi-layer perceptron (MLP) is used as a discriminator in the TDNN-CGAN; finally, a speaker classifier is trained using cross-entropy loss to identify the speaker's identity, thereby realizing speaker recognition in a noisy environment. The deep features extracted from the noisy speech by the present invention are close to the deep features extracted from the clean speech, thereby improving the performance of the speaker recognition system in a noisy environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech processing and relates to a robust speaker recognition method based on spectrogram denoising and adversarial learning. Background Art

[0002] In real-world environments, the speech input to speaker recognition systems is often contaminated by various background noises and reverberation. The added noise on clean speech blurs acoustic details, degrading speech intelligibility and quality, and thus degrading speaker recognition system performance. Common approaches to improving the robustness of speaker recognition systems include training the system with datasets consisting of both clean and noisy data, or adding a speech enhancement front-end. Speech enhancement involves extracting useful speech signals from a noisy background after the speech signal has been corrupted by noise. However, the speech enhancement process can produce speech distortion and even degrade speaker recognition system performance. Due to the powerful feature extraction capabilities of neural networks, they can be used to directly extract noise-free frequency domain features from the frequency domain features of noisy speech. Furthermore, generative adversarial networks (GANs) are currently being widely researched and applied to many speech and audio-related tasks, primarily focusing on domain conversion and generating more realistic data distributions. GANs also hold promise for extracting noise-resistant features. Summary of the Invention

[0003] The object of the present invention is to provide a robust speaker recognition method based on spectrogram denoising and adversarial learning to solve the problems raised in the above background technology.

[0004] The method first collects a dataset of mel-spectrograms of clean speech and a dataset of noisy mel-spectrograms of clean speech after adding noise; uses the mean square error loss function to train a U-net with a multi-level encoding and decoding structure to extract enhanced mel-spectrograms from the noisy mel-spectrograms; uses the least squares loss function to train a conditional generative adversarial network based on a time-delay neural network (TDNN-CGAN), adopts a multi-layer perceptron (MLP) as the discriminator, and a time-delay neural network (TDNN) as the generator to extract deep features of the enhanced mel-spectrograms; uses the cross-entropy loss to train an MLP-based speaker classifier to identify the speaker's identity, thereby realizing speaker recognition in a noisy environment.

[0005] The specific steps of the above speaker recognition method are as follows:

[0006] Step 1: Clean speech c Add noise n to get noisy speech s n =s c +n, using the Hamming window to transform the clean speech s c , noisy speech nDivide into short frames, extract Mel feature vectors from each frame, and form two feature matrices respectively: where x c (t), x n (t) represents the Mel feature vector of the tth frame of the clean and noisy speech Mel spectrograms, respectively, T represents the number of speech frames, t∈{1,...,T}, the superscript T represents the transpose, and D represents the dimension of the feature vector.

[0007] Step 2: Transform the noisy Mel spectrogram X of the noisy speech n Input the U-Net with multi-level encoding and decoding structure, and get X n * =Enhance(X n ), where Enhance(·) represents the number of mel spectrograms from the noisy n Extract enhanced mel-spectrogramX n * The mean square error loss function is used as the spectrum denoising loss to train U-Net. The expression of the spectrum denoising loss is as follows:

[0008]

[0009] Step 3: Clean Mel spectrogram X c , Enhanced Mel Spectrogram X n * The clean Mel spectrogram and enhanced Mel spectrogram are extracted through the generator-time delay neural network (TDNN) in TDNN-CGAN. c =G(X c ), E n =G(X n * The extracted deep features are input into the discriminator - multi-layer perceptron (MLP), G(·) and D(·) represent the outputs of the generator and discriminator respectively. The discriminator is trained according to the discriminant loss in the least squares generative adversarial network (LSGAN). The expression of the discriminant loss is as follows:

[0010]

[0011] Fixing the network parameters of the identification network will enhance the deep features E of the Mel spectrogram n Input the discriminator and calculate the generation loss in adversarial learning, which is used to train the generator TDNN so that the deep features extracted from noisy speech are closer to the deep features of clean speech. The expression of generation loss is as follows:

[0012]

[0013] Step 4: Extract the deep features E from the enhanced Mel spectrogram n The speaker classifier is input and trained simultaneously through cross entropy loss to achieve speaker recognition in a noisy environment, that is, robust speaker recognition.

[0014] The beneficial effects of the present invention are:

[0015] Through spectrogram denoising and adversarial learning, the present invention adopts a joint training scheme of U-Net Mel-spectrogram enhancement network, conditional generative adversarial network TDNN-CGAN based on time-delay neural network, and speaker classifier, so that the deep features extracted from noisy speech are close to the deep features extracted from clean speech, thereby improving the performance of speaker recognition system in noisy environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This paper proposes a robust speaker recognition method based on spectrogram denoising and adversarial learning. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. The specific embodiments described herein are only used to explain the technical solution of the present invention and are not intended to limit the present invention.

[0018] like Figure 1 As shown, the present invention proposes a robust speaker recognition method based on spectrogram denoising and adversarial learning. First, the clean speech signal and the noise signal are mixed to obtain a noisy speech signal, and the frequency domain features (such as Mel-spectrogram) of the clean speech signal and the noisy speech signal are extracted. Secondly, the Mel-spectrogram extracted from the noisy speech is input into the spectrogram enhancement network based on U-Net, and the noise interference of the Mel-spectrogram of the noisy speech is removed to obtain an enhanced spectrogram. The Mel-spectrogram of the clean speech and the enhanced Mel-spectrogram of the noisy speech are respectively input into the conditional generative adversarial network TDNN-CGAN based on the time delay neural network, and the depth features of the clean speech signal and the depth features of the noisy speech signal are obtained through generator encoding. Then, adversarial learning is performed through the discriminator, so that the enhanced deep features extracted from the noisy speech are closer to the deep features of the clean speech; finally, the enhanced deep features are input into the speaker classifier, thereby realizing speaker recognition in a noisy environment.

[0019] The present invention will be further described through the following implementation steps.

[0020] Step 1: First, perform voice activity detection (VAD) on a speaker's speech to remove silent segments, and intercept a 3-second speech as the clean speech. Randomly select a 3-second noise signal from the noise database and perform linear addition to obtain a noisy speech copy of the clean speech. Then, pre-emphasize the clean speech and its noisy speech copy, add Hamming window frames, and extract Mel features to obtain the clean speech Mel spectrogram. Mel-spectrogram of noisy speech where x c (t), x n (t) represents the Mel feature vector of the tth frame of the clean and noisy speech Mel spectrograms, respectively, T represents the number of speech frames, t∈{1,...,T}, the superscript T represents the transpose, and D represents the dimension of the feature vector.

[0021] Step 2: Transform the noisy Mel spectrogram X of the noisy speech signal n As the input of the U-net based spectral denoising network, the network has a multi-level encoding-decoding structure. In the encoding stage, the input feature map first passes through 5 layers of convolutional layers for feature compression to obtain a hidden vector c. In the decoding stage, the hidden vector c passes through 5 layers of deconvolution layers for feature reconstruction to obtain the enhanced feature X. n * =Enhance(X n ), where Enhance(·) represents the spectrogram denoising process based on U-net, from the Mel spectrogram X of noisy speech n Extract enhanced mel-spectrogramX n * .

[0022] The five convolutional layers in the encoding phase all use 2D convolution, with input channels of 1, 16, 32, 32, and 64, respectively, and output channels of 16, 32, 32, 64, and 64, respectively. The convolution kernel size of each convolutional layer is 4×1. The five convolutional layers in the decoding phase all use 2D deconvolution, with input channels of 64, 128, 64, 64, and 32, respectively, and output channels of 64, 32, 32, 16, and 1, respectively. The convolution kernel size of each deconvolutional layer is 4×1. PReLu activation functions are added after all convolutional layers. Each encoding layer is connected to its corresponding decoding layer, bypassing the feature compression process performed in the middle of the model and directly passing the fine-grained information of the feature map to the decoding stage.

[0023] Step 3: Clean Mel spectrogram X c , Enhanced Mel Spectrogram X n * The conditional generative adversarial network TDNN-CGAN based on the time delay neural network is input respectively to obtain the deep features E of the clean Mel spectrogram. c, Enhanced Mel-spectrogram deep features E n In the encoding network, the input feature map will pass through 4 layers of 1D convolutional layers in sequence. After the 4 convolutional layers, batch normalization operation is performed and a Dropout layer is added. The parameter value p of the Dropout layer is set to 0.1. The output of each convolutional layer is represented as Where l = 1, 2, 3, 4, T′ represents the number of frames, and D′ represents the dimension of the depth feature vector of each frame.

[0024] Considering that in the neural network, the deep features extracted by each layer of the network contain relevant information of the original input, the output of each convolutional layer is linearly added to achieve feature aggregation and obtain the aggregated features. Then, the aggregated features are statistically pooled and the mean and standard deviation of the aggregated features are concatenated to obtain the discourse-level features. Finally, the fully connected layer is used to transform E Statistics Converted to a fixed 256-dimensional vector as the deep feature extracted by the encoder.

[0025] Step 4: Through adversarial learning, first train the discriminator to correctly distinguish E c and E n Then fix the discriminator and train the generator TDNN. The discriminator consists of three fully connected layers, and the last layer has two output nodes. The objective function of training the discriminator is expressed as:

[0026]

[0027] Step 5: Input the deep features extracted from the enhanced mel-spectrogram into a speaker classifier consisting of a fully connected layer and a softmax layer, and train the speaker classifier using cross entropy loss.

[0028] Step 6:

[0029] ① Follow steps 1 and 2 to input the noisy Mel-spectrogram of the noisy speech into the U-net-based spectrogram denoising network to obtain the enhanced Mel-spectrogram. Calculate the mean square error (MSE) loss with the Mel-spectrogram of the clean speech to train the U-net-based spectrogram denoising network.

[0030] ② Follow steps 3 and 4. First, train the discriminator in TDNN-CGAN through the discriminant loss. Then, fix the network parameters of the discriminator, calculate the generation loss, and use the backpropagation algorithm to train the spectral graph enhancement network and the generator in TDNN-CGAN.

[0031] ③Execute step 5, combine the real speaker label corresponding to the speech and the predicted label of the speaker classifier, calculate the cross entropy loss, and use the backpropagation algorithm to train the spectrogram enhancement network, generator and speaker classifier.

[0032] ④ Alternate training until the network loss value converges, stop training, and save the U-Net spectrogram enhancement network model, the generator network model in the conditional generative adversarial network TDNN-CGAN based on time-delay neural network, and the network model of the speaker classifier.

[0033] Step 7: Using the U-Net spectrogram enhancement network model saved in step 6, the generator network model in the conditional generative adversarial network TDNN-CGAN based on time-delay neural network, and the network model of the speaker classifier, follow steps 2, 3, and 5 to identify the speaker identity of the noisy speech and achieve speaker recognition in a noisy environment.

[0034] To verify the performance of the speaker recognition method, 340 speakers were selected from the Aishell-1 dataset. 40 sentences were selected from each speaker, and each sentence was cut into 3-second segments. 20 of these sentences were used as the clean speech for the training set. The clean speech was then randomly mixed with noise data from the Musan noise dataset at a signal-to-noise ratio of 0, 5, 10, 15, or 20 to obtain noisy transcripts. The clean speech and noisy transcripts constituted the training set. The remaining 20 sentences were mixed with three different types of noise from the Musan noise dataset at five signal-to-noise ratios of 0, 5, 10, 15, and 20 to obtain 16 test sets for calculating the speaker recognition accuracy. A higher speaker recognition accuracy indicates better recognition performance of the speaker recognition system.

[0035] In addition, we constructed a speaker recognition system consisting of a TDNN network and a speaker classifier, trained using cross-entropy loss. We also constructed a speaker recognition system consisting of a conditional generative adversarial network (TDNN-CGAN) based on a time-delay neural network and a speaker classifier, trained using adversarial learning and cross-entropy loss. These systems were used to evaluate the effectiveness of robust speaker recognition systems based on feature enhancement and adversarial learning. The experimental results are shown in Table 1.

[0036] As shown in Table 1, the robust speaker recognition method based on spectrogram denoising and adversarial learning (U-Net spectrogram augmentation network + TDNN-CGAN + speaker classifier) ​​achieves a speaker recognition accuracy of 99.68% without adding noise, and 89.72%, 92.50%, and 91.47% under three different noise types and a signal-to-noise ratio of 0dB, respectively. The TDNN + speaker classifier achieves a speaker recognition accuracy of 98.24% without adding noise, and 79.63%, 83.82%, and 86.29% under three different noise types and a signal-to-noise ratio of 0dB, respectively. The TDNN-CGAN + speaker classifier achieves a speaker recognition accuracy of 99.12% without adding noise, and 86.93%, 90.11%, and 87.64% under three different noise types and a signal-to-noise ratio of 0dB, respectively.

[0037] Under the same training data conditions, the robust speaker recognition method based on spectrogram denoising and adversarial learning (U-Net spectrogram augmentation network + TDNN-CGAN + speaker classifier) ​​achieved higher speaker recognition accuracy in both noise-free and noisy environments, indicating the best robust speaker recognition performance. Therefore, the robust speaker recognition system based on spectrogram denoising and adversarial learning of the present invention is effective.

[0038] Table 1 Speaker recognition accuracy

[0039]

[0040] The above description merely represents the preferred embodiments of the present invention, and while the description is relatively specific and detailed, it should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make various modifications, improvements, and substitutions without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A robust speaker recognition method based on spectrogram denoising and adversarial learning, characterized by: A multi-level encoding / decoding U-Net network is used to remove noise interference from the mel-spectrogram of noisy speech signals, generating an enhanced mel-spectrogram. A time-delay neural network-based conditional generative adversarial network (TDNN-CGAN) is used to extract deep features from the enhanced mel-spectrogram, which are then input into a speaker classifier to identify the speaker. The specific steps of the speaker recognition method are as follows: ① Add noise to the clean speech to obtain the noisy speech, and then perform frame division, windowing, and Mel spectrogram extraction on the clean speech and the noisy speech to obtain the Mel spectrogram of the clean speech. Mel-spectrogram of noisy speech where x c (t), x n (t) represents the Mel feature vector of the tth frame of the clean and noisy speech Mel spectrograms, respectively. T represents the number of speech frames, t∈{1,...,T}, the superscript T represents the transpose, and D represents the dimension of the Mel feature vector of each frame. ② X n Input U-Net spectrogram enhancement network to obtain enhanced Mel spectrogram X n * , the mean square error loss function is used as the spectral enhancement loss to train the U-Net spectral enhancement network. The expression of the spectral enhancement loss is as follows: ③ X c 、X n * Input TDNN-CGAN respectively, and use the generator in TDNN-CGAN to extract X respectively c The deep feature E c =G(X c ), X n * The deep feature E n =G(X n * );E c 、E n Input the discriminator in TDNN-CGAN respectively, and train the discriminator according to the discriminant loss in the least squares generative adversarial network. The expression of the discriminant loss is as follows: G(·) and D(·) represent the outputs of the generator and discriminator respectively; the network parameters of the discriminator are fixed, and E n Input the discriminator and train the generator according to the generation loss in the least squares generative adversarial network. The expression of the generation loss is as follows: ④E c 、E n Input the speaker classifier and train the speaker classifier through cross entropy loss to identify the speaker.

Images