End-to-end speaker recognition method based on multi-scale SincNet and CGAN

Abstract

The application belongs to the technical field of speaker recognition, and specifically discloses an end-to-end speaker recognition method based on multi-scale SincNet and CGAN. The multi-scale SincNet is introduced, so that important information is avoided to be lost in manual feature conversion, the multi-scale SincNet captures low-level speech representation of three channels in a waveform according to three customized filter banks, the SincGAN model better captures important narrowband speaker features, the end-to-end recognition is realized based on the improved conditional generative adversarial network, the speaker is recognized by using a small amount of training sentences, and the loss function of the application comprises an adversarial loss of a classic GAN and a classification cross-entropy loss of a classification task. Experimental results show that on TIMIT and LIBRISPEECH corpus, the model of the application shows better performance, and when lacking training data, the model of the application shows stronger robustness than a baseline method.

Description

Technical Field

[0001] This invention belongs to the field of speaker recognition technology, and specifically relates to an end-to-end speaker recognition method based on multi-scale SincNet and CGAN. Background Technology

[0002] Speaker identification (SI) is a biometric technology that identifies the speaker from a pool of known speakers based on a given speech segment, representing a one-to-many selection relationship. This task presents a significant challenge for humans. Speaker identification is widely used in numerous fields, primarily due to the advantages of voiceprint recognition, such as ease of acquisition, non-contact processing, and stable features. With the development of deep learning, deep neural networks have demonstrated outstanding performance in feature extraction and model classification, pointing to new directions for the further development of speaker identification technology. Recently, several proposed recognition models have achieved considerably high accuracy, but still face challenges in practical industrial applications. For example, problems encountered in practical applications include short speech recognition, dialect recognition, and unordered speech recognition. Short speech segments, due to the inability to extract sufficient discriminative information, lead to reduced robustness of SI systems. Furthermore, under the constraint of limited training data, overfitting occurs because more effective speaker feature parameters cannot be extracted. In particular, data-driven modeling methods based on deep neural networks require massive amounts of training data. However, due to limitations in the real-world environment, it is difficult to obtain a large amount of user speech data, making it impossible to extract sufficient information representing speaker features.

[0003] Traditional speaker recognition is complex and has a low recognition rate. The process includes speech signal preprocessing, acoustic feature extraction, classification model construction, and learning model evaluation. Generally, establishing and applying a speaker recognition system requires two stages: training and testing. However, both training and testing require preprocessing and feature extraction of the original input signal. In feature extraction, most attempts rely on manually designed features, such as Mel frequency cepstral coefficients (MFCC) and Mel filter bank coefficients (FBank). Reynolds et al. used extracted acoustic features to train a Gaussian Mixture Model-Universal Background Model (GMM-UBM) to alleviate data sparsity. Furthermore, to address the performance degradation caused by channel interference, Campbell et al. added Support Vector Machines (SVM) to GMM-UBM in 2006, effectively improving model recognition performance. The following year, Kenny et al. conducted in-depth research on joint factor analysis, extracting only speaker-related features to overcome the influence of channel variability. In 2010, Dehak et al. proposed mapping speech to a fixed, low-dimensional vector, i.e., using an I-vector to represent a given utterance. This method improves the robustness and generalization ability of the SI system.

[0004] With the development of deep neural networks (DNNs), researchers have begun to favor using DNN-based methods to replace traditional approaches. Data-driven modeling methods based on DNNs rely on large-scale training data, but in reality, environmental limitations often prevent the acquisition of large amounts of user speech data. Furthermore, the handcrafted features often used in deep learning may lose important information during the conversion process, leading to a decline in recognition performance. Summary of the Invention

[0005] The purpose of this invention is to propose an end-to-end speaker recognition method based on multi-scale SincNet and CGAN. By introducing multi-scale SincNet to directly recognize the original input waveform, important information is avoided during manual feature transformation. At the same time, conditional generative adversarial networks are used for end-to-end recognition, enabling speaker recognition with a small number of training sentences.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An end-to-end speaker identification method based on multi-scale SincNet and CGAN includes the following steps:

[0008] Step 1. Perform speech frame preprocessing on the input raw speech signal to obtain speech frames, and use the speech frames as real speech samples; divide the real speech samples into training samples and test samples, which are used for model training and testing respectively.

[0009] Step 2. Build the speaker recognition model SincGAN;

[0010] The speaker recognition model SincGAN consists of a generator network and a discriminator network;

[0011] The generator network consists of a multi-scale SincNet layer, three convolutional layers, two transposed convolutional layers, and an adaptive average pooling layer.

[0012] Define the multi-scale SincNet layer in the generator network as the first multi-scale SincNet layer; define the three convolutional layers in the generator network as the first, second, and third convolutional layers, and the two transposed convolutional layers as the first and second transposed convolutional layers, respectively;

[0013] The processing flow of real speech samples in the generator network is as follows:

[0014] The real speech sample first passes through the first multi-scale SincNet layer for feature extraction to obtain the two-dimensional features of the speech signal. Then, the two-dimensional features of the speech signal pass through the first convolutional layer, the second convolutional layer, the first transposed convolutional layer, the second transposed convolutional layer, the third convolutional layer, and the adaptive average pooling layer in sequence to generate fake speech samples.

[0015] The discriminator network consists of a multi-scale SincNet layer, five convolutional layers, three bottleneck residual block stacked layers, and four fully connected layers;

[0016] Define the multi-scale SincNet layer in the discriminator network as the second multi-scale SincNet layer;

[0017] The five convolutional layers in the discriminator network are defined as the fourth, fifth, sixth, seventh, and eighth convolutional layers;

[0018] The discriminator network is defined with three bottleneck residual block stacked layers as the first, second, and third bottleneck residual block stacked layers, and four fully connected layers as the first, second, third, and fourth fully connected layers.

[0019] The processing flow of real and fake speech samples in the discriminator network is as follows:

[0020] Real speech samples and fake speech samples are first processed through the second multi-scale SincNet layer to extract features, resulting in two-dimensional features of the speech signal. Then, the two-dimensional features of the speech signal are sequentially processed through the fourth convolutional layer, the first bottleneck residual block stacking layer, the fifth convolutional layer, the second bottleneck residual block stacking layer, the sixth convolutional layer, the second bottleneck residual block stacking layer, the seventh convolutional layer, the eighth convolutional layer, the first fully connected layer, and the second fully connected layer.

[0021] The output of the second fully connected layer is divided into two paths: one path goes through the third fully connected layer to output true / false flags, and the other path goes through the fourth fully connected layer to output an N-dimensional vector, which corresponds to the speaker category label of the real speech sample.

[0022] The N-dimensional vector output by the discriminator network is input into the Softmax function. By mapping the output vector onto a probability distribution, the speaker category label of the most probable predicted category is used as the prediction output.

[0023] Step 3. Use the real speech samples from Step 1 to train the speaker recognition model SincGAN built in Step 2. Optimize the parameters of the speaker recognition model through backpropagation to minimize the loss function and obtain the trained speaker recognition model SincGAN.

[0024] Step 4. Use the trained SincGAN to predict the given speech signal and output the corresponding speaker label.

[0025] Furthermore, based on the aforementioned end-to-end speaker recognition based on multi-scale SincNet and CGAN, this invention also proposes a computer device comprising a memory and one or more processors.

[0026] The memory stores executable code, and when the processor executes the executable code, it implements the end-to-end speaker recognition steps based on multi-scale SincNet and CGAN mentioned above.

[0027] Furthermore, based on the aforementioned end-to-end speaker recognition based on multi-scale SincNet and CGAN, this invention also proposes a computer-readable storage medium storing a program thereon. When executed by a processor, this program is used to implement the steps of the aforementioned end-to-end speaker recognition based on multi-scale SincNet and CGAN.

[0028] The present invention has the following advantages:

[0029] As described above, this invention discloses an end-to-end speaker recognition method based on multi-scale SincNet and CGAN. The method introduces multi-scale SincNet to avoid losing important information during manual feature transformation. Multi-scale SincNet captures low-level speech representations of three channels in the waveform using three custom filter banks, enabling the SincGAN model to better capture important narrowband speaker features. This invention's method performs end-to-end recognition based on an improved conditional generative adversarial network, achieving speaker recognition with a small number of training sentences. The loss function of this invention includes the adversarial loss of classic GANs and the classification cross-entropy loss for classification tasks. Experimental results show that the model of this invention exhibits better performance on the TIMIT and LIBRISPEECH corpora. In the absence of training data, the model of this invention demonstrates stronger robustness than baseline methods. Attached Figure Description

[0030] Figure 1 This is a flowchart of an end-to-end speaker recognition method based on multi-scale SincNet and CGAN in an embodiment of the present invention.

[0031] Figure 2 This is an architecture diagram of the speaker recognition model SincGAN built in an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of the generator network structure in an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of the discriminator network in an embodiment of the present invention;

[0034] Figure 5 This is a schematic diagram of the bottleneck-type residual block stacking layer in an embodiment of the present invention;

[0035] Figure 6 This is a schematic diagram of the improved conditional generative adversarial network in an embodiment of the present invention;

[0036] Figure 7 This is an architecture diagram of the multi-scale SincNet layer in an embodiment of the present invention.

[0037] Figure 8 This is a schematic diagram showing the distribution of speech frame lengths in the LIBRISPEECH(a) and TIMIT(b) corpora.

[0038] Figure 9 This is a graph showing the changing trends of the speaker recognition models SincGAN and CGAN-MFCC in this embodiment of the invention on the LIBRISPEECH corpus, specifically the recognition accuracy and test loss values. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0040] Example 1

[0041] like Figure 1 As shown in the figure, this embodiment describes an end-to-end speaker recognition method based on multi-scale SincNet and CGAN. This end-to-end speaker recognition method based on multi-scale SincNet and CGAN includes the following steps:

[0042] Step 1. Perform speech framing preprocessing on the input raw speech signal to obtain speech frames, which are then used as real speech samples. These real speech samples are divided into training samples and test samples, used for model training and model testing, respectively.

[0043] The raw audio signal can be acquired through a microphone or by uploading a local audio file. The acquired audio signal is preprocessed, including audio framing, with a default frame length of 200ms and a frame shift of 10ms.

[0044] Suppose there are N speakers, and each speaker has m sentences. The set of m sentences for the k-th speaker is:

[0045] S k ={h1,h2,...,h m}, k = 1, 2, ..., N.

[0046] Where h1, h2, ..., h m This represents m different statements.

[0047] Since the speaker recognition model SincGAN in step 2 uses speech frames as input data, the utterance input to the model is divided into n fixed-length speech frames {x1, x2, ..., xn}. n}

[0048] Each segmented speech frame is then sequentially fed into the speaker recognition model SincGAN in step 2 below.

[0049] Step 2. Build the speaker recognition model SincGAN.

[0050] SincGAN is a deep learning model based on multi-scale SincNet and conditional generative adversarial networks. It uses multi-scale SincNet to extract features from preprocessed speech signals to obtain two-dimensional feature representations.

[0051] These features serve as input to the discriminator in the improved Conditional Generative Adversarial Network (CGAN) and output a speech signal and a corresponding speaker identifier vector. The output vector is then mapped onto a probability distribution using the Softmax function, where each possible speaker has a probability value representing the likelihood that the speech signal belongs to that speaker.

[0052] Ultimately, the speaker recognition model SincGAN will output the speaker with the highest probability as the prediction result.

[0053] This invention utilizes conditional generative adversarial networks to perform end-to-end recognition of the original waveform, and improves the regularization of the recognition network by jointly optimizing the "true / false" objective function and the classification objective function.

[0054] like Figure 2 As shown, the speaker recognition model SincGAN consists of a generator network and a discriminator network.

[0055] First, multi-scale SincNet is used to extract two-dimensional shallow speaker features from a series of raw input signals. Then, a generator is used to perform a series of convolution and transpose convolution operations to generate speech waveforms of the same size. The purpose is to use the generated samples to alleviate the overfitting problem of the discriminator. When training data is lacking, using fake samples generated by the generator to train the model is currently the best solution because it is closer to the distribution of real data.

[0056] like Figure 3 As shown, the generator network consists of a multi-scale SincNet layer, three convolutional layers, two transposed convolutional layers, and an adaptive average pooling layer.

[0057] Define the multi-scale SincNet layer in the generator network as the first multi-scale SincNet layer; define the three convolutional layers in the generator network as the first, second, and third convolutional layers, and the two transposed convolutional layers as the first and second transposed convolutional layers.

[0058] The goal of generator networks is to generate high-quality fake speech samples, which are then labeled with tags learned from real speech data. Traditional generators typically use simple convolutional or fully connected layers.

[0059] To ensure that the generated speech frame samples have the same size as the original speech frames, this embodiment adds a transposed convolutional layer to the generator to achieve the mapping operation of speaker feature maps from small resolution to large resolution.

[0060] Transposed convolutional layers, also known as deconvolutional layers, are widely used in various upsampled deep models, such as semantic segmentation. The main difference between them and convolutional layers is that the speaker feature map output by this layer is larger than the speaker feature map input.

[0061] The processing flow of real speech samples in the generator network is as follows:

[0062] The real speech sample first passes through the first multi-scale SincNet layer for feature extraction to obtain the two-dimensional features of the speech signal. Then, the two-dimensional features of the speech signal pass through the first convolutional layer, the second convolutional layer, the first transposed convolutional layer, the second transposed convolutional layer, the third convolutional layer, and the adaptive average pooling layer in sequence to generate fake speech samples.

[0063] This invention introduces a multi-scale SincNet layer to extract speaker features. It captures low-level speech representations of three channels in the waveform using three custom filter banks, enabling the model to better capture important narrowband speaker features (such as pitch and formants) and effectively avoid the loss of important information during the conversion of handcrafted features.

[0064] like Figure 4 As shown, the discriminator network consists of a multi-scale SincNet layer, five convolutional layers, three bottleneck residual block stacked layers, and four fully connected layers.

[0065] The multi-scale SincNet layer in the discriminator network is defined as the second multi-scale SincNet layer.

[0066] The five convolutional layers in the discriminator network are defined as the fourth, fifth, sixth, seventh, and eighth convolutional layers.

[0067] The discriminator network is defined with three bottleneck residual block stacks as the first, second, and third bottleneck residual block stacks, and four fully connected layers as the first, second, third, and fourth fully connected layers.

[0068] For discriminators, increasing network depth is generally considered very effective in improving model performance. However, when the number of network layers exceeds a certain order of magnitude, classification performance actually deteriorates. This is because in excessively deep network structures, information transmission across multiple layers can cause gradients to gradually vanish during backpropagation, making it difficult to update network parameters and affecting model training and performance.

[0069] Therefore, this invention adds a bottleneck residual block, different from the standard residual block, to the discriminator network to avoid gradient vanishing and reduce training time. The bottleneck residual block structure is as follows: Figure 5 As shown, it uses two 1x1 convolution kernels for dimensionality reduction and dimensionality increase operations, which significantly reduces the number of parameters compared to standard residual blocks that directly perform 3x3 convolution on the input data.

[0070] Compared with standard residual blocks, the embodiments of the present invention can effectively improve the recognition performance and training speed of the discriminator by introducing bottleneck residual blocks into the discriminator network, and avoid the gradient vanishing problem.

[0071] The processing flow of real and fake speech samples in the discriminator network is as follows:

[0072] Real and fake speech samples are first processed by the second multi-scale SincNet layer for feature extraction, resulting in two-dimensional features of the speech signal. These two-dimensional features are then sequentially processed through the fourth convolutional layer, the first bottleneck residual block stacked layer ResBlock1, the fifth convolutional layer, the second bottleneck residual block stacked layer ResBlock2, the sixth convolutional layer, the second bottleneck residual block stacked layer ResBlock3, the seventh convolutional layer, the eighth convolutional layer, the first fully connected layer, and the second fully connected layer. The output of the second fully connected layer is split into two paths: one path passes through the third fully connected layer to output a true / false flag, and the other path passes through the fourth fully connected layer to output an N-dimensional vector {S1, S2, ..., S...}. N}. S1,S2,...,S N These represent the vectors corresponding to the speech sounds for N speakers.

[0073] The N-dimensional vector output by the discriminator network is input into the Softmax function, which maps the output vector to a probability distribution. Each possible speaker has a probability value, representing the likelihood that the speech signal belongs to that speaker. Finally, SincGAN outputs the speaker category label of the most probable predicted category as the prediction output.

[0074] Assuming the last layer of the discriminator network uses the Softmax function to output... This means that the i-th speech frame comes from the k-th speaker S out of N speakers. k The probability of.

[0075] The estimated speaker identity for the entire discourse x corresponds to the speaker label with the highest probability, as shown in the following formula:

[0076]

[0077] Where S k Let P(y = k|x) represent the probability that the input speech frame x belongs to category k.

[0078] like Figure 6As shown, this invention utilizes an improved conditional generative adversarial network (CGAN) to augment small datasets, expanding the dataset size and directly identifying speakers. The CGAN improves the generalization ability of the model for classification tasks by reconstructing the input samples and using the generated sample data. Unlike conventional CGANs, the generator network discards random noise z and directly reconstructs the input samples using real sample data as conditions.

[0079] The reconstruction here can be viewed as a feature transformation, converting the real input sample into a generated sample. In the improved CGAN, its discriminator has two outputs: one is the same as the standard GAN, i.e., true / false. The other is a standard classification output, corresponding to the speaker category label of the real input sample, i.e., an N-dimensional vector {S1, S2, ..., S...}. N}

[0080] Building upon SincNet, this invention designs a multi-scale SincNet, a deep neural network model that uses discrete convolutions to capture different signal frequencies. Typically, filters in standard convolutional layers have the same kernel size, making it difficult to simultaneously learn high-frequency and low-frequency information of a signal. The solution to this problem is to divide a convolutional branch into several parallel branches of different sizes. This allows for the customization of different parameter sets for the convolutional layer, including the number of filters, kernel size, and stride. This enables filters at each scale to effectively respond to different frequency components.

[0081] In this embodiment, the first multi-scale SincNet layer and the second multi-scale SincNet layer have the same structure. The explanation will be based on any one of the multi-scale SincNet layers as an example. Figure 7 As shown.

[0082] like Figure 7 As shown, each multi-scale SincNet layer consists of layer normalization and three different parallel branches; each parallel branch includes a SincNet filter, a batch normalization layer and a one-dimensional adaptive average pooling layer.

[0083] The SincNet filters on each parallel branch have different kernel lengths.

[0084] For example, the SincNet filter on the first parallel branch has a length L = 251 and a number K = 160, and consists of a batch normalization layer, a ReLU nonlinear activation function, and a one-dimensional adaptive average pooling layer.

[0085] The SincNet filter on the second parallel branch has a length L = 501 and a number K = 160, and consists of a batch normalization layer, a ReLU activation function, and a one-dimensional adaptive average pooling layer.

[0086] The SincNet filter on the third parallel branch has a length L = 1001 and a number K = 160, and consists of a batch normalization layer, a ReLU activation function, and a one-dimensional adaptive average pooling layer.

[0087] This invention uses the mel-scale cutoff frequency to initialize the parameters of the multi-scale SincNet.

[0088] The feature extraction process for speech frames on multi-scale SincNet is as follows:

[0089] First, the speech frame is normalized by a layer and then enters the SincNet filters in three parallel branches to learn speaker feature maps at different frequency resolutions. Then, the speaker feature maps output by the SincNet filters in each parallel branch are processed by the batch normalization layer and the one-dimensional adaptive average pooling layer (AdaptiveAvgPool1d) in their respective branches. Finally, the one-dimensional speaker feature maps output by the three parallel branches are stacked into a two-dimensional speaker feature map and fed into a two-dimensional convolutional layer.

[0090] This processing method can simultaneously extract feature maps across time and frequency. The two-dimensional speaker feature map serves two purposes: 1. It guides the generator to perform controlled feature transformations, thereby expanding the scale of the training data; 2. It provides the discriminator with more discriminative speaker features, enabling the model to more accurately identify the speaker's identity.

[0091] Step 3. Use the training samples from Step 1 to train the speaker recognition model SincGAN built in Step 2. Optimize the parameters of the speaker recognition model through backpropagation to minimize the loss function, and obtain the trained speaker recognition model SincGAN. Then, use the test samples to test the trained speaker recognition model SincGAN.

[0092] During training, Wasserstein is used to measure the distance between the distributions of features of real and fake speech samples to stabilize the training environment of the SincGAN model and avoid mode collapse during model training. Least squares loss is used to further refine the objective function of SincGAN to improve the generalization ability of the model.

[0093] Objective function of generator G As shown in formula (1);

[0094]

[0095] Where x represents the real sample, i.e. the real speech sample; G(x) is the generated sample output by the generator, i.e. the fake speech sample; and D(G(x)) represents the probability that the discriminator will identify the fake speech sample as real.

[0096] P r (x) represents the distribution of the true sample x; E(·) represents the squared probability of the discriminator identifying a fake speech sample as a real speech sample minus 1; E(·) is the calculation of the expected value.

[0097] By adding an L1 norm to the generator G, the distance between generated samples and real samples is minimized and the generated samples are made more realistic. The weight of the L1 norm is controlled by the hyperparameter ω.

[0098] Objective function of discriminator D It includes adversarial loss and classification cross-entropy loss, as shown in Equation (2);

[0099]

[0100] in, It signifies resistance to loss.

[0101] This represents the expected value of the square of the probability that the discriminator will identify a fake speech sample as a real speech sample; This represents the expected value of the square of the probability that the discriminator judges a real speech sample minus 1.

[0102] loss D Let represent the category cross-entropy loss for all speaking humans, as shown in formula (3):

[0103]

[0104] This indicates that x and y follow a P-order. r Expected value under the (x,y) distribution.

[0105] x and y represent real speech samples and labels, respectively. i y represents the i-th audio frame with the real label. i ∈R N The vector that constitutes each speech frame i; k i Let represent the probability that the model predicts y to belong to the i-th category, and N represent the total number of speakers.

[0106] As can be seen from formula (2), during the training process, the discriminator aims to make the output probability value of the input real speech sample x as close to 1 as possible, that is, D(x) tends to 1, and the output probability value of the input generated data G(x) as close to 0 as possible, that is, D(G(x)) tends to 0. After many experiments, it was found that adding an L1 norm to the generator G can minimize the distance between the generated samples and the real samples and make the generated samples more realistic. In addition, the ω hyperparameter is used to control the weight of the L1 norm. The final loss function of the generator G is shown in formula (1).

[0107] Step 4. Use the trained SincGAN to predict the given speech signal and output the corresponding speaker label.

[0108] This invention employs an end-to-end deep learning approach to directly distinguish raw speech signals. During training, the model learns how to extract useful features from the raw speech signal, providing a data foundation for model training. Therefore, this invention is applicable to multiple speakers and various speech environments, eliminating the need for separate modeling for each speaker.

[0109] To verify the effectiveness of the method of this invention, the two publicly available datasets, LIBRISPEECH and TIMIT, are used to evaluate it. All utterances were sampled at a sampling rate of 16 kHz and a bit rate of 16 bits.

[0110] For the LIBRISPEECH corpus, the train-clean-100 subset was used. This subset contains 251 speakers (125 females and 126 males), each speaker having an average of 113 utterances ranging from 1 to 15 seconds in duration. 60% of all utterances were randomly selected as training data, and the remainder as test data.

[0111] The TIMIT corpus contains 630 speakers (192 females and 438 males) from eight major dialect regions in the United States, with 10 utterances per speaker, totaling 6300 speech data points. This invention selects 5 utterances ('SX') from 462 speakers in the training set to train the model, and the remaining three sentences (SI) to test the model. Since the experiments in this invention are based on the original waveforms, statistical analysis is performed on the segmented speech frames. Each speech utterance is divided into speech frames with a length of 200ms and an overlap of 10ms. Therefore, each speech frame contains 3200 samples. In the TIMIT corpus, the minimum speech frame length for each speaker's training and test sentences is 72 and 79, respectively, with averages of approximately 275 and 307, respectively. Discourses with a speech frame length less than 300 account for 61.9% of the total. In the LIBRISPEECH corpus, the minimum number of speech frames in each speaker's training and testing sentences is 122 and 145, respectively, and the average number is approximately 1249 and 1250, respectively, which is larger than the TIMIT corpus.

[0112] Furthermore, to evaluate the recognition performance of the proposed model in real-world scenarios, this invention uses the VoxCeleb1 corpus to train and evaluate the SincGAN system. This corpus was collected from public multimedia data, and the speech contains some real noise, not artificial white noise. The VoxCeleb1 corpus consists of over 100,000 sentences from 1251 speakers, 55% of whom are male and the remainder are female. The speakers come from diverse ethnicities, accents, occupations, and ages. This invention selects 100 speakers, taking 100 sentences from each. 80% of the data is used for training, and 20% is used for testing.

[0113] The SincNet filter settings in the SincGAN model follow the original design. When the mini-batch size is set to 64, the Adam optimizer is used to optimize the network, and batch normalization is applied after the convolutional and transposed convolutional layers. To improve the model's convergence speed, different learning rates are set for different epochs: 0.0005 for the first 20 epochs and 0.0002 for the rest. To make the generated fake samples more realistic, the hyperparameter ω in the formula is set to 100. Leaky ReLU with a negative slope of 0.03 is used as the non-linear activation function in both the generator and classifier. The experimental procedure is implemented using the PyTorch framework, with the hyperparameter K set to 4.

[0114] To evaluate the performance of the proposed model, this invention selected three speaker recognition methods: SincNet, CNN-Raw, and CGAN-FBANK. Unlike the SincGAN experiment, energy-based speech activity detection (VAD) was performed on each sentence to eliminate silence before the baseline model experiment, and then the data was fed into the baseline model.

[0115] The SincNet framework comprises three convolutional layers and three fully connected layers. The first convolutional layer performs Sinc-based convolutions using K (K=80) filters of length L (L=251). The filters are initialized with Mel-scale cutoff frequencies, which are adjusted using standard backpropagation, just like any other layer. Next are two standard convolutional layers, each consisting of 60 filters of length L=5, using layer normalization. Finally, there are three fully connected layers, each with 2048 neurons, using batch normalization to accelerate model training.

[0116] In the CNN-Raw model, the raw waveform is first fed directly into the first layer, and then three convolutional layers are used for feature mapping. Each convolutional layer consists of 80 filters and a max pooling layer, followed by three fully connected layers (2048 neurons), and batch normalization is used.

[0117] The CGAN-FBANK baseline adopts a similar framework to the method proposed in this invention, highlighting the advantages of the proposed model. Specifically, the input raw waveform undergoes silence removal and mean-variance normalization, followed by the extraction of 64-dimensional FBANK features with a fixed frame count of 240, resulting in a 64×240 feature map that is input into CGAN. Furthermore, to increase nonlinearity, all hidden layers of the three baseline models utilize the Leaky ReLU activation function, and speaker classification is performed using the Softmax function and cross-entropy criterion.

[0118] This invention selects recognition accuracy to evaluate the proposed speaker recognition model. Recognition accuracy is represented by the number of correctly recognized samples out of the speech samples to be recognized divided by the total number of test speech samples, as shown in Table 1.

[0119] Table 1 shows the ablation study conducted on the filter initialization method, number of filters, and number of bottleneck residual blocks for the SincGAN architecture proposed in this invention, using the TIMIT and LIBRISPEECH corpora.

[0120] Table 1

[0121]

[0122] In Table 1, the baseline SincNet achieves state-of-the-art classification accuracies of 99.15% (K=80, L=251) and 99.11% (K=160, L=501) on the TIMIT and LIBRISPEECH datasets, respectively. It can be observed that the baseline SincNet struggles to find a filter length L that achieves optimal classification accuracy across all datasets. On the TIMIT dataset, CGAN-FBANK and CGAN-MFCC achieve classification accuracies of 87.14% and 86.89%, respectively. SincGAN, however, improves its classification accuracy to 98.98% when K=160 and L=(251,501,1001).

[0123] Therefore, it can be concluded that the multi-scale SincNet of this invention can learn more speaker-discriminative features, thereby achieving higher classification accuracy than when based on manual feature recognition. When using multi-scale SincNet and three residual blocks, SincGAN achieved a state-of-the-art classification accuracy of 99.27% ​​on the LIBEISPEECH corpus.

[0124] Table 2 compares the classification error rate (CER) of the model in this invention with that of the state-of-the-art model on the LIBRISPEECH corpus. The I-vector performs classification evaluation on the 20-dimensional MFCC, while the x-vector takes the 24-dimensional FBANK as input.

[0125] Table 2

[0126]

[0127] As shown in Table 2, traditional handcrafted features based on speech signals—MFCC and FBANK—achieved different CERs on speaker recognition models based on CGAN, i-vector, and x-vector, respectively. FBANK exhibits strong correlation, enabling neural networks to model spectral correlations in speaker features, thus achieving better performance. The proposed method reduces the classification error rate (CER) to 0.73% on the Librispeech-100 dataset, while the CGAN model, using FBANK features as input, achieved a CER of 1.96%. Compared to SincNet and CNN-Raw, SincGAN outperforms SincNet and CNN-Raw by 12% and 27%, respectively, on the LIBRISPEECH corpus.

[0128] Compared to CGAN-MFCC, the learning curve of SincGAN is as follows: Figure 9 As shown. Among them Figure 9(a) shows the trend of recognition accuracy for the two models. Both SincGAN and CGAN-MFCC show an upward trend with the increase of training epochs. However, SincGAN's recognition accuracy increases faster and converges to better performance, achieving a recognition accuracy of 99.27%. The baseline CGAN-MFCC, on the other hand, has a recognition accuracy of 97.06%. Figure 9 (b) shows the changes in the loss values ​​of the two models. The loss values ​​decrease with increasing training time. Although SincGAN's test loss value is higher than CGAN-MFCC, its recognition performance is better than CGAN-MFCC. This demonstrates that the magnitude of the loss value and the recognition accuracy are not inversely proportional.

[0129] Table 3 shows the test losses of different models (CNN-Raw, SincNet, and the SincGAN of this invention) on the LIBRISPEECH corpus.

[0130] Table 3

[0131]

[0132] As shown in Table 3, the loss value of the proposed model is higher than that of the baseline model. This is because the real / false distinction in the classifier interferes with the classification during training, but it does not affect the final performance of the system. On the LibriSpeech-100 dataset, 10-25 utterances were selected for training for each speaker. According to Table 2, the model with the best performance was selected to configure the baseline.

[0133] Table 4 reports the changes in recognition accuracy under different training utterances.

[0134] Table 4

[0135]

[0136] As shown in Table 4, SincGAN achieves significant enhancements when training data is limited. With only 10 utterances per speaker, SincGAN achieves a recognition accuracy of 90.75%, which is approximately 14% and 35% lower than the CER of SincNet and CNN-Raw, respectively. Experimental results demonstrate that the method of this invention improves the model's versatility.

[0137] Compared to the baseline, the method of this invention alleviates the overfitting problem when training data is limited. This is mainly because the generator has already learned about the distribution of real speech data and generated corresponding fake data. Under the adversarial mechanism, new generated data information is introduced to correct the original optimization process of the classifier.

[0138] Table 5 shows the classification error rate (CER%) of each speaker recognition system on the VoxCeleb dataset.

[0139] Table 5

[0140]

[0141] As shown in Table 5, the SincGAN model of this invention achieved a classification error rate of 23.19% when using 3-second utterances. For the full-length utterances, the SincGAN model achieved a classification error rate of 11.65%, representing reductions of 50.53% and 10.93% compared to CNN-Raw and SincNet methods, respectively. This demonstrates that the SincGAN model outperforms baseline methods on the VoxCeleb corpus and exhibits strong competitiveness in real-world scenarios.

[0142] Example 2

[0143] This embodiment 2 describes a computer device for implementing the end-to-end speaker recognition steps based on multi-scale SincNet and CGAN described in embodiment 1 above.

[0144] The computer device includes memory and one or more processors.

[0145] The executable code is stored in the memory, and when the processor executes the executable code, it is used to implement the steps of end-to-end speaker recognition based on multi-scale SincNet and CGAN.

[0146] In this embodiment, the computer device can be any device or apparatus with data processing capabilities, and will not be described in detail here.

[0147] Example 3

[0148] This embodiment 3 describes a computer-readable storage medium for implementing the end-to-end speaker recognition steps based on multi-scale SincNet and CGAN described in embodiment 1 above.

[0149] The computer-readable storage medium in this embodiment 3 stores a program that, when executed by a processor, is used to implement the steps of end-to-end speaker recognition based on multi-scale SincNet and CGAN.

[0150] The computer-readable storage medium can be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc.

[0151] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. An end-to-end speaker recognition method based on multi-scale SincNet and CGAN, characterized in that, Includes the following steps: Step 1. Perform speech frame preprocessing on the input raw speech signal to obtain speech frames, and use the speech frames as real speech samples; divide the real speech samples into training samples and test samples, which are used for model training and model testing, respectively. Step 2. Build the speaker recognition model SincGAN; The speaker recognition model SincGAN consists of a generator network and a discriminator network; The generator network consists of a multi-scale SincNet layer, three convolutional layers, two transposed convolutional layers, and an adaptive average pooling layer. Define the multi-scale SincNet layer in the generator network as the first multi-scale SincNet layer; define the three convolutional layers in the generator network as the first, second, and third convolutional layers, and the two transposed convolutional layers as the first and second transposed convolutional layers, respectively; The processing flow of real speech samples in the generator network is as follows: The real speech sample first passes through the first multi-scale SincNet layer for feature extraction to obtain the two-dimensional features of the speech signal. Then, the two-dimensional features of the speech signal pass through the first convolutional layer, the second convolutional layer, the first transposed convolutional layer, the second transposed convolutional layer, the third convolutional layer, and the adaptive average pooling layer in sequence to generate fake speech samples. The discriminator network consists of a multi-scale SincNet layer, five convolutional layers, three bottleneck residual block stacked layers, and four fully connected layers; Define the multi-scale SincNet layer in the discriminator network as the second multi-scale SincNet layer; The five convolutional layers in the discriminator network are defined as the fourth, fifth, sixth, seventh, and eighth convolutional layers; The discriminator network is defined with three bottleneck residual block stacked layers as the first, second, and third bottleneck residual block stacked layers, and four fully connected layers as the first, second, third, and fourth fully connected layers. The processing flow of real and fake speech samples in the discriminator network is as follows: Real speech samples and fake speech samples are first processed through the second multi-scale SincNet layer to extract features, resulting in two-dimensional features of the speech signal. Then, the two-dimensional features of the speech signal are sequentially processed through the fourth convolutional layer, the first bottleneck residual block stacking layer, the fifth convolutional layer, the second bottleneck residual block stacking layer, the sixth convolutional layer, the second bottleneck residual block stacking layer, the seventh convolutional layer, the eighth convolutional layer, the first fully connected layer, and the second fully connected layer. The output of the second fully connected layer is divided into two paths: one path goes through the third fully connected layer to output true / false flags, and the other path goes through the fourth fully connected layer to output an N-dimensional vector, which corresponds to the speaker category label of the real speech sample. The N-dimensional vector output by the discriminator network is input into the Softmax function. By mapping the output vector onto a probability distribution, the speaker category label of the most probable predicted category is used as the prediction output. Step 3. Use the training samples from Step 1 to train the speaker recognition model SincGAN built in Step 2. Optimize the parameters of the speaker recognition model through backpropagation to minimize the objective function, obtain the trained speaker recognition model SincGAN, and test the trained speaker recognition model SincGAN using test samples. Step 4. Use the trained SincGAN to predict the given speech signal and output the corresponding speaker label.

2. The end-to-end SincGAN speaker identification method according to claim 1, characterized in that, The first multi-scale SincNet and the second multi-scale SincNet have the same structure; Each multi-scale SincNet consists of a layer normalization layer and three different parallel branches; each parallel branch includes a SincNet layer, a batch normalization layer, and a one-dimensional adaptive average pooling layer. The filters in the SincNet layers on each parallel branch have different kernel lengths; The feature extraction process for speech frames on multi-scale SincNet is as follows: First, the speech frame is normalized by the layer and then enters the SincNet filter in three parallel branches to learn speaker feature maps at different frequency resolutions. Then, the speaker feature maps output by the SincNet filter in each parallel branch are processed by the batch normalization layer and the one-dimensional adaptive average pooling layer in their respective branches. Finally, the one-dimensional speaker feature maps output from the three parallel branches are stacked into a two-dimensional speaker feature map.

3. The end-to-end SincGAN speaker identification method according to claim 1, characterized in that, There are N speakers, each speaker has m sentences, and the set of the m-th sentences of the k-th speaker is S. k ={h1,h2,...,h m }, k=1,2,...,N, h1,h2,...,h m Each represents a different statement; The speaker recognition model SincGAN takes speech frames as input data. The utterance x input to SincGAN is divided into n fixed-length speech frames {x1, x2, ..., xn}. n }; Each speech frame is sequentially fed into SincGAN, and the last layer of the discriminator network uses a Softmax layer as its output. This means that the i-th speech frame comes from the k-th speaker S out of N speakers. k The probability of; The estimated speaker identity for the entire discourse x corresponds to the speaker label with the highest probability, as shown in the following formula:

4. The end-to-end SincGAN speaker identification method according to claim 1, characterized in that, In step 3, during the training process, Wasserstein is used to measure the distance between the distributions of features of real speech samples and fake speech samples, and least squares loss is used to further modify the objective function of SincGAN. Objective function of generator G As shown in formula (1); Where x represents a real sample, i.e. a real speech sample, G(x) represents a sample generated by the generator, i.e. a fake speech sample; D(G(x)) represents the probability that the discriminator will identify a fake speech sample as a real speech sample. P r (x) represents the distribution of the true sample. E(·) represents the squared probability of the discriminator identifying a fake speech sample as a real speech sample minus 1; E(·) is the calculation of the expected value. By adding an L1 norm to the generator G, the distance between generated samples and real samples can be minimized and the generated samples can be made more realistic. The weight of the L1 norm is controlled by the hyperparameter ω. Objective function of discriminator D As shown in formula (2); D(x) represents the probability that x is a true sample; This represents the expected value of the square of the probability that the discriminator will classify a fake speech sample as a real speech sample. This represents the expected value of the square of the probability that the discriminator judges a real speech sample minus 1. loss D Let represent the category cross-entropy loss for all speaking humans, as shown in formula (3): in, This indicates that x and y follow a P-order. r Expected value under the (x,y) distribution; x and y represent real speech samples and labels, respectively; y i y represents the i-th audio frame with the real label. i ∈R N The vector that constitutes each speech frame i, k i Let represent the probability that the model predicts y to belong to the i-th category, and N represent the total number of speakers.

5. A computer device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements the steps of the end-to-end speaker recognition method based on multi-scale SincNet and CGAN as described in any one of claims 1 to 4.

6. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the end-to-end speaker recognition method based on multi-scale SincNet and CGAN as described in any one of claims 1 to 4.

Images