A speaker recognition method based on deep convolutional encoder

Abstract

This invention discloses a speaker recognition method based on a deep convolutional encoder, belonging to the field of speech recognition technology. The method first receives the speech to be recognized from a speaker; then, it segments the speech according to a preset duration and performs data augmentation; next, it extracts Mel-spectral features from each audio segment; finally, it inputs the Mel-spectral features into a pre-trained recognition model and outputs the speaker's identity from the speech. The speaker recognition method proposed in this invention significantly improves accuracy compared to mainstream speaker recognition methods.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition technology, and more specifically, relates to a speaker recognition method based on a deep convolutional encoder. Background Technology

[0002] With the advent of the deep learning era, neural networks have achieved outstanding performance in speaker recognition tasks. The conventional operation in deep learning methods is to classify speakers to achieve speaker recognition, and to obtain the speaker's speech and specific dimension embeddings as the speaker's voiceprint feature representation in the speaker classification network using different pooling operations. In recent years, the transformer model has sparked a research boom in the field of natural language processing due to its high parallelism and high computational power. More and more researchers are using this model in fields such as computer vision and speech processing. However, there are certain challenges in using the transformer model in speaker recognition tasks. The existing speaker recognition methods have the following shortcomings: (1) First, the transformer lacks the inductive bias inherent in CNNs, such as translation equivalence and locality. Directly using it for speaker recognition will cause the loss of local dependencies; (2) Second, the transformer's encoder-decoder is more suitable for end-to-end sequence tasks. Directly using it for speaker recognition will not only increase the computational complexity of the model but also reduce training efficiency; (3) Traditional CNNs can effectively extract local features of images. However, increasing the number of network models inevitably leads to an increase in model parameters and training difficulty. In speaker recognition tasks, not all voiceprint information is useful. Neural networks need to extract the representations that best represent the speaker's characteristics in order to more effectively distinguish speakers. Summary of the Invention

[0003] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a speaker recognition method based on a deep convolutional encoder, which aims to improve the accuracy of speech recognition identity.

[0004] In a first aspect, the present invention provides a method for training a speaker recognition model based on a deep convolutional encoder, the method comprising:

[0005] The audio data in the training set is segmented according to a preset duration and then augmented.

[0006] Extract Mel-spectral features from each audio segment;

[0007] A recognition model is constructed, which first extracts feature vectors containing local dependencies of the voiceprint from the Mel spectrum features, and then extracts voiceprint embeddings containing global correlations of the voiceprint from the feature vectors through a multi-head attention mechanism.

[0008] The voiceprint embedding is classified using a loss function, and the recognition model is trained iteratively by reducing the loss, until the recognition model is finally obtained after training.

[0009] Further, extracting feature vectors containing local dependencies of the voiceprint from the Mel spectrum features includes:

[0010] The first feature is obtained by performing an initial convolution operation on the Mel spectrum features through a one-dimensional dilated convolutional layer;

[0011] The first feature is then processed by an attention residual module (SE-Res2Block) to obtain the second feature;

[0012] The second feature is then processed by an attention residual module to obtain the third feature;

[0013] The third feature is then processed by an attention residual module to obtain the fourth feature;

[0014] The first feature, second feature, third feature and fourth feature are aggregated through a one-dimensional dilated convolutional layer to obtain aggregated features;

[0015] The aggregated features are then processed by a max-pooling layer to extract voiceprint features.

[0016] The voiceprint features are processed through a linear layer and a normalization layer to obtain a feature vector containing local voiceprint dependencies.

[0017] Furthermore, a voiceprint embedding containing global voiceprint correlation is extracted from the feature vector using a multi-head attention mechanism, specifically as follows:

[0018] The feature vector is input into a deep encoder, and a voiceprint embedding containing global voiceprint correlation is obtained through the learning of the multi-head attention mechanism in the deep encoder (Transformer Encoder). The feature vector is divided into multiple vector segments according to the number of heads of the multi-head attention mechanism, and each vector segment is input into each multi-head attention mechanism to capture contextual information.

[0019] Secondly, the present invention provides a speaker recognition method based on a deep convolutional encoder, the method comprising:

[0020] Receive the speaker's speech to be recognized;

[0021] The speech to be recognized is segmented into segments according to a preset duration and data augmentation is performed;

[0022] Extract Mel-spectral features from each audio segment;

[0023] The Mel spectrum features are input into a pre-trained recognition model, which outputs the speaker identity of the speech to be recognized; wherein the recognition model is trained according to any one or more methods described in the first aspect above.

[0024] Thirdly, the present invention provides a speaker recognition model training device based on a deep convolutional encoder, characterized in that the device comprises:

[0025] The preprocessing module is used to segment the audio data in the training set according to a preset duration and perform data augmentation.

[0026] The feature extraction module is used to extract Mel-spectral features from each audio segment;

[0027] The model building module is used to build a recognition model. The recognition model first extracts feature vectors containing local dependencies of the voiceprint from the Mel spectrum features, and then extracts voiceprint embeddings containing global correlations of the voiceprint from the feature vectors through a multi-head attention mechanism.

[0028] The model training module is used to classify the voiceprint embedding using a loss function, and to train and iterate the recognition model by reducing the loss, and finally obtain the recognition model after training is completed.

[0029] Furthermore, the model building module includes:

[0030] The first submodule is used to perform an initial convolution operation on the Mel spectrum features through a one-dimensional dilated convolutional layer to obtain the first feature;

[0031] The second submodule is used to process the first feature through an attention residual module (SE-Res2Block) to obtain the second feature;

[0032] The third submodule is used to process the second feature through an attention residual module to obtain the third feature;

[0033] The fourth submodule is used to process the third feature through an attention residual module to obtain the fourth feature;

[0034] The fifth submodule is used to aggregate the first feature, the second feature, the third feature and the fourth feature through a one-dimensional dilated convolutional layer to obtain aggregated features;

[0035] The sixth submodule is used to extract voiceprint features from the aggregated features through a max-pooling layer;

[0036] The seventh submodule is used to process the voiceprint features through a linear layer and a normalization layer to obtain a feature vector containing local voiceprint dependencies.

[0037] Furthermore, the model building module includes an eighth sub-module:

[0038] The eighth submodule is specifically used to input the feature vector into the deep encoder (TransformerEncoder) and obtain a voiceprint embedding containing global voiceprint correlation through the learning of the multi-head attention mechanism in the deep encoder; wherein, the feature vector is divided into multiple vector segments according to the number of heads of the multi-head attention mechanism, and each vector segment is input into each multi-head attention mechanism to capture context information.

[0039] Fourthly, the present invention provides a speaker recognition device based on a deep convolutional encoder, characterized in that the device comprises:

[0040] A receiving unit is used to receive the speaker's speech to be recognized.

[0041] The preprocessing unit is used to segment the speech to be recognized into segments according to a preset duration and perform data enhancement;

[0042] The feature extraction unit is used to extract Mel-spectral features from each audio segment;

[0043] The recognition unit is used to input the Mel spectrum features into a pre-trained recognition model and output the speaker identity of the speech to be recognized; wherein the recognition model is trained according to any one or more methods described in the first aspect above.

[0044] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0045] (1) In order to extract the local dependencies of the speaker’s speech, this invention improves the ECAPA-TDNN network model by introducing additional residual connections: the output of the first convolutional layer is aggregated with the output of SE-Res2Block at multiple scales, thereby fusing coarse-grained features and fine-grained features in the speech, which can maximize the transmission of information and improve the robustness of the model; secondly, the maximum pooling layer is used to replace the statistical pooling attention layer at the end of the model, which can obtain the most important features in the speech information.

[0046] (2) This invention introduces the transformer architecture, which is widely used in natural language processing and machine vision, into the speaker recognition task, effectively extracting the global relationship of speaker voiceprint information;

[0047] (3) This invention extracts the local dependencies of voiceprints based on the improved ECAPA-TDNN network model and performs fusion training with the encoder in the transformer architecture, which can make up for the lack of induction bias inherent in CNN in transformer and effectively extract the local features of speaker voiceprints.

[0048] (4) The method of the present invention was verified on a public corpus through speaker identification and speaker confirmation tasks, proving that the model proposed in the invention has a significant improvement over the mainstream model ECAPA. Attached Figure Description

[0049] Figure 1 This is an overall flowchart of the speaker recognition method in this invention;

[0050] Figure 2 This is a flowchart of the recognition model training method in this invention;

[0051] Figure 3 This is a flowchart of the learning method for extracting global correlation of voiceprints in this invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0053] This invention uses the ECAPA-TDNN framework in the field of speech recognition technology as a basis and improves it to extract local dependencies in voiceprints. It uses the encoder in the transformer to introduce a multi-head attention mechanism into the extracted feature vectors to obtain voiceprint embeddings containing global context information of the voiceprints. The voiceprint embeddings are then filtered to obtain more valuable voiceprint embeddings to improve the performance of speaker recognition tasks.

[0054] The speaker identification method in this invention can be divided into three parts: data preprocessing, feature extraction, and speaker identification through a recognition model.

[0055] The technical solution of the present invention will now be described in detail with reference to the embodiments. The implementation process of the embodiments is as follows: Figure 1 As shown, the specific steps for training the recognition model are as follows:

[0056] (1) First, the audio data in the training set and the test set should be preprocessed. Audio preprocessing mainly includes random audio cropping and data augmentation. In this embodiment, 3 seconds is used as the fixed duration of each audio file for cropping. If it is insufficient, the audio signal is automatically padded with 0. In order to enhance the robustness of the model, background noise, speech rate, volume and speech enhancement operations are added to the audio.

[0057] (2) After preprocessing the audio, features are extracted for each audio segment. In feature extraction, the popular tool librosa is used to extract 80-dimensional mel-spectrogram features, where the window length FFT in librosa is set to 400 and the distance between adjacent windows hop_length is set to 160.

[0058] (3) Constructing a speaker recognition model. This embodiment aims to combine some modules of the transformer with an improved convolutional neural network, using local dependency extraction and global contextual information attention methods respectively to improve the performance of the speaker recognition model; such as Figure 2 The diagram shows the detailed process of building and training the speaker recognition model in this embodiment, specifically as follows:

[0059] (31) Extract feature vectors containing local dependencies of voiceprints from Mel spectrum features; adopt the ECAPA-TDNN network model in the field of speech recognition as the basic framework and improve it into the Transfer_ECAPA model to extract feature vectors with local dependencies of voiceprints, the specific structure of which is as follows. Figure 2 The right half is shown:

[0060] The 80-dimensional Mel-spectral features are input and first pass through a one-dimensional dilated convolutional layer for initial convolution. Then, they pass through three stacked SE-Res2Block modules to extract frame-level features of the speaker's voiceprint. By introducing residual connections between SE-Res2Block modules and an additional one-dimensional dilated convolutional layer, coarse-grained and fine-grained features in the voiceprint are fused. The output of the first convolutional layer is also aggregated with the output of SE-Res2Block to ensure the effective transmission of speaker information in the neural network. The speaker's voiceprint features are extracted through a max-pooling layer and finally passed through a linear layer and a normalization layer to obtain a 192-dimensional feature vector containing local dependencies of the voiceprint.

[0061] (32) Input the 192-dimensional feature vector containing local dependencies of the voiceprint into the Transformer Encoder, and extract the contextual relevance of the feature vector from the 192-dimensional feature vector through a multi-head attention mechanism, where Patch hEmbedding represents the vector segmentation of the 192-dimensional feature vector according to the number of heads in the multi-head attention mechanism; through learning the multi-head attention mechanism in the Transformer Encoder, a voiceprint embedding representation that includes both local voiceprint dependencies and global voiceprint correlations can be obtained, which is the final 192-dimensional voiceprint embedding.

[0062] Figure 2 In this context, Conv1D represents a one-dimensional dilated convolutional layer, ReLU represents the activation function, BN represents a batch normalization layer, FC represents a fully connected layer, SE represents a squeezed activation module, and Res2Block represents a residual module; Attentive Max-Pooling represents a maximum self-attention pooling layer, k represents the kernel size, and d represents the stride of the dilated convolution.

[0063] Figure 2 In the process, the one-dimensional compression excitation module SE first compresses the mean vector z in the channel dimension, which can be expressed by the formula:

[0064]

[0065] Where T represents the number of frames for the input features, h t Let z represent all features input to this module in frame t; the mean vector z is then input to the excitation module to calculate the weight percentage for each channel. The excitation operation can be described as follows:

[0066] s=σ(ω2f(ω1z+b1)+b2) (2)

[0067] Where σ(·) represents the Sigmoid activation function, f(·) represents the nonlinear layer, ω1 and ω2 represent the weight parameters in the nonlinear layer, and b1 and b2 represent the bias parameters; the resulting vector s encompasses the weight coefficients of each channel, which are between 0 and 1. Therefore, the weight coefficients s of each channel are... c Multiply by the original channel vector h c The final channel vector can then be obtained.

[0068]

[0069] (4) The softmax classifier is used at the back end of the model to classify the obtained 192-dimensional voiceprint embedding. The AAM-loss loss is reduced to achieve the purpose of model optimization and iteration, thereby training a discriminative speaker voiceprint embedding representation.

[0070] Step (32) uses the Encoder module from the Transformer. The Transformer relies on a multi-head attention mechanism and has a strong data modeling capability, enabling it to capture the global contextual information of a sequence. The traditional Transformer consists of an encoder and a decoder, and is typically used for sequence-to-sequence tasks, such as machine translation. This paper only uses the encoder module for global speaker recognition information extraction. Its basic structure is as follows: Figure 3 As shown,

[0071] The encoder consists of two stacked multi-head attention mechanism modules, each followed by a feedforward neural network. Residual connections are also introduced, directly adding the encoder's input features to the output vector of the attention mechanism. In this embodiment, to prevent overfitting, the model is simplified, with dropout set to 0.3.

[0072] Specifically, in the diagram, Multi-head attention represents a multi-head attention mechanism, Add represents the vector addition operation, Norm represents normalization, and *N represents the number of encoders introduced into the model. In this embodiment, the number of encoder layers N is set to 6, and the multi-head attention mechanism uses scaled dot product attention with 8 attention heads. Scaled dot product attention can be defined by the following formula:

[0073]

[0074] Where Q represents the query vector, K represents the searched vector, and V represents the content vector. In this embodiment, Q, K, and V are all 192-dimensional vectors; d k Let T denote the dimension of the vector; T represents the transpose; and softmax() is the normalization operation. Therefore, multi-head attention can be described as:

[0075] MH(Q,K,V)=Concat(head1,…,head h )ω o (5)

[0076] Where Concat(…,…,…) represents the aggregation operation; ω o For the parameter matrix; head1,…,head h The attention mechanism of each head is represented by formula (4).

[0077] Experimental Analysis:

[0078] The experiments used the publicly available corpora Voxceleb2 and Voxceleb1 as the training and testing sets for the proposed model, respectively. The training set was further divided into training and validation sets in a 98:2 ratio for model learning and optimization. Furthermore, the experiments compared the proposed model with ECAPA-TDNN as a baseline model, using the same environment and training iterations to compare their performance.

[0079] The experimental environment was an NVIDIA GeForce RTX 4090, with 100 training epochs, a batch size of 64, an initial learning rate of 0.01 and an epoch decay strategy, and SGD as the optimizer.

[0080] The speaker identification results are shown in Table 1:

[0081]

[0082] Table 1

[0083] This experiment compares the baseline model ECAPA-TDNN and the speaker recognition model proposed in this invention in a speaker identification task. This task involves 5994 speakers and 5994 categories. The classification performance on the VoxCeleb2 validation set shows that the speaker recognition model proposed in this invention outperforms the mainstream model ECAPA-TDNN in speaker recognition.

[0084] The speaker's identity verification results are shown in Table 2:

[0085]

[0086] Table 2

[0087] This paper compares the baseline model ECAPA-TDNN with the speaker recognition model proposed in this invention in the speaker identification task. The performance on the three test sets of the publicly available VoxCeleb1 dataset shows that the speaker recognition model proposed in this invention improves the performance of ECAPA-TDNN on two datasets, while its performance slightly declines on one dataset, but it still performs very well. Therefore, the speaker recognition model proposed in this invention improves the performance of the best model on the publicly available dataset.

[0088] The above content is readily understood by those skilled in the art. The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for training a speaker recognition model based on a deep convolutional encoder, characterized in that, The method includes: The audio data in the training set is segmented according to a preset duration and then augmented. Extract Mel-spectral features from each audio segment; A recognition model is constructed, which first extracts feature vectors containing local voiceprint dependencies from the Mel-spectral features, and then extracts voiceprint embeddings containing global voiceprint correlations from the feature vectors through a multi-head attention mechanism; the extraction of feature vectors containing local voiceprint dependencies from the Mel-spectral features includes: The first feature is obtained by performing an initial convolution operation on the Mel spectrum features through a one-dimensional dilated convolutional layer; The first feature is then processed by an attention residual module to obtain the second feature; The second feature is then processed by an attention residual module to obtain the third feature; The third feature is then processed by an attention residual module to obtain the fourth feature; The first feature, second feature, third feature and fourth feature are aggregated through a one-dimensional dilated convolutional layer to obtain aggregated features; The aggregated features are then processed by a max-pooling layer to extract voiceprint features. The voiceprint features are processed through a linear layer and a normalization layer to obtain a feature vector containing local voiceprint dependencies. The voiceprint embedding is classified using a loss function, and the recognition model is trained iteratively by reducing the loss, until the recognition model is finally obtained after training.

2. The method of claim 1, wherein, A multi-head attention mechanism is used to extract the voiceprint embedding, which includes global voiceprint correlation, from the feature vector, specifically as follows: The feature vector is input into a deep encoder, and a voiceprint embedding containing global voiceprint correlation is obtained through the learning of the multi-head attention mechanism in the deep encoder. The feature vector is divided into multiple vector segments according to the number of heads of the multi-head attention mechanism, and each vector segment is input into each multi-head attention mechanism to capture contextual information.

3. A speaker recognition method based on a deep convolutional encoder, characterized in that, The method includes: Receive the speaker's speech to be recognized; The speech to be recognized is segmented into segments according to a preset duration and data augmentation is performed; Extract Mel-spectral features from each audio segment; The Mel spectrum features are input into a pre-trained recognition model, which outputs the speaker identity of the speech to be recognized; wherein the recognition model is trained by the method according to any one of claims 1-2.

4. A speaker recognition model training device based on a deep convolutional encoder, characterized in that, The device includes: The preprocessing module is used to segment the audio data in the training set according to a preset duration and perform data augmentation. The feature extraction module is used to extract Mel-spectral features from each audio segment; The model building module is used to build a recognition model. The recognition model first extracts feature vectors containing local voiceprint dependencies from the Mel spectrum features, and then extracts voiceprint embeddings containing global voiceprint correlations from the feature vectors through a multi-head attention mechanism. The model building module includes: The first submodule is used to perform an initial convolution operation on the Mel spectrum features through a one-dimensional dilated convolutional layer to obtain the first feature; The second submodule is used to process the first feature through an attention residual module to obtain the second feature; The third submodule is used to process the second feature through an attention residual module to obtain the third feature; The fourth submodule is used to process the third feature through an attention residual module to obtain the fourth feature; The fifth submodule is used to aggregate the first feature, the second feature, the third feature and the fourth feature through a one-dimensional dilated convolutional layer to obtain aggregated features; The sixth submodule is used to extract voiceprint features from the aggregated features through a max-pooling layer; The seventh submodule is used to process the voiceprint features through a linear layer and a normalization layer to obtain a feature vector containing local voiceprint dependencies. The model training module is used to classify the voiceprint embedding using a loss function, and to train and iterate the recognition model by reducing the loss, and finally obtain the recognition model after training is completed.

5. The apparatus according to claim 4, characterized in that, The model building module includes an eighth sub-module: The eighth submodule is specifically used to input the feature vector into the deep encoder and obtain a voiceprint embedding containing global voiceprint correlation through the learning of the multi-head attention mechanism in the deep encoder; wherein, the feature vector is divided into multiple vector segments according to the number of heads of the multi-head attention mechanism, and each vector segment is input into each multi-head attention mechanism to capture context information.

6. A speaker recognition device based on a deep convolutional encoder, characterized in that, The device includes: A receiving unit is used to receive the speaker's speech to be recognized. The preprocessing unit is used to segment the speech to be recognized into segments according to a preset duration and perform data enhancement; The feature extraction unit is used to extract Mel-spectral features from each audio segment; The recognition unit is used to input the Mel spectrum features into a pre-trained recognition model and output the speaker identity of the speech to be recognized; wherein the recognition model is trained by the method according to any one of claims 1-2.

Images