Method and system for identifying large-scale speakers
By using dynamic full connection layer and dynamic class queue to store pseudo-identity center features in large-scale speaker identification tasks, the hardware resource constraints and training time are solved due to the single-hot encoding full connection layer, and the hardware resources are realized and save time.
Patent Information
- Application Number
- CN202310765307.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-27
- Publication Date
- 2024-12-27
AI Technical Summary
In the large-scale speaker recognition task, the number of parameters of the single-hot coded fully connected layer used for classification in the model increases linearly, resulting in the problem of hardware resource limitation and training time being too long.
The dynamic full connection layer is used to replace the single hot coded full connection layer, and the pseudo-identity center features are stored through the dynamic class queue, the number of parameters is reduced and the loss calculation is used using the dynamic class queue, and the parameter method is updated in combination with data enhancement and momentum.
It effectively reduces the use of hardware resources and training time, realizes savings in GPU memory and time, and is suitable for large-scale speaker recognition tasks.
Smart Images
Figure BDA0004305889840000022 
Figure BDA0004305889840000024 
Figure BDA0004305889840000025
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of speaker recognition, and in particular relates to a method and system for large-scale speaker recognition. Background Art
[0002] Speaker recognition is a biometric recognition technology that extracts and analyzes the speaker's voice to identify the individual. In recent years, deep neural networks (such as TDNN, ResNet, ECAPA-TDNN, and MFA-Conformer) have made great progress in speaker recognition tasks and continuously improved system performance.
[0003] For models applied to speaker recognition tasks, the x-vector network framework is usually used, which mainly includes four parts: frame-level feature learning module, pooling module, segment-level feature learning module and loss function. The segment-level module usually contains two fully connected layers. The first fully connected layer outputs a fixed-dimensional voiceprint feature vector for distinguishing between speakers in the inference stage. The second fully connected layer is mainly used to calculate the loss of labeled speakers. The parameter dimension of the fully connected layer to be trained is M×N, where M is the dimension of the voiceprint feature vector during model training and N is the number of speakers in the dataset. Using a large-scale speaker recognition dataset will improve system performance and make the model more generalizable. However, the large number of speakers will cause the number of parameters of the fully connected layer (i.e., the second fully connected layer) used for classification in the model to increase linearly. When the dataset contains millions or more speakers, the proportion of model parameters is mainly concentrated in the fully connected layer used for classification, and a single GPU cannot directly train the model. In addition, as the number of parameters in this layer increases, a lot of time will be spent on training the parameters of this layer, increasing the model training time. Summary of the invention
[0004] The purpose of this invention is to solve the problem of limited hardware resources and high training time in the training process of large-scale speaker recognition tasks. It proposes to use a dynamic fully connected layer method to replace the fully connected layer using one-hot encoding for classification, thereby reducing hardware resource usage and training time. Figure 1 shown.
[0005] The dynamic fully connected layer adopts the idea of contrastive learning and uses a dynamic class queue to store a subset of pseudo-identity centers for loss calculation. The specific implementation details are as follows:
[0006] 1. Audio reading. Load the original audio, randomly cut each audio segment to a fixed length, and if the audio length is less than the selected cut length, use 0 to fill or repeat the audio segment to make it complete.
[0007] 2. Feature extraction: Calculate the logarithmic Mel-spectrogram features of the original audio data.
[0008] 3. Data enhancement. Perform different data enhancements on audio features to generate sample pairs. Data enhancement methods include Figure 2 As shown, it consists of four parts:
[0009] 3.1) Pre-normalize the input data x, calculate the mean μ and variance σ in the time dimension, and subtract the mean from the input data and divide by the variance to get the pre-normalized value
[0010]
[0011] 3.2) Mixing foreground acoustic features, using normalized log-mel spectrograms as input. The mixing module mixes the past randomly selected input audio in a small proportion. The added audio becomes part of the background sound in the mixed audio, encouraging the model to learn an invariant representation of the foreground acoustic event sound. Since the audio features are logarithmic, the input is converted to linear values before the mixing calculation, and then converted back to logarithmic values again. For the i-th input x i The foreground acoustic features are mixed using the following formula to obtain
[0012]
[0013] Here x k is a mixed audio feature, λ is selected from the uniform distribution U(0.0,α), and usually the smaller the α value, the better the performance, such as 0.4. k It is randomly selected from a memory queue that stores past inputs. The queue size is larger than the training batch size and large enough to keep the data random.
[0014] 3.3) Random cropping adjustment: When the frequency domain dimension of the input spectrogram is F and the time domain dimension is T, the time domain is first expanded, for example, 1.5 times the original time domain, and the expanded part is filled with 0. Random sampling is performed in the expanded range as follows:
[0015]
[0016]
[0017] Here F C and T C are the frequency domain length and time domain length of the cropping area respectively, (h1,h2) and (w1,w2) are the ranges of the frequency domain and time domain cropping lengths respectively. is the floor function, min(·,·) is the minimum function, and U(·,·) is the uniform distribution function. Randomly select F in the spectrum containing the virtual boundary C and T C After the area is sized, the bicubic interpolation method is used to adjust it to the original frequency domain and time domain dimensions.
[0018] 3.4) Post-normalization: Calculate the mean and variance of the frames in the batch samples, and use the mean and variance to normalize the samples. Make each sample conform to the normal distribution to prevent the previous operation from causing statistical drift in the output.
[0019] 4. Training network training. The training network consists of a deep neural network (TDNN, ResNet, ECAPA-TDNN or MFA-Conformer, etc.), a pooling layer (attention statistical pooling, etc.) and a fully connected layer. Each sample can obtain a fixed-dimensional voiceprint feature vector through this network.
[0020] 5. Forward network forward propagation. The forward network has the same network structure as the training network. The parameters in the network are frozen and only forward propagation is performed. The parameters are updated using momentum update. The training network parameters are θ p , the forward network parameter is θ g , the parameter update formula is as follows:
[0021] θ g ←αθ g +(1-α)θ p
[0022] Among them, α∈[0,1).
[0023] 6. Dynamic class queue storage. Use a dynamic class queue to store the pseudo identity center features generated by the forward network. When there are enough identities in the training set, such as millions or even more speakers, the dynamic class queue size can be set to 1% of the number of speakers.
[0024] 7. Calculate logits. Calculate the inner product of the voiceprint feature vector generated by the training network and the pseudo-identity center feature in the dynamic class queue to obtain the corresponding logits. There are two types of inner products in logits, namely the inner product of positive sample pairs (the voiceprint feature vector and the pseudo-identity center feature label are the same) and the inner product of negative sample pairs (the voiceprint feature vector and the pseudo-identity center feature label are different).
[0025] 8. Calculate the loss and perform back propagation. Use loss functions such as AAM-Softmax to calculate the loss of one iteration and perform back propagation. The AAM-Softmax loss function allows the voiceprint feature vectors of positive and negative samples in the network to be distinguished. The AAM-Softmax loss is as follows:
[0026]
[0027] N is the training batch size during training, C is the dynamic class queue size, s is the scaling factor, and m is the spacing between different classes. i represents the i-th sample in the training batch, y i represents the label corresponding to the i-th sample in the training batch, j represents the j-th pseudo identity center in the dynamic class queue, and y j Represents the label corresponding to the jth pseudo identity center in the dynamic class queue. is the inner product of a pair of positive samples in a training batch, corresponding to is the inner product of negative sample pairs in a training batch.
[0028] The characteristics and beneficial effects of the present invention are:
[0029] The dynamic fully connected layer proposed in the present invention effectively solves the problems of limited hardware resources and high training time in the large-scale speaker recognition training stage. Compared with the traditional fully connected layer that uses one-hot encoding for classification, it uses a dynamic class queue to store a subset of pseudo-identity centers. The size of the dynamic class queue can be set to a value much smaller than the number of speakers to reduce the number of parameters of this layer, thereby saving GPU memory and training time. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 This is a dynamic fully connected layer method for large-scale speaker recognition proposed by the present invention.
[0031] Figure 2 yes Figure 1 The data augmentation method used in . DETAILED DESCRIPTION
[0032] The present invention proposes to use a dynamic fully connected layer to replace the fully connected layer that uses one-hot encoding for classification in traditional speaker recognition tasks to reduce hardware and time costs in large-scale speaker recognition tasks, which is further described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0033] The dynamic fully connected layer is used during the speaker recognition task training process, including the following steps:
[0034] 1. Audio reading. Use torchaudio.load method to read audio. Each audio segment is randomly cut to a fixed length of 2s. If the audio length is less than 2s, add 0 to 2s.
[0035] 2. Feature extraction: Calculate the 80-dimensional logarithmic Mel-spectrogram features, and randomly select 200 sample features for each training batch for iteration.
[0036] 3. Data enhancement. Perform different data enhancements on audio features to generate sample pairs. Data enhancement methods include Figure 2 As shown, it consists of four parts:
[0037] 3.1) Pre-normalize the input data x, calculate the mean μ and variance σ in the time dimension, and subtract the mean from the input data and divide by the variance to get the pre-normalized value
[0038]
[0039] 3.2) Mixing foreground acoustic features, using normalized log-mel spectrograms as input. The mixing module mixes the past randomly selected input audio in a small proportion. The added audio becomes part of the background sound in the mixed audio, encouraging the model to learn an invariant representation of the foreground acoustic event sound. Since the audio features are logarithmic, the input is converted to linear values before the mixing calculation, and then converted back to logarithmic values again. For the i-th input x i The foreground acoustic features are mixed using the following formula to obtain
[0040]
[0041] Here x k is a mixed audio feature, λ is selected from the uniform distribution U(0.0,α), usually the smaller the α value, the better the performance, here we choose 0.4. k It is randomly selected from a memory queue that stores past inputs. The queue size is larger than the training batch size and large enough to maintain data randomness. The queue size is set to store 2048 samples.
[0042] 3.3) Random cropping adjustment: When the frequency domain dimension of the input spectrogram is F and the time domain dimension is T, the time domain is first expanded to 1.5 times the original length, and the expanded part is padded with 0. Random sampling is performed in the expanded range as follows:
[0043]
[0044]
[0045] Here F C and T C are the frequency domain length and time domain length of the cropping area, respectively. (h1, h2) and (w1, w2) are the ranges of the frequency domain and time domain cropping lengths, respectively, and are both set to (0.6, 1.5). is the floor function, min(·,·) is the minimum function, and U(·,·) is the uniform distribution function. Randomly select F in the spectrum containing the virtual boundaryC and T C After the area is sized, the bicubic interpolation method is used to adjust it to the original frequency domain and time domain dimensions.
[0046] 3.4) Post-normalization: Calculate the mean and variance of the frames in the batch samples, and use the mean and variance to normalize the samples. Make each sample conform to the normal distribution to prevent the previous operation from causing statistical drift in the output.
[0047] 4. Training network training. The training network consists of a deep neural network (TDNN, ResNet, ECAPA-TDNN or MFA-Conformer, etc.), a pooling layer (attention statistical pooling, etc.) and a fully connected layer. Each sample can obtain a fixed-dimensional voiceprint feature vector through this network.
[0048] 5. Forward network forward propagation. The forward network has the same network structure as the training network. The parameters in the network are frozen and only forward propagation is performed. The parameters are updated using momentum update. The training network parameters are θ p , the forward network parameter is θ g , the parameter update formula is as follows:
[0049] θ g ←αθ g +(1-α)θ p
[0050] Among them, α∈[0,1), α is set to 0.999.
[0051] 6. Dynamic class queue storage. Use a dynamic class queue to store the pseudo identity center features generated by the forward network. When there are enough identities in the training set, such as millions or even more speakers, the dynamic class queue size can be set to 1% of the number of speakers.
[0052] 7. Calculate logits. Calculate the inner product of the voiceprint feature vector generated by the training network and the pseudo-identity center feature in the dynamic class queue to get the corresponding logits. Logits consists of two types of inner products, namely the inner product of positive sample pairs (the voiceprint feature vector and the pseudo-identity center feature label are the same) and the inner product of negative sample pairs (the voiceprint feature vector and the pseudo-identity center feature label are different).
[0053] 8. Calculate the loss and perform back propagation. Use loss functions such as AAM-Softmax to calculate the loss of one iteration and perform back propagation. The AAM-Softmax loss function allows the voiceprint feature vectors of positive and negative samples in the network to be distinguished. The AAM-Softmax loss is as follows:
[0054]
[0055] N is the training batch size during training, which is 200 in this example. C is the dynamic class queue size, which is 1% of the number of speakers in the training set in this example. s is the scaling factor, which is 32 in this example. m is the spacing between different classes, which is 0.3 in this example. i represents the i-th sample in the training batch, y i represents the label corresponding to the i-th sample in the training batch, j represents the j-th pseudo identity center in the dynamic class queue, and y j Represents the label corresponding to the jth pseudo identity center in the dynamic class queue. is the inner product of a pair of positive samples in a training batch, corresponding to is the inner product of negative sample pairs in a training batch.
[0056] The above is only a specific embodiment of the present invention, which obviously cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.
Claims
1. The present invention proposes a method and system for large-scale speaker recognition, characterized in that: Using the contrastive learning method, positive and negative sample pairs are selected and sent to the network for training, and a dynamic class queue is used to store features. By controlling the size of the dynamic class queue, the number of parameters of the fully connected layer can be reduced, thus saving memory and time during model training.
2. The method and system for large-scale speaker recognition according to claim 1, characterized in that Different data enhancement methods are used to generate positive and negative sample pairs for model training.
3. The method and system for large-scale speaker recognition according to claim 1, characterized in that The contrastive learning method is used, which includes two network training networks and a forward network. The two networks have the same structure, consisting of a deep neural network (TDNN, ResNet, ECAPA-TDNN or MFA-Conformer, etc.), a pooling layer (attention statistics pooling, etc.) and a fully connected layer. The parameters in the forward network are updated using the parameters in the training network using momentum update.
4. The method and system for large-scale speaker recognition according to claim 1, characterized in that A dynamic class queue is used to dynamically store a subset of pseudo-identity centers generated by the forward network without backpropagation.
5. The method and system for large-scale speaker recognition according to claim 1, characterized in that Logits are obtained by calculating the inner product of the voiceprint feature vector generated by the network and the pseudo identity center in the queue. Logits contains two types of inner products, namely the inner product of positive sample pairs (the voiceprint feature vector and the pseudo identity center feature label are the same) and the inner product of negative sample pairs (the voiceprint feature vector and the pseudo identity center feature label are different).
6. The method and system for large-scale speaker recognition according to claim 1, characterized in that The Softmax-based loss function is used to calculate the loss for model training.
7. The data enhancement method according to claim 2, characterized in that it comprises four types of data enhancement: 7.1) Pre-normalize the input data x, calculate the mean μ and variance σ in the time dimension, and subtract the mean from the input data and divide by the variance to get the pre-normalized value 7.2) Mixing foreground acoustic features, using normalized log-mel spectrograms as input. The mixing module mixes a small proportion of the input audio randomly selected in the past. The added audio becomes part of the background sound in the mixed audio, encouraging the model to learn an invariant representation of the foreground acoustic event sound. Since the audio features are logarithmic, the input is converted to linear values before the mixing calculation, and then converted back to logarithmic values again. For the i-th input x i The foreground acoustic features are mixed using the following formula to obtain Here x k is a mixed audio feature, and λ is selected from the uniform distribution U(0.0,α). k It is randomly selected from a memory queue that stores past inputs, and the queue size is larger than the training batch size. 7.3) Random cropping adjustment: When the frequency domain dimension of the input spectrogram is F and the time domain dimension is T, the time domain is first expanded, the expanded part is filled with 0, and random sampling is performed in the expanded range as follows: Here F C and T C are the frequency domain length and time domain length of the cropping area respectively, (h1,h2) and (w1,w2) are the ranges of the frequency domain and time domain cropping lengths respectively. is the floor function, min(·,·) is the minimum function, and U(·,·) is the uniform distribution function. Randomly select F in the spectrum containing the virtual boundary C and T C After the area is sized, the bicubic interpolation method is used to adjust it to the original frequency domain and time domain dimensions. 7.4) Post-normalization: Calculate the mean and variance of the frames in the batch samples, and use the mean and variance to normalize the samples. Make each sample conform to the normal distribution to prevent the previous operation from causing statistical drift in the output.