Speaker verification method and system based on transfer learning
By combining transfer learning and deep residual convolutional neural networks with channel attention mechanism, the problem of feature distribution offset in cross-channel voiceprint recognition is solved, and the accuracy and robustness of voiceprint recognition are improved, especially the recognition effect when there is insufficient data in the domain.
Patent Information
- Application Number
- CN202510851928.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-24
- Publication Date
- 2025-09-09
AI Technical Summary
Existing voiceprint recognition technology suffers from feature distribution offset problems in cross-channel scenarios, especially when given large-scale out-of-domain annotated data and a small amount of in-domain annotated data, the recognition accuracy and robustness are insufficient.
A transfer learning-based method is adopted to pre-train the voiceprint feature extraction model with large-scale out-of-domain data, and deep features are extracted through residual blocks and channel attention mechanisms. Combined with the gradient amplitude adaptive weight adjustment mechanism, a small amount of in-domain data is used for fine-tuning to optimize model performance.
It improves the accuracy and robustness of voiceprint recognition, solves the problem of feature distribution offset in cross-channel scenarios, and improves the recognition effect, especially when the domain data is scarce.
Smart Images
Figure CN120612946A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech recognition technology, and more particularly to a speaker verification method and system based on transfer learning. Background Art
[0002] At present, voiceprint recognition, as an important biometric recognition technology, has been widely used in identity authentication, security access control and other fields.
[0003] Existing voiceprint recognition technologies can be mainly divided into the following categories: (1) Template matching-based methods. This type of method extracts the acoustic features of speech (such as Mel-frequency cepstral coefficients MFCC, linear predictive coding LPC) to establish a voiceprint template, and then performs recognition by calculating the distance or similarity between the speech to be recognized and the template. However, this type of method is very sensitive to noise and channel changes and has poor robustness. (2) Gaussian mixture model (GMM)-based methods. The GMM method uses multiple Gaussian distributions to model the voiceprint feature distribution of each speaker, which can better adapt to the complexity of speech. However, the GMM method requires a large amount of training data to achieve good performance and has high computational complexity. (3) Deep learning-based methods. Deep learning models such as deep neural networks (DNN), convolutional neural networks (CNN), and recurrent neural networks (RNN) can automatically learn the deep features of speech and have strong robustness. Although deep learning-based voiceprint recognition models (such as x-vector systems) have achieved great success, they still face challenges in cross-channel scenarios. The differences in acoustic features between different channels will cause the model performance to degrade significantly.
[0004] Transfer learning is a machine learning method that aims to transfer knowledge learned on one task to another related task. In the field of voiceprint recognition, transfer learning can use large-scale, well-annotated out-of-domain data (for example, data recorded in a clear speech environment) to pre-train the model, and then use a small amount of in-domain data (for example, data recorded in a noisy or different channel environment) to fine-tune the model, thereby improving the model's performance in the domain. This method is particularly suitable when in-domain data is scarce.
[0005] Therefore, how to provide a speaker verification method based on transfer learning is an urgent problem that those skilled in the art need to solve. Summary of the Invention
[0006] In view of this, the present invention provides a speaker verification method and system based on transfer learning, which can effectively solve the problem of feature distribution offset in cross-channel scenarios, especially in scenarios where large-scale out-of-domain labeled data and a small amount of in-domain labeled data are given, thereby improving the accuracy and robustness of voiceprint recognition.
[0007] In order to achieve the above object, the present invention adopts the following technical solutions:
[0008] A speaker verification method based on transfer learning includes the following steps:
[0009] Build a voiceprint feature extraction model and pre-train it using out-of-domain annotated data;
[0010] Optimizing the pre-trained voiceprint feature extraction model;
[0011] The voiceprint feature extraction model that has been optimized and pre-trained is used to extract cross-channel voiceprint features of the speaker to be identified, thereby realizing cross-channel speaker verification.
[0012] Preferably, the specific structure of the voiceprint feature extraction model includes:
[0013] The voiceprint feature extraction model includes multiple residual blocks and a channel attention mechanism, wherein the residual block is used to extract deep features and alleviate the gradient vanishing problem, including multiple convolutional layers, batch normalization layers and ReLU activation functions;
[0014] The channel attention mechanism is used to adaptively learn the importance of each channel and enhance the model's attention to important channels. It is implemented using the SE module and generates the channel attention weight using the following formula:
[0015]
[0016] Where z c represents the global features of each channel, Represents the value of the input feature map at the spatial position (i, j) and channel c, H and W are the spatial dimensions of the feature map, and s c is the channel attention weight, W1, W2, b1, b2 are linear layer parameters, δ is the ReLU activation function, and σ is the sigmoid function.
[0017] Preferably, the specific process of pre-training the voiceprint feature extraction model using out-of-domain labeled data includes:
[0018] Use out-of-domain labeled data to train the voiceprint feature extraction model;
[0019] A loss function is constructed to optimize the model training results, with the goal of learning discriminative voiceprint feature representations.
[0020] Preferably, the specific process of optimizing the pre-trained voiceprint feature extraction model includes:
[0021] A mixed fine-tuning strategy of out-of-domain data and in-domain data is adopted, and an adaptive weight adjustment mechanism based on gradient amplitude is introduced to balance the impact of in-domain and out-of-domain data on the voiceprint feature extraction model, and optimize the performance of the voiceprint feature extraction model under the target channel.
[0022] Preferably, the specific processing process of adopting the mixed fine-tuning strategy of out-of-domain data and in-domain data includes:
[0023] Mixing out-of-domain data with in-domain data to fine-tune the pre-trained voiceprint feature extraction model;
[0024] An adaptive weight adjustment mechanism based on gradient amplitude is adopted to dynamically adjust the weights of in-domain and out-of-domain data in the loss function according to the gradient amplitude of in-domain and out-of-domain data during training.
[0025] Preferably, the specific processing process of the adaptive weight adjustment mechanism based on gradient amplitude includes:
[0026] Calculate the gradient amplitude of in-domain data and out-of-domain data on each batch;
[0027] According to the magnitude of the gradient, the weights of the in-domain data and out-of-domain data in the loss function are dynamically adjusted.
[0028] Preferably, the specific process of adjusting the weights of in-domain data and out-of-domain data in the loss function includes:
[0029] If the gradient amplitude of the in-domain data is larger, the weight of the in-domain data is increased; if the gradient amplitude of the out-domain data is larger, the weight of the out-domain data is increased.
[0030] Preferably, the specific processing process for implementing cross-channel speaker verification includes:
[0031] Input the two audio segments into the voiceprint feature extraction model and extract the corresponding two voiceprint feature vectors;
[0032] Calculate the cosine similarity of the two voiceprint feature vectors as the voiceprint matching score;
[0033] The score is compared with the preset threshold, and the comparison result is used to determine whether the two voiceprint features come from the same person.
[0034] The present invention also provides a speaker verification system based on transfer learning, comprising:
[0035] The model building module is used to build a voiceprint feature extraction model and pre-train the voiceprint feature extraction model using out-of-domain annotated data;
[0036] A model optimization module, used to optimize the pre-trained voiceprint feature extraction model;
[0037] The recognition module is used to extract the cross-channel voiceprint features of the speaker to be identified by using the voiceprint feature extraction model that has been optimized and pre-trained as described above, so as to realize cross-channel speaker verification.
[0038] As can be seen from the above technical solutions, compared with existing technologies, the present invention provides a speaker verification method and system based on transfer learning. Transfer learning can use large-scale, well-annotated out-of-domain data (for example, data recorded in a clear speech environment) to pre-train the model, and then use a small amount of in-domain data (for example, data recorded in a noisy or different channel environment) to fine-tune the model, thereby improving the model's performance in the domain. This method is particularly suitable for situations where in-domain data is scarce.
[0039] Based on this, the cross-channel voiceprint recognition model based on transfer learning proposed in the present invention can effectively solve the problem of feature distribution offset in cross-channel scenarios, especially in scenarios where large-scale out-of-domain labeled data and a small amount of in-domain labeled data are given, thereby improving the accuracy and robustness of voiceprint recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0041] Figure 1 This is an overall flow chart of a speaker verification method based on transfer learning provided by the present invention;
[0042] Figure 2 This is a block diagram of the structural principles of a speaker verification system based on transfer learning provided by the present invention. DETAILED DESCRIPTION
[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0044] See also Figure 1 As shown, the embodiment of the present invention discloses a speaker verification method based on transfer learning, comprising the following steps:
[0045] Build a voiceprint feature extraction model based on a deep residual convolutional neural network (DRCNN) and pre-train the model using out-of-domain annotated data. The out-of-domain annotated data can use the VoxCeleb1 and VoxCeleb2 datasets. These datasets contain rich speech data covering multiple languages and different real-world scenarios, providing sufficient information for the model to learn universal voiceprint feature representations.
[0046] Optimize the pre-trained voiceprint feature extraction model;
[0047] The optimized and pre-trained voiceprint feature extraction model is used to extract the cross-channel voiceprint features of the speaker to be identified, thereby realizing cross-channel speaker verification.
[0048] In a specific embodiment, the specific structure of the voiceprint feature extraction model includes:
[0049] The voiceprint feature extraction model includes multiple residual blocks and a channel attention mechanism. The residual blocks are used to extract deep features and alleviate the vanishing gradient problem. The model includes multiple convolutional layers (kernel size = 3x3), batch normalization layers, and ReLU activation functions. The convolutional layers extract speech features through convolution operations. The batch normalization layers normalize the data to accelerate model convergence and prevent overfitting. The ReLU activation function increases the model's nonlinear expression capabilities, enabling the model to learn more complex features. By stacking multiple residual blocks, deep features can be extracted while alleviating the vanishing gradient problem, ensuring effective model training.
[0050] The channel attention mechanism is used to adaptively learn the importance of each channel and enhance the model's attention to important channels. It is implemented using the SE module. The SE module performs global average pooling on the channel dimension of the input feature map to obtain the global feature z of each channel. c , the specific expression is:
[0051]
[0052] The global features are processed through two fully connected layers and activation functions to generate channel attention weights s c , the specific expression is:
[0053] s c =σ(W2δ(W1z c +b1)+b2) (2)
[0054] Where z c represents the global features of each channel, Represents the value of the input feature map at the spatial position (i, j) and channel c, H and W are the spatial dimensions of the feature map, and s cis the channel attention weight, W1, W2, b1, b2 are linear layer parameters, δ is the ReLU activation function, and σ is the sigmoid function;
[0055] Finally, the channel attention weight is multiplied with the original feature map to enhance important channels and suppress unimportant channels, thereby adaptively learning the importance of each channel and improving the model's ability to extract voiceprint features.
[0056] Specifically, each residual block can contain two convolutional layers, a batch normalization layer, and a ReLU activation function layer. The batch normalization layer and the ReLU activation function are used after the first convolutional layer, and only the batch normalization layer is used after the second convolutional layer. The input of this layer is then added to the output of the second convolutional layer and then passed through the ReLU activation function.
[0057] In a specific embodiment, the specific process of pre-training the voiceprint feature extraction model using out-of-domain annotated data includes:
[0058] Use out-of-domain labeled data to train the voiceprint feature extraction model;
[0059] Construct a loss function to optimize the model training results. The goal is to learn a discriminative voiceprint feature representation. The loss function can be a cross entropy loss function or a triplet loss function. The specific expression is:
[0060]
[0061]
[0062] Where N represents the number of samples, y i is the true label, is the model prediction probability, [x] + =max(0,x), α is the boundary value, x a 、x p 、x n They are anchor samples, positive samples, and negative samples respectively. is the feature vector of sample x, and the goal is to learn a discriminative voiceprint feature representation.
[0063] The cross-entropy loss function minimizes the difference between the model's predictions and the true labels, enabling the model to learn discriminative voiceprint feature representations. Triplet Loss, on the other hand, enhances the model's discriminative capabilities by constructing anchor samples, positive samples, and negative samples, ensuring that the feature vectors learned by the model satisfy a certain distance relationship in space. Specifically, the distance between the feature vectors of the positive sample and the anchor sample is less than the distance between the feature vectors of the negative sample and the anchor sample plus a margin value α. Through pre-training, the model can learn universal voiceprint feature representations, laying the foundation for subsequent application in the target domain.
[0064] In a specific embodiment, the specific process of optimizing the pre-trained voiceprint feature extraction model includes:
[0065] A mixed fine-tuning strategy of out-of-domain data and in-domain data is adopted, and an adaptive weight adjustment mechanism based on gradient amplitude is introduced to balance the impact of in-domain and out-of-domain data on the voiceprint feature extraction model, and optimize the performance of the voiceprint feature extraction model under the target channel.
[0066] In a specific embodiment, the specific processing process of using the mixed fine-tuning strategy of out-of-domain data and in-domain data includes:
[0067] Mixing out-of-domain data with in-domain data to fine-tune the pre-trained voiceprint feature extraction model;
[0068] An adaptive weight adjustment mechanism based on gradient amplitude is adopted to dynamically adjust the weights of in-domain and out-of-domain data in the loss function according to the gradient amplitude of in-domain and out-of-domain data during training.
[0069] In a specific embodiment, the specific processing process of the adaptive weight adjustment mechanism based on gradient amplitude includes:
[0070] Calculate the gradient magnitude g of the in-domain data and out-of-domain data in each batch in and g out ;
[0071] According to the magnitude of the gradient, the weights of the in-domain data and out-of-domain data in the loss function are dynamically adjusted. The specific expression is:
[0072]
[0073] Where w in is the data weight in the domain, w out is the weight of out-of-domain data, g in is the gradient amplitude of the data in the domain on each batch, g out It is the gradient amplitude of the out-of-domain data on each batch, g1 corresponds to the in-domain data, g2 corresponds to the out-of-domain data, and β is the weight adjustment coefficient.
[0074] In a specific embodiment, the specific process of adjusting the weights of in-domain data and out-of-domain data in the loss function includes:
[0075] If the gradient amplitude of the in-domain data is larger, the weight of the in-domain data is increased; if the gradient amplitude of the out-domain data is larger, the weight of the out-domain data is increased.
[0076] When the gradient amplitude of the in-domain data is large, it means that the in-domain data is more important to the current training of the model. In this case, the weight w of the in-domain data is increased. in , increasing its influence on model updating; on the contrary, when the gradient amplitude of out-of-domain data is large, increasing the weight w of out-of-domain data out In this way, the impact of in-domain and out-of-domain data on the model is balanced, the performance of the model in the target channel is optimized, and the problem of cross-channel voiceprint feature distribution differences is solved.
[0077] In a specific embodiment, the specific process of implementing cross-channel speaker verification includes:
[0078] Input two audio segments into the voiceprint feature extraction model and extract the corresponding two voiceprint feature vectors
[0079] and
[0080] Calculate the cosine similarity of the two voiceprint feature vectors as the voiceprint matching score. The specific expression is:
[0081]
[0082] Compare the score with the preset threshold and determine whether the two voiceprint features come from the same person based on the comparison result. The specific expression is:
[0083]
[0084] Specifically, the closer the cosine similarity value is to 1, the more similar the two vectors are, meaning the more likely the two audio clips are from the same speaker. Conversely, the closer the value is to 0, the more likely the two audio clips are from different speakers. Cross-channel speaker verification is achieved by setting a threshold and then determining whether two voiceprint features are from the same person based on the preset threshold.
[0085] See also Figure 2 As shown, an embodiment of the present invention further provides a system for a speaker verification method based on transfer learning using any one of the above embodiments, including:
[0086] The model building module is used to build a voiceprint feature extraction model and pre-train the voiceprint feature extraction model using out-of-domain annotated data;
[0087] Model optimization module, used to optimize the pre-trained voiceprint feature extraction model;
[0088] The recognition module is used to extract the cross-channel voiceprint features of the speaker to be identified using the voiceprint feature extraction model that has been optimized and pre-trained as described above, thereby realizing cross-channel speaker verification.
[0089] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.
[0090] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A speaker verification method based on transfer learning, characterized in that: The following steps are involved: Build a voiceprint feature extraction model and pre-train it using out-of-domain annotated data; Optimizing the pre-trained voiceprint feature extraction model; The voiceprint feature extraction model that has been optimized and pre-trained is used to extract cross-channel voiceprint features of the speaker to be identified, thereby realizing cross-channel speaker verification.
2. A speaker verification method based on transfer learning according to claim 1, characterized in that: The specific structure of the voiceprint feature extraction model includes: The voiceprint feature extraction model includes multiple residual blocks and a channel attention mechanism, wherein the residual block is used to extract deep features and alleviate the gradient vanishing problem, including multiple convolutional layers, batch normalization layers and ReLU activation functions; The channel attention mechanism is used to adaptively learn the importance of each channel and enhance the model's attention to important channels. It is implemented using the SE module and generates the channel attention weight using the following formula: Where z c represents the global features of each channel, Represents the value of the input feature map at the spatial position (i, j) and channel c, H and W are the spatial dimensions of the feature map, and s c is the channel attention weight, W1, W2, b1, b2 are linear layer parameters, δ is the ReLU activation function, and σ is the sigmoid function.
3. The speaker verification method based on transfer learning according to claim 1, characterized in that The specific process of pre-training the voiceprint feature extraction model using out-of-domain annotated data includes: Use out-of-domain labeled data to train the voiceprint feature extraction model; A loss function is constructed to optimize the model training results, with the goal of learning discriminative voiceprint feature representations.
4. The speaker verification method based on transfer learning according to claim 3, characterized in that The specific process of optimizing the pre-trained voiceprint feature extraction model includes: A mixed fine-tuning strategy of out-of-domain data and in-domain data is adopted, and an adaptive weight adjustment mechanism based on gradient amplitude is introduced to balance the impact of in-domain and out-of-domain data on the voiceprint feature extraction model, and optimize the performance of the voiceprint feature extraction model under the target channel.
5. The speaker verification method based on transfer learning according to claim 4, characterized in that: The specific processing process of using the mixed fine-tuning strategy of out-of-domain data and in-domain data includes: Mixing out-of-domain data with in-domain data to fine-tune the pre-trained voiceprint feature extraction model; An adaptive weight adjustment mechanism based on gradient amplitude is adopted to dynamically adjust the weights of in-domain and out-of-domain data in the loss function according to the gradient amplitude of in-domain and out-of-domain data during training.
6. The speaker verification method based on transfer learning according to claim 4, characterized in that The specific processing process of the adaptive weight adjustment mechanism based on gradient amplitude includes: Calculate the gradient amplitude of in-domain data and out-of-domain data on each batch; According to the magnitude of the gradient, the weights of the in-domain data and out-of-domain data in the loss function are dynamically adjusted.
7. The speaker verification method based on transfer learning according to claim 6, characterized in that: The specific process of adjusting the weights of in-domain data and out-of-domain data in the loss function includes: If the gradient amplitude of the in-domain data is larger, the weight of the in-domain data is increased; if the gradient amplitude of the out-domain data is larger, the weight of the out-domain data is increased.
8. The speaker verification method based on transfer learning according to claim 6, characterized in that: The specific process of implementing cross-channel speaker verification includes: Input the two audio segments into the voiceprint feature extraction model and extract the corresponding two voiceprint feature vectors; Calculate the cosine similarity of the two voiceprint feature vectors as the voiceprint matching score; The score is compared with the preset threshold, and the comparison result is used to determine whether the two voiceprint features come from the same person.
9. A system using the speaker verification method based on transfer learning according to any one of claims 1 to 8, characterized in that: include: The model building module is used to build a voiceprint feature extraction model and pre-train the voiceprint feature extraction model using out-of-domain annotated data; A model optimization module, used to optimize the pre-trained voiceprint feature extraction model; The recognition module is used to extract the cross-channel voiceprint features of the speaker to be identified by using the voiceprint feature extraction model that has been optimized and pre-trained as described above, so as to realize cross-channel speaker verification.