Speaker recognition method and device, equipment and storage medium
By selecting and pruning Transformer layers in a self-supervised speech pre-training model, and combining hierarchical learning rate and target loss function optimization, a simplified speaker recognition model is constructed, which solves the problems of large model parameters and high computational cost, and achieves efficient speaker recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-23
- Publication Date
- 2026-03-13
AI Technical Summary
Existing self-supervised speech pre-training models suffer from problems such as a large number of parameters, high computational cost, uneven information content in different Transformer layers, and loss of low-level information due to uniform learning rate updates in speaker recognition, making them difficult to deploy effectively in resource-constrained environments.
By selecting target Transformer layers that meet the preset optimal performance conditions from the initial speech pre-trained model, pruning unnecessary layers, combining them with the target speaker recognition model, and optimizing the speaker recognition model using a hierarchical learning rate strategy and a target loss function, a simplified target speaker recognition model is constructed.
While reducing computing costs, it maintains or improves the accuracy and robustness of speaker recognition, making it suitable for deployment on mobile terminals and edge devices.
Smart Images
Figure CN121662054A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a speaker recognition method, apparatus, device, and storage medium. Background Technology
[0002] Speaker recognition is a technology that automatically identifies a speaker by utilizing the stable individual differences inherent in speech signals. It is mainly divided into two categories: speaker verification ("Are you a certain person?") and speaker identification ("Who are you?"). It can be further divided into text-related forms with fixed commands and text-free forms with completely free speech content. Speaker recognition has significant application value in various fields such as mobile terminals, smart homes, and financial risk control.
[0003] In recent years, self-supervised speech pre-training models, represented by wav2vec 2.0, HuberT, and WavLM, have gradually become an effective paradigm for speaker recognition due to their ability to learn general representations with both semantic and acoustic information from large-scale unlabeled speech. However, these models have significant shortcomings: First, speech pre-training models consist of convolutional encoders and multiple layers of Transformers, resulting in a large number of parameters. Full parameter fine-tuning significantly increases memory usage and computational overhead, making them difficult to deploy in resource-constrained environments. Second, different Transformer layers carry significantly different amounts of speaker information. Higher layers focus on semantic modeling, while lower layers retain more identity-related acoustic and physiological features. The traditional approach of "using only the features from the last layer" is not optimal. Third, simultaneously updating all layers with a uniform learning rate can easily destroy the personalized information learned by lower layers during fine-tuning, preventing the full realization of the advantages of pre-training.
[0004] In conclusion, accurately identifying speakers while reducing computational costs is a pressing technical problem that needs to be solved. Summary of the Invention
[0005] In view of this, the purpose of this invention is to provide a speaker recognition method, apparatus, device, and storage medium that can accurately identify speakers and reduce computational costs. The specific solution is as follows:
[0006] Firstly, this application provides a speaker recognition method, including:
[0007] Obtain an initial speaker speech dataset, and input several initial speaker speech data from the initial speaker speech dataset into an initial speech pre-training model;
[0008] From each Transformer layer of the initial speech pre-trained model, a target Transformer layer that meets the preset optimal performance condition is determined, and each Transformer layer in the initial speech pre-trained model that is located after the target Transformer layer is pruned, so as to determine the target speech pre-trained model based on the pruned model.
[0009] An initial speaker recognition model is constructed based on the target speech pre-trained model and the target voiceprint recognition model. The initial speaker recognition model is trained based on several initial speaker speech data in the initial speaker speech dataset. During the training process, the initial speaker recognition model is optimized based on a preset hierarchical learning rate strategy and a target loss function to obtain the target speaker recognition model.
[0010] The target speaker recognition model is used to perform speaker recognition on the speech data of the speaker to be identified.
[0011] Optionally, determining the target Transformer layer that satisfies the preset optimal performance condition from each Transformer layer of the initial speech pre-trained model includes:
[0012] Obtain the frame-level features output by each Transformer layer of the initial speech pre-trained model, corresponding to the initial speaker's speech data;
[0013] For any Transformer layer, average pooling is performed on each frame-level feature in the time dimension to obtain each feature vector, and the cosine similarity between any two feature vectors is calculated. Based on each cosine similarity, a target curve corresponding to the Transformer layer is constructed; the target curve is a DET curve or a ROC curve.
[0014] Based on the target curve corresponding to each Transformer layer, determine the equal error rate of each Transformer layer;
[0015] The Transformer layer with the lowest error rate is determined as the target Transformer layer that satisfies the preset optimal performance condition.
[0016] Optionally, training the initial speaker recognition model based on several initial speaker speech data in the initial speaker speech dataset includes:
[0017] Data augmentation processing is performed on several initial speaker speech data in the initial speaker speech dataset to obtain several target speaker speech data, and a target speaker speech dataset is constructed based on each of the target speaker speech data;
[0018] The initial speaker recognition model is trained using the speech data of each target speaker in the target speaker speech dataset.
[0019] Optionally, optimizing the initial speaker recognition model based on a preset hierarchical learning rate strategy and a target loss function to obtain the target speaker recognition model includes:
[0020] The learning rate of the target Transformer layer in the target speech pre-training model is set to the first learning rate.
[0021] According to the preset hierarchical learning rate strategy and the first learning rate, the second learning rate is set for each Transformer layer in the target speech pre-trained model that is located before the target Transformer layer.
[0022] The target learning rate is determined for each Transformer layer in the target speech pre-training model based on the first learning rate and the second learning rate, and the initial speaker recognition model is optimized based on the target learning rate and the target loss function to obtain the target speaker recognition model.
[0023] The preset hierarchical learning rate strategy is used to control the second learning rate and the number of layers corresponding to each Transformer layer before the target Transformer layer to be positively correlated.
[0024] Optionally, optimizing the initial speaker recognition model based on the target learning rate and the target loss function to obtain the target speaker recognition model includes:
[0025] Using the AdamW optimizer, the target speech pre-trained model in the initial speaker recognition model is optimized based on the Warmup policy, cosine annealing policy and the target learning rate to obtain the optimized speech pre-trained model.
[0026] Using the AdamW optimizer, the target voiceprint recognition model in the initial speaker recognition model is optimized based on the AAM-Softmax loss function to obtain the optimized voiceprint recognition model;
[0027] The target speaker recognition model is determined based on the optimized speech pre-training model and the optimized voiceprint recognition model.
[0028] Optionally, after optimizing the initial speaker recognition model based on a preset hierarchical learning rate strategy and a target loss function to obtain the target speaker recognition model, the method further includes:
[0029] The target speaker recognition model is used as the target teacher model, and the target student model corresponding to the target teacher model is generated using knowledge distillation technology.
[0030] The target student model is deployed to the target mobile device or the target edge device so that the target mobile device or the target edge device can use the target speaker recognition model to perform speaker recognition on the speech data of the speaker to be recognized.
[0031] Optionally, the step of using the target speaker recognition model to perform speaker recognition on the speech data of the speaker to be recognized includes:
[0032] Using the optimized speech pre-training model in the target speaker recognition model, the speech data of the speaker to be recognized is mapped into a corresponding frame-level sequence, and the frame-level sequence is input into the optimized voiceprint recognition model in the target speaker recognition model;
[0033] Using the optimized voiceprint recognition model in the target speaker recognition model, a target embedding vector is generated based on the frame-level sequence;
[0034] The target embedding vector is scored using cosine similarity, and speaker recognition results are generated based on the scoring results.
[0035] Secondly, this application provides a speaker recognition device, comprising:
[0036] The dataset acquisition module is used to acquire an initial speaker speech dataset and input several initial speaker speech data from the initial speaker speech dataset into an initial speech pre-training model;
[0037] The model pruning module is used to determine the target Transformer layer that meets the preset optimal performance conditions from each Transformer layer of the initial speech pre-training model, and to prune each Transformer layer in the initial speech pre-training model that is located after the target Transformer layer, so as to determine the target speech pre-training model based on the pruned model.
[0038] The model optimization module is used to construct an initial speaker recognition model based on the target speech pre-trained model and the target voiceprint recognition model, and to train the initial speaker recognition model based on several initial speaker speech data in the initial speaker speech dataset. During the training process, the initial speaker recognition model is optimized based on a preset hierarchical learning rate strategy and a target loss function to obtain the target speaker recognition model.
[0039] The speaker recognition module is used to perform speaker recognition on the speech data of the speaker to be recognized using the target speaker recognition model.
[0040] Thirdly, this application provides an electronic device, comprising:
[0041] Memory, used to store computer programs;
[0042] A processor is used to execute the computer program to implement the aforementioned speaker recognition method.
[0043] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned speaker recognition method.
[0044] In this application, firstly, an initial speaker speech dataset is obtained, and several initial speaker speech data from the initial speaker speech dataset are input into an initial speech pre-training model. Then, a target Transformer layer satisfying a preset optimal performance condition is determined from each Transformer layer of the initial speech pre-training model, and each Transformer layer in the initial speech pre-training model located after the target Transformer layer is pruned to determine the target speech pre-training model based on the pruned model. Subsequently, an initial speaker recognition model is constructed based on the target speech pre-training model and the target speaker recognition model, and the initial speaker recognition model is trained based on several initial speaker speech data from the initial speaker speech dataset. During training, the initial speaker recognition model is optimized based on a preset hierarchical learning rate strategy and a target loss function to obtain the target speaker recognition model. Finally, the target speaker recognition model is used to perform speaker recognition on the speaker speech data to be recognized. As can be seen from the above, this application first obtains an initial speaker speech dataset, inputs the speech data from the initial speaker speech dataset into an initial speech pre-training model, selects a target Transformer layer that meets the preset optimal performance conditions from the initial speech pre-training model, and prunes the Transformer layer after the target Transformer layer to obtain a simplified target speech pre-training model; then, it combines the target speaker recognition model to construct an initial speaker recognition model, trains it using the initial speaker speech dataset, and optimizes the model parameters during the training process through a preset hierarchical learning rate strategy and a target loss function, finally forming the target speaker recognition model; finally, it uses the target speaker recognition model to perform speaker recognition on the speech data of the speaker to be recognized. In this way, on the one hand, this application obtains a simplified target speech pre-training model by selecting the target Transformer layer with the best speaker discrimination performance from the initial speech pre-training model and pruning the Transformer layer, forming a simplified sub-network front end, which effectively reduces the number of parameters and computational complexity while maintaining or even improving the overall recognition performance, achieving a dual optimization of performance and efficiency. On the other hand, this application optimizes the initial speaker recognition model by pre-setting a hierarchical learning rate strategy and a target loss function, which balances recognition performance and deployment efficiency, and improves the accuracy and robustness of the model in speaker recognition. Attached Figure Description
[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0046] Figure 1 A flowchart of a speaker recognition method provided in this application;
[0047] Figure 2 A specific target speaker recognition model architecture diagram is provided for this application;
[0048] Figure 3 This application provides a schematic diagram of the structure of a speaker recognition device;
[0049] Figure 4 This application provides a structural diagram of an electronic device. Detailed Implementation
[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0051] Speaker recognition is a technology that automatically identifies a speaker by utilizing the stable individual differences inherent in speech signals. It has significant applications in mobile terminals, smart homes, financial risk control, and many other fields. In recent years, self-supervised speech pre-training models, such as wav2vec 2.0, HuberT, and WavLM, have gradually become an effective paradigm for speaker recognition due to their ability to learn general representations with both semantic and acoustic information from large-scale unlabeled speech. However, these models have significant drawbacks: First, speech pre-training models consist of convolutional encoders and multiple layers of Transformers, resulting in a large number of parameters. Fine-tuning all parameters significantly increases memory usage and computational overhead, making them difficult to deploy in resource-constrained environments. Second, different Transformer layers carry significantly different amounts of speaker information; higher layers focus on semantic modeling, while lower layers retain more identity-related acoustic and physiological features. The traditional approach of "using only the features from the last layer" is not optimal. Third, simultaneously updating all layers with a uniform learning rate can easily destroy the personalized information learned by lower layers during fine-tuning, preventing the full realization of the advantages of pre-training. Therefore, this application provides a speaker recognition scheme that can accurately identify speakers and reduce computational costs.
[0052] See Figure 1 As shown in the figure, an embodiment of the present invention discloses a speaker recognition method, which may include:
[0053] Step S11: Obtain the initial speaker speech dataset and input several initial speaker speech data from the initial speaker speech dataset into the initial speech pre-training model.
[0054] In this embodiment, a small initial speaker speech dataset can be independently extracted from the training corpus, for example, containing about 1,000 speech pairs, in which the number of positive and negative sample pairs is roughly balanced.
[0055] It should be noted that the initial speech pre-training model can be wav2vec 2.0, HuBERT, UniSpeech-SAT, WavLM, etc., and WavLM will be used in this embodiment. Then, the initial speaker speech data from the initial speaker speech dataset can be used as input data into the initial speech pre-training model WavLM, and the speaker discrimination ability of WavLM's 24-layer Transformer is evaluated layer by layer.
[0056] Step S12: Determine the target Transformer layer that meets the preset optimal performance condition from each Transformer layer of the initial speech pre-training model, and prune each Transformer layer in the initial speech pre-training model that is located after the target Transformer layer, so as to determine the target speech pre-training model based on the pruned model.
[0057] In this embodiment, the target Transformer layer that meets the preset optimal performance condition is determined from each Transformer layer of the initial speech pre-training model. The specific process may include: first, obtaining the frame-level features output by each Transformer layer of the initial speech pre-training model corresponding to each initial speaker's speech data; then, for any Transformer layer, performing average pooling on each frame-level feature in the time dimension to obtain each feature vector, and calculating the cosine similarity between any two feature vectors, and constructing a target curve corresponding to the Transformer layer based on each cosine similarity; the target curve is a DET curve or a ROC curve; subsequently, determining the equal error rate of each Transformer layer based on the target curve corresponding to each Transformer layer; finally, determining the Transformer layer with the smallest equal error rate as the target Transformer layer that meets the preset optimal performance condition.
[0058] Specifically, the frame-level features corresponding to the initial speaker's speech data are obtained from the output of each Transformer layer of the initial speech pre-trained model. Where T is the time step and D is the feature dimension. This represents the frame-level feature matrix output by the l-th Transformer layer. Then, a general frame-level feature matrix is generated for the output of any Transformer layer. Average pooling is performed over the time dimension to obtain a fixed-length feature vector. This serves as the speaker feature representation for the Transformer layer. Then, cosine similarity is used to score the frame-level features of all initial speaker speech data pairs. Specifically, the cosine similarity between any two feature vectors can be calculated, and a DET (Detection Error Tradeoff Curve) or ROC (Receiver Operating Characteristic Curve) curve corresponding to the Transformer layer is plotted based on each cosine similarity. The equal error rate (EER) corresponding to the Transformer layer is then calculated. By comparing the EERs of all Transformer layers, the target Transformer layer with the lowest EER is found, such as M, for example, the 10th layer.
[0059] Next, the Transformer layers following the target Transformer layer in the initial speech pre-training model can be pruned. Specifically, layers M+1 to 24 are removed, leaving only the convolutional encoder and the first M Transformer layers as the speech pre-training front end, thus obtaining the target speech pre-training model. Practical results show that this embodiment can reduce the number of parameters involved in training and inference by more than 50% with almost no loss or even a slight improvement in recognition performance, significantly reducing GPU memory usage and latency.
[0060] Step S13: Construct an initial speaker recognition model based on the target speech pre-trained model and the target voiceprint recognition model, and train the initial speaker recognition model based on several initial speaker speech data in the initial speaker speech dataset. During the training process, optimize the initial speaker recognition model based on a preset hierarchical learning rate strategy and a target loss function to obtain the target speaker recognition model.
[0061] In this embodiment, the voiceprint recognition model can be selected from TDNN (Time Delay Neural Network), ECAPA-TDNN (Extended Context-Aware Parallel Aggregation TDNN), etc., and ECAPA-TDNN will be used as the example in this embodiment. A two-stage architecture of WavLM sub-network front-end and ECAPA-TDNN back-end is adopted to construct the initial speaker recognition model.
[0062] It should be noted that the training of the initial speaker recognition model based on several initial speaker speech data in the initial speaker speech dataset can specifically include: firstly, performing data augmentation processing on several initial speaker speech data in the initial speaker speech dataset to obtain several target speaker speech data, and constructing a target speaker speech dataset based on each of the target speaker speech data; then, using each of the target speaker speech data in the target speaker speech dataset to train the initial speaker recognition model.
[0063] Specifically, in this embodiment, various data augmentation techniques can be used, such as adding MUSAN noise and music, using RIR to simulate room reverberation, and applying time and frequency masking to the spectrum, to augment several initial speaker speech data in the initial speaker speech dataset, thereby obtaining several target speaker speech data. A target speaker speech dataset is then constructed based on these target speaker speech data to train the initial speaker recognition model, improving the model's robustness to different environments and devices. In this embodiment, the target speaker speech data can be input into the initial speaker recognition model. The pruned WavLM subnetwork maps the original waveforms corresponding to the target speaker speech data into high-dimensional frame-level sequences of variable duration. These frame-level sequences not only contain speech content and contextual information but also retain rich speaker-related acoustic information. The frame-level sequence is then input into the ECAPA-TDNN network. The network leverages multi-scale convolutional Res2Block, channel attention SE-Block, and dilated convolutional structures to enhance its ability to model speech features at different time scales. Attentional statistical pooling aggregates mean and variance information over time, automatically assigning higher weights to more discriminative frames, resulting in speaker embedding vectors with stronger discriminative power and better noise resistance. During training, after obtaining the embedding vectors, AAM-Softmax (Additive Angular Margin Softmax) loss can be applied to make intra-class embeddings more compact and inter-class margins larger.
[0064] In this embodiment, a layer-wise learning rate (LLR) strategy can be applied to each Transformer layer of the target speech pre-trained model WavLM to explicitly distinguish the roles of different Transformer layers in speaker information and task adaptation. In one specific implementation, the process may include: first, setting the learning rate of the target Transformer layer in the target speech pre-trained model as a first learning rate; then, setting second learning rates for each Transformer layer preceding the target Transformer layer in the target speech pre-trained model according to a preset layer-wise learning rate strategy and the first learning rate; and then determining the target learning rate for each Transformer layer in the target speech pre-trained model based on the first and second learning rates; wherein the preset layer-wise learning rate strategy is used to control a positive correlation between the second learning rate of each Transformer layer preceding the target Transformer layer and the number of layers.
[0065] Specifically, the target speech pre-trained model currently retains M Transformer layers, such as 10 layers. First, the learning rate of the highest layer, i.e., the target Transformer layer M, is set as the baseline learning rate, the first learning rate LR. Then, the second learning rate is determined sequentially for each Transformer layer preceding the target Transformer layer, the m-th (…). The learning rate for layer ) is shown below:
[0066] ;
[0067] in, This is used to control the rate of decrease in the learning rate, so that the learning rate of the lower Transformer layers closer to the input is smaller and the parameter updates are slower, which is beneficial to retain the personalized representations in the original pre-training; the learning rate of the higher Transformer layers closer to the output is larger, which is convenient for quickly adapting to the downstream speaker recognition task.
[0068] In another specific implementation, the Transformer layers in the target speech pre-trained model can be divided into several groups, such as the first group consisting of the bottom 1 to 3 Transformer layers, the second group consisting of the middle 4 to 7 Transformer layers, and the third group consisting of the top 8 to 10 Transformer layers. Then, different learning rates and freeze strategies are set for each group: the bottom layer can be completely frozen or use a very small learning rate, the middle layer can be moderately adjusted, and the top layer can be fully updated; it is even possible to incorporate the EER of each layer in the initial layer selection phase or the gradient norm during training into the learning rate design, so that layers with greater discrimination contribution or more sensitive gradients receive higher learning rates, thus constructing a layered learning rate (LLR) based on layer importance adaptation.
[0069] It should be noted that the process of optimizing the initial speaker recognition model based on the target learning rate and target loss function to obtain the target speaker recognition model can include: firstly, using the AdamW optimizer, the target speech pre-trained model in the initial speaker recognition model is optimized based on the Warmup strategy, cosine annealing strategy, and the target learning rate to obtain the optimized speech pre-trained model; simultaneously, using the AdamW optimizer, the target voiceprint recognition model in the initial speaker recognition model is optimized based on the AAM-Softmax loss function to obtain the optimized voiceprint recognition model; then, the target speaker recognition model is determined based on the optimized speech pre-trained model and the optimized voiceprint recognition model.
[0070] Specifically, in this embodiment, the AdamW optimizer, based on the Warmup strategy, cosine annealing strategy, and the target learning rate of each Transformer layer can be used to optimize each Transformer layer of the target speech pre-trained model in the initial speaker recognition model, to obtain an optimized speech pre-trained model. LLR can achieve a better balance between "preserving the original speaker information structure" and "adapting to downstream voiceprint tasks" while ensuring training stability, thereby improving the convergence speed and final performance of the overall system. Simultaneously, in this embodiment, the AdamW optimizer can be used to optimize the target voiceprint recognition model in the initial speaker recognition model based on the AAM-Softmax loss function, to obtain an optimized voiceprint recognition model. Then, the final target speaker recognition model can be determined based on the optimized speech pre-trained model and the optimized voiceprint recognition model. It can be seen that, through the hierarchical learning rate strategy in this embodiment, the roles of high-level "task adaptation" and low-level "personality preservation" can be explicitly distinguished. The top-down decreasing learning rate design can maximize the protection of low-level speaker information during fine-tuning, while enhancing the adaptability of high-level layers to downstream tasks.
[0071] Step S14: Use the target speaker recognition model to perform speaker recognition on the speech data of the speaker to be recognized.
[0072] It should be noted that, to balance performance and deployment overhead, knowledge distillation techniques can be introduced on top of the target speaker recognition model. This high-performance target speaker recognition model can be used as the teacher network, and corresponding student models can be determined. Embedding or posterior-like distillation can be performed on the lightweight TDNN / ResNet student network, allowing the student model to approximate the recognition performance of the teacher model as closely as possible while significantly reducing the number of parameters and computational cost, thus making it more suitable for deployment on mobile or edge devices. The specific process may include: first, using the target speaker recognition model as the target teacher model, and using knowledge distillation to generate the corresponding target student model; then, deploying the target student model to the target mobile or edge device, so that the target mobile or edge device can use the target speaker recognition model to perform speaker recognition on the speech data of the speaker to be recognized.
[0073] In this embodiment, the speaker recognition model is used to perform speaker recognition on the speech data of the speaker to be recognized. The specific process may include: first, using the optimized speech pre-training model in the target speaker recognition model, the speech data of the speaker to be recognized is mapped to a corresponding frame-level sequence, and the frame-level sequence is input into the optimized voiceprint recognition model in the target speaker recognition model; then, using the optimized voiceprint recognition model in the target speaker recognition model, a target embedding vector is generated based on the frame-level sequence; finally, the target embedding vector is scored using cosine similarity, and a speaker recognition result is generated based on the scoring result. Specifically, in this embodiment, only forward computation is required. The speech data of the speaker to be recognized is mapped to a frame-level sequence using the optimized speech pre-training model in the target speaker recognition model. Then, the frame-level sequence is input into the optimized voiceprint recognition model in the target speaker recognition model to extract robust target embedding vectors using multi-scale convolution, channel attention, and attention statistical pooling structures. Next, the target embedding vector is scored using cosine similarity, and speaker verification and recognition are completed based on the scoring result, generating a speaker recognition result.
[0074] In one specific implementation, the structure of the target speaker recognition model is shown below. Figure 2As shown, the core approach is as follows: First, based on a small initial speaker speech dataset, the error rates of sentence-level features extracted from each Transformer layer of the initial speech pre-trained model, such as WavLM, are calculated. The intermediate layer with the best speaker discrimination performance is selected as the "information high ground," and only a few Transformer layers before this Transformer layer are retained to form a simplified front end, resulting in the target speech pre-trained model. This effectively reduces the number of model parameters and inference FLOPs (Floating Point Operations) while ensuring sufficient preservation of speaker information. Based on this, an initial speaker recognition model is constructed using the target speech pre-trained model and a target-specific speaker recognition network, such as ECAPA-TDNN. This initial speaker recognition model is then trained and optimized using the initial speaker speech dataset. During training, a top-down decreasing hierarchical learning rate is applied to each Transformer layer of the target speech pre-trained model, enabling higher-level parameters to have stronger task adaptability and lower-level parameters to update more smoothly, thus achieving a better balance between "adapting to downstream tasks" and "preserving pre-trained individual information." Meanwhile, the target-specific speaker network is optimized based on the AAM-Softmax loss function, and then the target speaker recognition model that balances performance and efficiency is determined based on the optimized speech pre-training model and the optimized speaker recognition model.
[0075] As can be seen from the above, in this embodiment, an initial speaker speech dataset is first obtained, and several initial speaker speech data from the initial speaker speech dataset are input into an initial speech pre-training model. Then, a target Transformer layer satisfying a preset optimal performance condition is determined from each Transformer layer of the initial speech pre-training model, and each Transformer layer in the initial speech pre-training model located after the target Transformer layer is pruned to determine the target speech pre-training model based on the pruned model. Subsequently, an initial speaker recognition model is constructed based on the target speech pre-training model and the target speaker recognition model, and the initial speaker recognition model is trained based on several initial speaker speech data from the initial speaker speech dataset. During the training process, the initial speaker recognition model is optimized based on a preset hierarchical learning rate strategy and a target loss function to obtain the target speaker recognition model. Finally, the target speaker recognition model is used to perform speaker recognition on the speaker speech data to be recognized. As can be seen from the above, this embodiment first obtains an initial speaker speech dataset, inputs the speech data from the initial speaker speech dataset into an initial speech pre-training model, selects a target Transformer layer that meets the preset optimal performance conditions from the initial speech pre-training model, and prunes the Transformer layer after the target Transformer layer to obtain a simplified target speech pre-training model; then, it combines the target speaker recognition model to construct an initial speaker recognition model, trains it using the initial speaker speech dataset, and optimizes the model parameters during the training process through a preset hierarchical learning rate strategy and a target loss function, finally forming the target speaker recognition model; finally, it uses the target speaker recognition model to perform speaker recognition on the speech data of the speaker to be recognized. In this way, on the one hand, this embodiment selects the target Transformer layer with the best speaker discrimination performance from the initial speech pre-training model and obtains a simplified target speech pre-training model after pruning the Transformer layer, forming a simplified sub-network front end, which effectively reduces the number of parameters and computational complexity while maintaining or even improving the overall recognition performance, achieving a dual optimization of performance and efficiency. On the other hand, in this embodiment, the initial speaker recognition model is optimized by pre-setting a hierarchical learning rate strategy and a target loss function, which takes into account both recognition performance and deployment efficiency, and improves the accuracy and robustness of the model in speaker recognition.
[0076] Accordingly, see Figure 3 As shown in the illustration, this application also provides a speaker recognition device, which may include:
[0077] The dataset acquisition module 11 is used to acquire an initial speaker speech dataset and input several initial speaker speech data from the initial speaker speech dataset into an initial speech pre-training model;
[0078] The model pruning module 12 is used to determine the target Transformer layer that meets the preset optimal performance conditions from each Transformer layer of the initial speech pre-training model, and to prune each Transformer layer in the initial speech pre-training model that is located after the target Transformer layer, so as to determine the target speech pre-training model based on the pruned model.
[0079] The model optimization module 13 is used to construct an initial speaker recognition model based on the target speech pre-trained model and the target voiceprint recognition model, and to train the initial speaker recognition model based on several initial speaker speech data in the initial speaker speech dataset. During the training process, the initial speaker recognition model is optimized based on a preset hierarchical learning rate strategy and a target loss function to obtain the target speaker recognition model.
[0080] The speaker recognition module 14 is used to perform speaker recognition on the speech data of the speaker to be recognized using the target speaker recognition model.
[0081] In some specific embodiments, the model trimming module 12 may include:
[0082] A frame-level feature acquisition unit is used to acquire the frame-level features output by each Transformer layer of the initial speech pre-training model that correspond to each of the initial speaker's speech data.
[0083] The target curve construction unit is used to perform average pooling on each frame-level feature in the time dimension for any Transformer layer to obtain each feature vector, calculate the cosine similarity between any two feature vectors, and construct the target curve corresponding to the Transformer layer based on each cosine similarity; the target curve is a DET curve or a ROC curve.
[0084] An error rate determination unit is used to determine the error rate of each Transformer layer based on the target curve corresponding to each Transformer layer.
[0085] The target Transformer layer determination unit is used to determine the Transformer layer with the lowest error rate as the target Transformer layer that satisfies the preset optimal performance condition.
[0086] In some specific embodiments, the model optimization module 13 may include:
[0087] The data augmentation unit is used to perform data augmentation processing on several initial speaker speech data in the initial speaker speech dataset to obtain several target speaker speech data, and to construct a target speaker speech dataset based on each of the target speaker speech data;
[0088] The model training unit is used to train the initial speaker recognition model using the speech data of each target speaker in the target speaker speech dataset.
[0089] In some specific embodiments, the model optimization module 13 may include:
[0090] The first learning rate setting submodule is used to set the learning rate of the target Transformer layer in the target speech pre-training model as the first learning rate.
[0091] The second learning rate setting submodule is used to set the second learning rate corresponding to each Transformer layer located before the target Transformer layer in the target speech pre-trained model according to the preset hierarchical learning rate strategy and the first learning rate;
[0092] The model optimization submodule is used to determine the target learning rate corresponding to each Transformer layer in the target speech pre-trained model based on the first learning rate and the second learning rate, and to optimize the initial speaker recognition model based on the target learning rate and the target loss function to obtain the target speaker recognition model; wherein, the preset hierarchical learning rate strategy is used to control that there is a positive correlation between the second learning rate and the number of layers corresponding to each Transformer layer located before the target Transformer layer.
[0093] In some specific implementations, the model optimization submodule may include:
[0094] The speech pre-training model optimization unit is used to optimize the target speech pre-training model in the initial speaker recognition model using the AdamW optimizer, based on the Warmup strategy, the cosine annealing strategy and the target learning rate, so as to obtain the optimized speech pre-training model.
[0095] The voiceprint recognition model optimization unit is used to optimize the target voiceprint recognition model in the initial speaker recognition model using the AdamW optimizer and based on the AAM-Softmax loss function to obtain the optimized voiceprint recognition model.
[0096] The target speaker recognition model determination unit is used to determine the target speaker recognition model based on the optimized speech pre-training model and the optimized voiceprint recognition model.
[0097] In some specific embodiments, the speaker recognition device may further include:
[0098] The student model generation module is used to take the target speaker recognition model as the target teacher model and generate the target student model corresponding to the target teacher model using knowledge distillation technology.
[0099] The student model deployment module is used to deploy the target student model to the target mobile terminal or the target edge device, so that the target mobile terminal or the target edge device can use the target speaker recognition model to perform speaker recognition on the speech data of the speaker to be recognized.
[0100] In some specific embodiments, the speaker recognition module 14 may include:
[0101] A frame-level sequence determination unit is used to map the speech data of the speaker to be identified into a corresponding frame-level sequence using the optimized speech pre-training model in the target speaker recognition model, and input the frame-level sequence into the optimized voiceprint recognition model in the target speaker recognition model.
[0102] The target embedding vector generation unit is used to generate a target embedding vector based on the frame-level sequence using the optimized voiceprint recognition model in the target speaker recognition model.
[0103] The speaker recognition result generation unit is used to score the target embedding vector by cosine similarity and generate speaker recognition results based on the scoring results.
[0104] Furthermore, embodiments of this application also disclose an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the speaker recognition method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0105] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0106] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0107] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the speaker recognition method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.
[0108] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned speaker recognition method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0109] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0110] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0111] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0112] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0113] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A speaker recognition method, characterized in that, include: Obtain an initial speaker speech dataset, and input several initial speaker speech data from the initial speaker speech dataset into an initial speech pre-training model; From each Transformer layer of the initial speech pre-trained model, a target Transformer layer that meets the preset optimal performance condition is determined, and each Transformer layer in the initial speech pre-trained model that is located after the target Transformer layer is pruned, so as to determine the target speech pre-trained model based on the pruned model. An initial speaker recognition model is constructed based on the target speech pre-trained model and the target voiceprint recognition model. The initial speaker recognition model is trained based on several initial speaker speech data in the initial speaker speech dataset. During the training process, the initial speaker recognition model is optimized based on a preset hierarchical learning rate strategy and a target loss function to obtain the target speaker recognition model. The target speaker recognition model is used to perform speaker recognition on the speech data of the speaker to be identified.
2. The speaker recognition method according to claim 1, characterized in that, The step of determining the target Transformer layer that satisfies the preset optimal performance condition from each Transformer layer of the initial speech pre-trained model includes: Obtain the frame-level features output by each Transformer layer of the initial speech pre-trained model, corresponding to the initial speaker's speech data; For any Transformer layer, average pooling is performed on each frame-level feature in the time dimension to obtain each feature vector, and the cosine similarity between any two feature vectors is calculated. Based on each cosine similarity, a target curve corresponding to the Transformer layer is constructed; the target curve is a DET curve or a ROC curve. Based on the target curve corresponding to each Transformer layer, determine the equal error rate of each Transformer layer; The Transformer layer with the lowest error rate is determined as the target Transformer layer that satisfies the preset optimal performance condition.
3. The speaker recognition method according to claim 1, characterized in that, The step of training the initial speaker recognition model based on several initial speaker speech data in the initial speaker speech dataset includes: Data augmentation processing is performed on several initial speaker speech data in the initial speaker speech dataset to obtain several target speaker speech data, and a target speaker speech dataset is constructed based on each of the target speaker speech data; The initial speaker recognition model is trained using the speech data of each target speaker in the target speaker speech dataset.
4. The speaker recognition method according to claim 1, characterized in that, The optimization of the initial speaker recognition model based on a preset hierarchical learning rate strategy and a target loss function to obtain the target speaker recognition model includes: The learning rate of the target Transformer layer in the target speech pre-training model is set to the first learning rate. According to the preset hierarchical learning rate strategy and the first learning rate, the second learning rate is set for each Transformer layer in the target speech pre-trained model that is located before the target Transformer layer. The target learning rate is determined for each Transformer layer in the target speech pre-training model based on the first learning rate and the second learning rate, and the initial speaker recognition model is optimized based on the target learning rate and the target loss function to obtain the target speaker recognition model. The preset hierarchical learning rate strategy is used to control the second learning rate and the number of layers corresponding to each Transformer layer before the target Transformer layer to be positively correlated.
5. The speaker recognition method according to claim 4, characterized in that, The step of optimizing the initial speaker recognition model based on the target learning rate and the target loss function to obtain the target speaker recognition model includes: Using the AdamW optimizer, the target speech pre-trained model in the initial speaker recognition model is optimized based on the Warmup policy, cosine annealing policy and the target learning rate to obtain the optimized speech pre-trained model. Using the AdamW optimizer, the target voiceprint recognition model in the initial speaker recognition model is optimized based on the AAM-Softmax loss function to obtain the optimized voiceprint recognition model; The target speaker recognition model is determined based on the optimized speech pre-training model and the optimized voiceprint recognition model.
6. The speaker recognition method according to any one of claims 1 to 5, characterized in that, After optimizing the initial speaker recognition model based on a preset hierarchical learning rate strategy and a target loss function to obtain the target speaker recognition model, the process further includes: The target speaker recognition model is used as the target teacher model, and the target student model corresponding to the target teacher model is generated using knowledge distillation technology. The target student model is deployed to the target mobile device or the target edge device so that the target mobile device or the target edge device can use the target speaker recognition model to perform speaker recognition on the speech data of the speaker to be recognized.
7. The speaker recognition method according to claim 5, characterized in that, The step of using the target speaker recognition model to perform speaker recognition on the speech data of the speaker to be recognized includes: Using the optimized speech pre-training model in the target speaker recognition model, the speech data of the speaker to be recognized is mapped into a corresponding frame-level sequence, and the frame-level sequence is input into the optimized voiceprint recognition model in the target speaker recognition model; Using the optimized voiceprint recognition model in the target speaker recognition model, a target embedding vector is generated based on the frame-level sequence; The target embedding vector is scored using cosine similarity, and speaker recognition results are generated based on the scoring results.
8. A speaker recognition device, characterized in that, include: The dataset acquisition module is used to acquire an initial speaker speech dataset and input several initial speaker speech data from the initial speaker speech dataset into an initial speech pre-training model; The model pruning module is used to determine the target Transformer layer that meets the preset optimal performance conditions from each Transformer layer of the initial speech pre-training model, and to prune each Transformer layer in the initial speech pre-training model that is located after the target Transformer layer, so as to determine the target speech pre-training model based on the pruned model. The model optimization module is used to construct an initial speaker recognition model based on the target speech pre-trained model and the target voiceprint recognition model, and to train the initial speaker recognition model based on several initial speaker speech data in the initial speaker speech dataset. During the training process, the initial speaker recognition model is optimized based on a preset hierarchical learning rate strategy and a target loss function to obtain the target speaker recognition model. The speaker recognition module is used to perform speaker recognition on the speech data of the speaker to be recognized using the target speaker recognition model.
9. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the speaker recognition method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the speaker recognition method as described in any one of claims 1 to 7.