A self-distillation and offline label correction-based unsupervised speaker recognition model training method and device, equipment and medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- MALANSHAN AUDIO & VIDEO LABORATORY
- Filing Date
- 2026-04-21
- Publication Date
- 2026-07-14
Smart Images
Figure CN122392492A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and medium for training an unsupervised speaker recognition model based on self-distillation and offline label correction. Background Technology
[0002] Speaker recognition technology is widely used in intelligent voice interaction, financial customer service, meeting transcription, and forensic evidence collection. Traditional speaker recognition methods typically rely on supervised training using a large number of manually labeled speaker identifiers. However, in practical engineering applications, obtaining large-scale, high-quality labeled speech data is costly and faces numerous restrictions regarding privacy compliance. Therefore, unsupervised speaker recognition has become a research hotspot in recent years. Its core idea is to utilize unlabeled speech data to automatically generate pseudo-labels through methods such as clustering, and then use these pseudo-labels as supervisory signals to train the speaker recognition model.
[0003] While existing unsupervised speaker recognition methods alleviate the reliance on manual annotation to some extent, they still have significant shortcomings. Initial clustering results often contain boundary samples, outliers, and mis-clustered samples. If all clustering results are directly used as hard labels for training, erroneous pseudo-labels will be continuously absorbed and amplified by the model, leading to representation learning bias and decreased model performance. Simultaneously, the model lacks the ability to assess sample reliability in the early stages of training, making it difficult to distinguish which samples are suitable as supervisory signals and which should be delayed in use, easily falling into a vicious cycle of "erroneous labels misleading the model, and the model reinforcing erroneous labels." Furthermore, existing methods handle low-reliability samples in a rather crude manner, either discarding them all, resulting in data waste, or including them all in training, exacerbating noise interference. Moreover, pseudo-labels are usually fixed during training, lacking effective re-clustering and label correction mechanisms, causing model performance to be consistently limited by the quality of the initial clustering.
[0004] As can be seen from the above, how to train a speaker recognition model from a large amount of unlabeled speech without human annotation, and how to continuously improve the quality of pseudo-labels and the model's discrimination ability, is an urgent problem to be solved. Summary of the Invention
[0005] In view of this, the purpose of this invention is to provide an unsupervised speaker recognition model training method, apparatus, device, and medium based on self-distillation and offline label correction, which can train a speaker recognition model from a large amount of unlabeled speech without manual annotation, and continuously improve the quality of pseudo-labels and the model's discriminative ability. The specific solution is as follows: Firstly, this application provides a method for training an unsupervised speaker recognition model based on self-distillation and offline label correction, including: Obtain an unlabeled speech sample set, and use a preset speech representation model or an initialized speaker coding network to extract the initial speaker embeddings corresponding to the speech samples in the unlabeled speech sample set, so as to obtain an initial embedding set consisting of several initial speaker embeddings; Unsupervised clustering is performed based on the initial embedding set to obtain initial pseudo-labels corresponding to each speech sample. Then, a student model from the speaker recognition model is used to perform forward prediction on the speech samples to obtain a prediction distribution corresponding to the category of the initial pseudo-labels. Finally, the speech samples are classified based on the prediction distribution to obtain a first pseudo-label sample set and a second pseudo-label sample set. The first pseudo-label sample set is a set of speech samples that meet a preset reliability condition; the second pseudo-label sample set is a set of speech samples that do not meet the preset reliability condition. The first pseudo-label sample set is used to perform pseudo-supervised classification training on the current student model, and the teacher model in the speaker recognition model generates a soft target based on the second pseudo-label sample set, so as to use the soft target to perform self-distillation training on the student model after pseudo-supervised classification training. The speaker embeddings corresponding to all speech samples are re-extracted using the current student model trained by self-distillation to construct an updated embedding set. Unsupervised clustering is then performed based on the updated embedding set to obtain the updated pseudo-labels corresponding to each speech sample. The updated pseudo-labels are corrected offline to obtain corrected labels. The speech samples are then reclassified based on the corrected labels to obtain an updated first pseudo-label sample set and a second pseudo-label sample set. The process then jumps to the step of using the first pseudo-label sample set to perform pseudo-supervised classification training on the current student model until a preset stopping condition is met, thus completing the training of the speaker recognition model.
[0006] Optionally, classifying the speech samples based on the predicted distribution to obtain a first pseudo-label sample set and a second pseudo-label sample set includes: The predicted probability of the speech sample corresponding to the category of the initial pseudo-label is obtained from the predicted distribution, and the predicted probability is compared with a preset confidence threshold. If the predicted probability is greater than or equal to the confidence threshold, the speech sample is assigned to the first pseudo-label sample set; otherwise, it is assigned to the second pseudo-label sample set. Alternatively, calculate the information entropy of the predicted distribution. If the predicted probability is greater than or equal to the confidence threshold and the information entropy is less than or equal to a preset entropy threshold, then the speech sample is assigned to the first pseudo-label sample set; otherwise, it is assigned to the second pseudo-label sample set. Alternatively, two different augmented views can be constructed for the same speech sample and input into the student model in the speaker recognition model respectively to obtain the corresponding prediction distributions. If the prediction categories of the two augmented views are consistent and the corresponding prediction probabilities are both greater than or equal to the confidence threshold, then the speech sample is assigned to the first pseudo-label sample set; otherwise, it is assigned to the second pseudo-label sample set.
[0007] Optionally, the unsupervised speaker recognition model training method based on self-distillation and offline label correction further includes: After each parameter update of the student model, the current parameters of the student model are obtained, and the current parameters of the teacher model are updated using an exponential moving average method to obtain the updated teacher model parameters. The parameters of the teacher model do not participate in gradient descent optimization during the entire training process, but are only updated by inheriting from the parameters of the student model through the exponential moving average method.
[0008] Optionally, the step of using the first pseudo-label sample set to perform pseudo-supervised classification training on the current student model includes: The initial pseudo-label or the corrected label obtained after offline label correction for each speech sample in the first pseudo-label sample set is used as the supervision signal, and the classification loss between the predicted output of the student model and the supervision signal is calculated using the cross-entropy loss function. The parameters of the student model are updated based on the backpropagation of the classification loss, so as to enable the student model to learn the class discrimination boundary of the speech samples in the first pseudo-label sample set.
[0009] Optionally, the step of generating soft targets based on the second pseudo-label sample set using the teacher model in the speaker recognition model, and then using the soft targets to perform self-distillation training on the student model after pseudo-supervised classification training, includes: For each speech sample in the second pseudo-label sample set, two different augmented views are constructed, and the two augmented views are respectively input into the student model and the teacher model to obtain the soft target output by the teacher model and the prediction distribution output by the student model. KL divergence is used as the distillation loss to constrain the prediction distribution of the student model to approximate the soft objective of the teacher model, thereby completing the self-distillation training of the student model.
[0010] Optionally, after constructing two different enhanced views for each speech sample in the second pseudo-label sample set, the method further includes: The student model is used to extract speaker representations corresponding to the two augmented views, and the consistency loss between the speaker representations is calculated. The consistency loss is used to update the parameters of the student model.
[0011] Optionally, the offline tag correction for the updated pseudo-tags includes: Each speech sample is segmented into multiple local segments, and the speaker embedding of each local segment is extracted and its pseudo-label category is predicted. The predicted category of each local segment is calculated, and the category that appears most frequently is used as the correction label for the speech sample; If the proportion of the most frequently occurring category is lower than a preset threshold, the speech sample is marked as an unstable label sample and included in the second pseudo-label sample set in the next round of training.
[0012] Secondly, this application provides an unsupervised speaker recognition model training device based on self-distillation and offline label correction, comprising: The sample processing module is used to obtain an unlabeled speech sample set and extract the initial speaker embeddings corresponding to the speech samples in the unlabeled speech sample set using a preset speech representation model or an initialized speaker coding network, so as to obtain an initial embedding set composed of several initial speaker embeddings. The sample classification module is used to perform unsupervised clustering based on the initial embedding set to obtain initial pseudo-labels corresponding to each speech sample, and to perform forward prediction on the speech samples using the student model in the speaker recognition model to obtain a prediction distribution corresponding to the category of the initial pseudo-labels. Then, the speech samples are classified based on the prediction distribution to obtain a first pseudo-label sample set and a second pseudo-label sample set; the first pseudo-label sample set is a set of speech samples that meet a preset reliability condition; the second pseudo-label sample set is a set of speech samples that do not meet the preset reliability condition. The model training module is used to perform pseudo-supervised classification training on the current student model using the first pseudo-label sample set, and to generate soft targets based on the second pseudo-label sample set using the teacher model in the speaker recognition model, so as to perform self-distillation training on the student model after pseudo-supervised classification training using the soft targets. The label update module is used to re-extract the speaker embeddings corresponding to all speech samples using the current student model trained by self-distillation, so as to construct an updated embedding set, and perform unsupervised clustering based on the updated embedding set to obtain the updated pseudo-labels corresponding to each speech sample. The offline correction module is used to perform offline label correction on the updated pseudo-labels to obtain corrected labels, and to reclassify the speech samples based on the corrected labels to obtain an updated first pseudo-label sample set and a second pseudo-label sample set. Then, it jumps to the step of using the first pseudo-label sample set to perform pseudo-supervised classification training on the current student model until a preset stopping condition is met to complete the training of the speaker recognition model.
[0013] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the aforementioned unsupervised speaker recognition model training method based on self-distillation and offline label correction.
[0014] Fourthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned unsupervised speaker recognition model training method based on self-distillation and offline label correction.
[0015] This application provides an unsupervised speaker recognition model training method based on self-distillation and offline label correction. The method involves obtaining an unlabeled speech sample set, extracting initial speaker embeddings corresponding to speech samples in the unlabeled speech sample set using a preset speech representation model or an initialized speaker coding network, to obtain an initial embedding set composed of several initial speaker embeddings; performing unsupervised clustering based on the initial embedding set to obtain initial pseudo-labels corresponding to each speech sample; and using a student model in the speaker recognition model to perform forward prediction on the speech samples to obtain a prediction distribution corresponding to the categories of the initial pseudo-labels. Then, classifying the speech samples based on the prediction distribution yields a first pseudo-label sample set and a second pseudo-label sample set. The first pseudo-label sample set is a set of speech samples that meet a preset reliability condition; the second pseudo-label sample set is a set of speech samples that do not meet the preset reliability condition. The first pseudo-label sample set is used to perform pseudo-supervised classification training on the current student model. A soft target is generated based on the second pseudo-label sample set using the teacher model in the speaker recognition model. This soft target is then used to perform self-distillation training on the student model after pseudo-supervised classification training. The speaker embeddings corresponding to all speech samples are re-extracted using the self-distilled current student model to construct an updated embedding set. Unsupervised clustering is then performed based on the updated embedding set to obtain updated pseudo-labels corresponding to each speech sample. Offline label correction is performed on the updated pseudo-labels to obtain corrected labels. The speech samples are then re-classified based on the corrected labels to obtain updated first and second pseudo-label sample sets. The process then jumps to the step of performing pseudo-supervised classification training on the current student model using the first pseudo-label sample set until a preset stopping condition is met, thus completing the training of the speaker recognition model.
[0016] As shown above, this application uses general initial clustering and predicted distribution evaluation to divide the samples into high-confidence samples (the first set of pseudo-label samples) and low-confidence samples (the second set of pseudo-label samples). Pseudo-supervised classification training and teacher-guided self-distillation training are employed respectively. This ensures that the model establishes accurate class boundaries on reliable samples while avoiding interference from erroneous labels in low-confidence samples, thus improving the overall utilization efficiency of unlabeled data. Based on this, the embeddedness is re-extracted and re-clustered using the trained student model, combined with offline label correction to obtain more accurate corrected labels, thereby iteratively updating the sample division and training process. Through repeated iterations, model capability and pseudo-label quality mutually promote and improve synchronously, ultimately obtaining a robust and highly discriminative speaker recognition model under completely unlabeled conditions, significantly improving the training stability and recognition performance of unsupervised speaker recognition. This enables the training of a speaker recognition model from a large amount of unlabeled speech without manual annotation, and continuously improves the quality of pseudo-labels and the model's discriminative ability. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0018] Figure 1 This is a flowchart of an unsupervised speaker recognition model training method based on self-distillation and offline label correction disclosed in this invention; Figure 2 This is a schematic diagram of an unsupervised speaker recognition model training device based on self-distillation and offline label correction disclosed in this invention. Figure 3 This is a structural diagram of an electronic device disclosed in this invention. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] While existing unsupervised speaker recognition methods alleviate the reliance on manual annotation to some extent, they still have significant shortcomings. Initial clustering results often contain boundary samples, outliers, and mis-clustered samples. If all clustering results are directly used as hard labels for training, erroneous pseudo-labels will be continuously absorbed and amplified by the model, leading to representation learning bias and decreased model performance. Simultaneously, the model lacks the ability to assess sample reliability in the early stages of training, making it difficult to distinguish which samples are suitable as supervision signals and which should be delayed, easily falling into a vicious cycle of "erroneous labels misleading the model, and the model reinforcing erroneous labels." Furthermore, existing methods handle low-reliability samples in a rather crude manner, either discarding them all, resulting in data waste, or including them all in training, exacerbating noise interference. Moreover, pseudo-labels are usually fixed during training, lacking effective re-clustering and label correction mechanisms, causing model performance to be limited by the initial clustering quality in the long term. To address these issues, this application provides an unsupervised speaker recognition model training scheme based on self-distillation and offline label correction. This scheme enables training of a speaker recognition model from a large amount of unlabeled speech without manual annotation, and continuously improves the quality of pseudo-labels and the model's discriminative ability.
[0021] See Figure 1 As shown in the embodiments of this application, an unsupervised speaker recognition model training method based on self-distillation and offline label correction is disclosed, including: Step S11: Obtain an unlabeled speech sample set, and use a preset speech representation model or an initialized speaker coding network to extract the initial speaker embeddings corresponding to the speech samples in the unlabeled speech sample set, so as to obtain an initial embedding set composed of several initial speaker embeddings.
[0022] In this embodiment, an unlabeled speech sample set is obtained. ,in, Let N represent the i-th speech sample, and N be the total number of samples. Further, an initial speaker embedding is extracted for each speech sample using a pre-trained speech representation model or an initialized speaker coding network. Specifically, in one implementation, the initial speaker embedding can be obtained by performing statistical pooling, average pooling, or other convergence methods on the speech frame-level representation.
[0023] Step S12: Perform unsupervised clustering based on the initial embedding set to obtain initial pseudo-labels corresponding to each speech sample, and use the student model in the speaker recognition model to perform forward prediction on the speech samples to obtain the prediction distribution corresponding to the category of the initial pseudo-labels. Then, classify the speech samples based on the prediction distribution to obtain a first pseudo-label sample set and a second pseudo-label sample set.
[0024] In this embodiment, based on the initial embedding set Perform unsupervised clustering to obtain the initial cluster affiliation label (initial pseudo-label) for each sample. The specific formula is as follows: ; in, This represents a clustering algorithm. This represents the initial pseudo-label of the i-th sample. The clustering algorithm can be graph clustering, K-means clustering, spectral clustering, hierarchical clustering, or other unsupervised clustering algorithms suitable for speech embedding grouping, and no specific limitation is made here.
[0025] Furthermore, after obtaining the initial pseudo-labels, the reliability of the samples is evaluated and classified based on these initial pseudo-labels. Specifically, based on the initialized student model, forward prediction can be performed on each sample to obtain its predicted class distribution. If the sample The initial pseudo-label is Then the predicted probability of the student model for that pseudo-label category can be extracted. The samples are evaluated using this as a reliability metric. Furthermore, the information entropy of the predicted distribution of the sample can be calculated as another reliability metric. A lower entropy value indicates a more concentrated predicted distribution for a particular sample, suggesting a more certain classification of that sample. Additionally, two different augmented views can be constructed for the same sample, input into the student model respectively, and their predicted categories compared to determine consistency. Alternatively, the distance between corresponding predicted distributions can be compared to use as a consistency metric before and after augmentation. The reliability of the samples is jointly evaluated using one or more of these metrics. When a sample meets a preset reliability condition, it is classified into a high-confidence pseudo-label sample set (the first pseudo-label sample set). Otherwise, it is divided into the remaining unlabeled sample set (the second pseudo-label sample set). .
[0026] The preset reliability condition can be that the predicted probability is higher than a preset confidence threshold. For example, in one specific embodiment, classifying the speech samples based on the prediction distribution to obtain a first pseudo-label sample set and a second pseudo-label sample set may include: obtaining the predicted probability of the speech sample corresponding to the category of the initial pseudo-label from the prediction distribution, comparing the predicted probability with a preset confidence threshold, and if the predicted probability is greater than or equal to the confidence threshold, then the speech sample is assigned to the first pseudo-label sample set; otherwise, it is assigned to the second pseudo-label sample set.
[0027] The preset reliability condition can also employ a joint criterion of "predicted probability higher than a threshold and prediction entropy lower than a threshold". For example, in another specific embodiment, classifying the speech samples based on the prediction distribution to obtain a first pseudo-label sample set and a second pseudo-label sample set may include: calculating the information entropy of the prediction distribution. The specific formula is shown below: ; Where c is the category index. If the predicted probability is greater than or equal to the confidence threshold and the information entropy is less than or equal to the preset entropy threshold, then the speech sample is assigned to the first pseudo-label sample set; otherwise, it is assigned to the second pseudo-label sample set.
[0028] The preset reliability condition can also adopt a joint criterion of "consistent predicted categories under different augmented views and corresponding prediction confidence levels higher than a threshold". For example, in another specific embodiment, classifying the speech samples based on the prediction distribution to obtain a first pseudo-label sample set and a second pseudo-label sample set may include: constructing two different augmented views for the same speech sample, inputting them into the student model in the speaker recognition model respectively to obtain corresponding prediction distributions; if the predicted categories of the two augmented views are consistent and the corresponding prediction probabilities are both greater than or equal to the confidence threshold, then the speech sample is assigned to the first pseudo-label sample set; otherwise, it is assigned to the second pseudo-label sample set.
[0029] Step S13: Use the first pseudo-label sample set to perform pseudo-supervised classification training on the current student model, and use the teacher model in the speaker recognition model to generate soft targets based on the second pseudo-label sample set, so as to use the soft targets to perform self-distillation training on the student model after pseudo-supervised classification training.
[0030] In this embodiment, after the initial sample partitioning is completed, the teacher-student dual-branch speaker recognition model is trained using the classified labeled sample set. The student model is denoted as... The teacher model is denoted as ,in, Represents the parameters of the student model. This represents the teacher model parameters. The student model receives high-confidence samples and unlabeled samples and performs training; the teacher model provides soft supervision for the unlabeled samples. The teacher model parameters are not directly optimized using gradients, but are updated using an exponential moving average of the student model parameters. That is, after each parameter update of the student model, the current parameters of the student model are obtained, and the current parameters of the teacher model are updated using an exponential moving average to obtain the updated teacher model parameters. The teacher model parameters do not participate in gradient descent optimization throughout the training process; they are only updated by inheriting from the student model parameters using the exponential moving average. The specific expression is shown below: ; in, This is the smoothing coefficient. Using this update mechanism allows the teacher model parameters to change more smoothly than the student model, resulting in more stable soft prediction results.
[0031] In this embodiment, for the high-confidence pseudo-label sample set, the present invention employs pseudo-supervised classification training. Specifically, the pseudo-supervised classification training of the current student model using the first pseudo-label sample set may include: using the initial pseudo-label or the corrected label obtained after offline label correction for each speech sample in the first pseudo-label sample set as a supervision signal; calculating the classification loss between the predicted output of the student model and the supervision signal using the cross-entropy loss function; and updating the parameters of the student model based on the classification loss through backpropagation to enable the student model to learn the class discrimination boundary of the speech samples in the first pseudo-label sample set. For example, in one specific implementation, suppose the student model... The output category distribution is Its corresponding pseudo-tag is The formula for calculating the supervision loss is as follows: ; in, To monitor losses, The initial pseudo-labels can be directly used as the samples, or they can be used as updated pseudo-labels after subsequent correction. The supervised loss is used to establish the speaker category discrimination boundary using high-confidence samples. It is worth mentioning that label noise suppression strategies can also be combined, such as reducing the loss weight of abnormal samples based on the historical prediction stability of the samples during training, or gradually introducing model predictions to participate in label correction, in order to reduce the interference of erroneous pseudo-labels on the training process.
[0032] In this embodiment, for the unlabeled sample set, a hard-label classification loss is not directly applied. Instead, a soft target is provided by the teacher model for self-distillation training. Specifically, the step of using the teacher model in the speaker recognition model to generate a soft target based on the second pseudo-labeled sample set, and then using the soft target to perform self-distillation training on the student model after pseudo-supervised classification training, may include: constructing two different augmented views for each speech sample in the second pseudo-labeled sample set, and inputting the two augmented views into the student model and the teacher model respectively to obtain the soft target output by the teacher model and the prediction distribution output by the student model; using KL divergence as the distillation loss to constrain the prediction distribution of the student model to approximate the soft target of the teacher model, thereby completing the self-distillation training of the student model. For example, in one specific implementation, for any sample... Construct two different enhanced views and Input the teacher model and student model respectively to obtain the teacher output distribution. and student output distribution The set of samples whose confidence levels in the teacher output meet the preset conditions. The student model output is approximated by the teacher model output using distillation loss constraints, as shown in the following expression: ; in, For distillation losses, This represents the KL divergence. Through this distillation process, unlabeled samples can participate in representation learning under the guidance of the teacher model without relying on rigid discrete class labels.
[0033] Furthermore, consistency constraints can be imposed on the two augmented views of the unlabeled samples to ensure the model maintains stable output in response to input perturbations. Specifically, after constructing two different augmented views for each speech sample in the second pseudo-labeled sample set, the process can further include: extracting the speaker representations corresponding to the two augmented views using the student model, and calculating the consistency loss between the speaker representations; wherein the consistency loss is used to update the parameters of the student model. For example, in one specific implementation, suppose the speaker representations output by the student model for the two augmented views are respectively... and Then, a consistency loss can be constructed, the specific expression of which is shown below: ; in, For consistency loss, Indicates an unlabeled sample set The consistency term is used to obtain a small batch of samples from the mid-sample. Its purpose is to enable the model to learn stable speaker features by maintaining consistency in the representations of the same sample across different views, even in the absence of explicit class supervision.
[0034] Based on the high-confidence sample supervision loss, unlabeled sample distillation loss, and consistency loss, the overall optimization objective expression of this embodiment is as follows: ; in, For the total loss, The coefficients are used to balance the weights of the distillation and consistency terms. The student model parameters are updated by minimizing the overall loss, and the teacher model parameters are updated simultaneously in the manner described above, thus completing one round of unsupervised self-training.
[0035] Step S14: Use the current student model trained by self-distillation to re-extract the speaker embeddings corresponding to all speech samples to construct an updated embedding set, and perform unsupervised clustering based on the updated embedding set to obtain the updated pseudo-labels corresponding to each speech sample.
[0036] In this embodiment, after one or more rounds of training, the updated student model is used to re-extract speaker embeddings from all speech samples, and unsupervised clustering is re-performed in the new embedding space to obtain updated pseudo-label results. Since the student model has already achieved stronger speaker discrimination ability through high-confidence pseudo-supervision, self-distillation, and consistency learning in the aforementioned training phase, the new embeddings usually have better intra-class compactness and inter-class separability, thus supporting higher-quality clustering results.
[0037] Step S15: Perform offline label correction on the updated pseudo-labels to obtain corrected labels, and reclassify the speech samples based on the corrected labels to obtain an updated first pseudo-label sample set and a second pseudo-label sample set. Then, jump to the step of using the first pseudo-label sample set to perform pseudo-supervised classification training on the current student model until a preset stopping condition is met to complete the training of the speaker recognition model.
[0038] In this embodiment, offline label correction is performed on the updated pseudo-labels. Specifically, the offline label correction of the updated pseudo-labels may include: segmenting each speech sample into multiple local segments, extracting the speaker embedding of each local segment and predicting its pseudo-label category; statistically analyzing the predicted categories of each local segment, and using the category with the most occurrences as the corrected label for the speech sample; if the proportion of the category with the most occurrences is lower than a preset proportion threshold, the speech sample is marked as a label unstable sample and included in the second pseudo-label sample set in the next round of training. That is, the same speech is segmented into multiple local segments, and prediction is performed on each segment separately; if a certain category dominates the prediction results of multiple segments, that category is used as the corrected label for the speech; if the segment prediction results are significantly dispersed, the sample is determined to be a label unstable sample, and its priority is reduced or it is temporarily excluded from the high-confidence supervision set in subsequent training. The label results after re-clustering and offline label correction can be used again for sample reliability assessment and high-confidence sample screening, and then enter the next round of pseudo-supervised training and self-distillation training. Thus, this invention forms a complete closed loop of "initial pseudo-label generation—high-confidence sample screening—joint training by teachers and students—re-embedding—re-clustering—offline label correction—iterative retraining". As the number of training rounds increases, the model's capabilities and pseudo-label quality can mutually promote and improve synchronously, thereby gradually approaching a better speaker representation space under conditions without manual labeling.
[0039] As can be seen from the above, the embodiments of this application, by introducing a closed-loop training framework of self-distillation and offline label correction, generate pseudo-labels using initial clustering under conditions of no manual annotation, and combine prediction probability, information entropy, and enhanced vision. Figure 1 High-confidence samples are selected for supervised training based on consistency and other indicators. Simultaneously, a teacher model is used to generate soft targets from low-confidence samples for self-distillation training, and consistency constraints are introduced to enhance model stability. Furthermore, pseudo-label quality is iteratively optimized through re-clustering and offline label correction (such as segment majority voting and unstable sample labeling), allowing model capability and pseudo-label quality to mutually promote and improve simultaneously. This method effectively avoids interference from erroneous pseudo-labels during training, significantly improves the utilization efficiency of unlabeled data, and ultimately obtains a highly discriminative speaker recognition model in unsupervised scenarios, possessing training stability, label updatableness, and sufficient data utilization.
[0040] See Figure 2 As shown in the figure, this application discloses an unsupervised speaker recognition model training device based on self-distillation and offline label correction, comprising: The sample processing module 11 is used to obtain an unlabeled speech sample set and extract the initial speaker embedding corresponding to the speech sample in the unlabeled speech sample set using a preset speech representation model or an initialized speaker coding network, so as to obtain an initial embedding set composed of several initial speaker embeddings. The sample classification module 12 is used to perform unsupervised clustering based on the initial embedding set to obtain initial pseudo-labels corresponding to each speech sample, and to perform forward prediction on the speech samples using the student model in the speaker recognition model to obtain a prediction distribution corresponding to the category of the initial pseudo-labels. Then, the speech samples are classified based on the prediction distribution to obtain a first pseudo-label sample set and a second pseudo-label sample set; the first pseudo-label sample set is a set of speech samples that meet a preset reliability condition; the second pseudo-label sample set is a set of speech samples that do not meet the preset reliability condition. The model training module 13 is used to perform pseudo-supervised classification training on the current student model using the first pseudo-label sample set, and to generate soft targets based on the second pseudo-label sample set using the teacher model in the speaker recognition model, so as to perform self-distillation training on the student model after pseudo-supervised classification training using the soft targets. The label update module 14 is used to re-extract the speaker embeddings corresponding to all speech samples using the current student model trained by self-distillation, so as to construct an updated embedding set, and perform unsupervised clustering based on the updated embedding set to obtain the updated pseudo-labels corresponding to each speech sample. The offline correction module 15 is used to perform offline label correction on the updated pseudo-labels to obtain corrected labels, and to reclassify the speech samples based on the corrected labels to obtain an updated first pseudo-label sample set and a second pseudo-label sample set. Then, it jumps to the step of using the first pseudo-label sample set to perform pseudo-supervised classification training on the current student model until a preset stopping condition is met to complete the training of the speaker recognition model.
[0041] In some specific embodiments, the sample classification module 12 may specifically include: The first classification unit is used to obtain the predicted probability of the category of the speech sample corresponding to the initial pseudo-label from the predicted distribution, and compare the predicted probability with a preset confidence threshold. If the predicted probability is greater than or equal to the confidence threshold, the speech sample is classified into the first pseudo-label sample set; otherwise, it is classified into the second pseudo-label sample set. The second classification unit is used to calculate the information entropy of the predicted distribution. If the predicted probability is greater than or equal to the confidence threshold and the information entropy is less than or equal to the preset entropy threshold, then the speech sample is classified into the first pseudo-label sample set; otherwise, it is classified into the second pseudo-label sample set. The third classification unit is used to construct two different augmented views for the same speech sample, and input them into the student model in the speaker recognition model to obtain the corresponding prediction distributions. If the prediction categories of the two augmented views are consistent and the corresponding prediction probabilities are both greater than or equal to the confidence threshold, then the speech sample is classified into the first pseudo-label sample set; otherwise, it is classified into the second pseudo-label sample set.
[0042] In some specific embodiments, the model training module 13 may specifically include: The loss calculation unit is used to take the initial pseudo-label or the corrected label obtained after offline label correction of each speech sample in the first pseudo-label sample set as the supervision signal, and use the cross-entropy loss function to calculate the classification loss between the predicted output of the student model and the supervision signal. The discrimination boundary construction unit is used to update the parameters of the student model based on the backpropagation of the classification loss, so as to enable the student model to learn the category discrimination boundary of the speech samples in the first pseudo-label sample set; An augmented view construction unit is used to construct two different augmented views for each speech sample in the second pseudo-label sample set, and input the two augmented views into the student model and the teacher model respectively to obtain the soft target output by the teacher model and the prediction distribution output by the student model. The self-distillation training unit is used to employ KL divergence as distillation loss to constrain the prediction distribution of the student model to approximate the soft objective of the teacher model, thereby completing the self-distillation training of the student model.
[0043] In some specific embodiments, the offline correction module 15 may specifically include: The sample segmentation unit is used to segment each speech sample into multiple local segments, extract the speaker embedding of each local segment, and predict its pseudo-label category. The label correction unit is used to count the predicted categories of each local segment and take the category that appears most frequently as the correction label of the speech sample. The labeling unit is used to mark the speech sample as an unstable label sample if the proportion of the most frequently occurring category is lower than a preset proportion threshold, and to include it in the second pseudo-label sample set in the next round of training.
[0044] In some specific embodiments, the unsupervised speaker recognition model training device based on self-distillation and offline label correction may further include: The parameter update unit is used to obtain the current parameters of the student model after each parameter update of the student model, and update the current parameters of the teacher model using an exponential moving average method to obtain the updated teacher model parameters; the parameters of the teacher model do not participate in gradient descent optimization during the entire training process, but are only updated by inheriting from the parameters of the student model through the exponential moving average method. A consistency loss determination unit is used to extract speaker representations corresponding to the two augmented views using the student model and to calculate the consistency loss between the speaker representations; wherein the consistency loss is used to update the parameters of the student model.
[0045] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the unsupervised speaker recognition model training method based on self-distillation and offline label correction disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0046] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0047] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0048] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the unsupervised speaker recognition model training method based on self-distillation and offline label correction disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.
[0049] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned unsupervised speaker recognition model training method based on self-distillation and offline label correction. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0050] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0051] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0052] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0053] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0054] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for training an unsupervised speaker recognition model based on self-distillation and offline label correction, characterized in that, include: Obtain an unlabeled speech sample set, and use a preset speech representation model or an initialized speaker coding network to extract the initial speaker embeddings corresponding to the speech samples in the unlabeled speech sample set, so as to obtain an initial embedding set consisting of several initial speaker embeddings; Unsupervised clustering is performed based on the initial embedding set to obtain initial pseudo-labels corresponding to each speech sample. Then, a student model from the speaker recognition model is used to perform forward prediction on the speech samples to obtain a prediction distribution corresponding to the category of the initial pseudo-labels. Finally, the speech samples are classified based on the prediction distribution to obtain a first pseudo-label sample set and a second pseudo-label sample set. The first pseudo-label sample set is a set of speech samples that meet a preset reliability condition; the second pseudo-label sample set is a set of speech samples that do not meet the preset reliability condition. The first pseudo-label sample set is used to perform pseudo-supervised classification training on the current student model, and the teacher model in the speaker recognition model generates a soft target based on the second pseudo-label sample set, so as to use the soft target to perform self-distillation training on the student model after pseudo-supervised classification training. The speaker embeddings corresponding to all speech samples are re-extracted using the current student model trained by self-distillation to construct an updated embedding set. Unsupervised clustering is then performed based on the updated embedding set to obtain the updated pseudo-labels corresponding to each speech sample. The updated pseudo-labels are corrected offline to obtain corrected labels. The speech samples are then reclassified based on the corrected labels to obtain an updated first pseudo-label sample set and a second pseudo-label sample set. The process then jumps to the step of using the first pseudo-label sample set to perform pseudo-supervised classification training on the current student model until a preset stopping condition is met, thus completing the training of the speaker recognition model.
2. The unsupervised speaker recognition model training method based on self-distillation and offline label correction according to claim 1, characterized in that, The step of classifying the speech samples based on the predicted distribution to obtain a first pseudo-label sample set and a second pseudo-label sample set includes: The predicted probability of the speech sample corresponding to the category of the initial pseudo-label is obtained from the predicted distribution, and the predicted probability is compared with a preset confidence threshold. If the predicted probability is greater than or equal to the confidence threshold, the speech sample is assigned to the first pseudo-label sample set; otherwise, it is assigned to the second pseudo-label sample set. Alternatively, calculate the information entropy of the predicted distribution. If the predicted probability is greater than or equal to the confidence threshold and the information entropy is less than or equal to a preset entropy threshold, then the speech sample is assigned to the first pseudo-label sample set; otherwise, it is assigned to the second pseudo-label sample set. Alternatively, two different augmented views can be constructed for the same speech sample and input into the student model in the speaker recognition model respectively to obtain the corresponding prediction distributions. If the prediction categories of the two augmented views are consistent and the corresponding prediction probabilities are both greater than or equal to the confidence threshold, then the speech sample is assigned to the first pseudo-label sample set; otherwise, it is assigned to the second pseudo-label sample set.
3. The unsupervised speaker recognition model training method based on self-distillation and offline label correction according to claim 1, characterized in that, Also includes: After each parameter update of the student model, the current parameters of the student model are obtained, and the current parameters of the teacher model are updated using an exponential moving average method to obtain the updated teacher model parameters. The parameters of the teacher model do not participate in gradient descent optimization during the entire training process, but are only updated by inheriting from the parameters of the student model through the exponential moving average method.
4. The unsupervised speaker recognition model training method based on self-distillation and offline label correction according to claim 1, characterized in that, The step of using the first pseudo-label sample set to perform pseudo-supervised classification training on the current student model includes: The initial pseudo-label or the corrected label obtained after offline label correction for each speech sample in the first pseudo-label sample set is used as the supervision signal, and the classification loss between the predicted output of the student model and the supervision signal is calculated using the cross-entropy loss function. The parameters of the student model are updated based on the backpropagation of the classification loss, so as to enable the student model to learn the class discrimination boundary of the speech samples in the first pseudo-label sample set.
5. The unsupervised speaker recognition model training method based on self-distillation and offline label correction according to claim 1, characterized in that, The step of generating soft targets based on the second pseudo-label sample set using the teacher model in the speaker recognition model, and then using the soft targets to perform self-distillation training on the student model after pseudo-supervised classification training, includes: For each speech sample in the second pseudo-label sample set, two different augmented views are constructed, and the two augmented views are respectively input into the student model and the teacher model to obtain the soft target output by the teacher model and the prediction distribution output by the student model. KL divergence is used as the distillation loss to constrain the prediction distribution of the student model to approximate the soft objective of the teacher model, thereby completing the self-distillation training of the student model.
6. The unsupervised speaker recognition model training method based on self-distillation and offline label correction according to claim 5, characterized in that, After constructing two different enhanced views for each speech sample in the second pseudo-label sample set, the process further includes: The student model is used to extract speaker representations corresponding to the two augmented views, and the consistency loss between the speaker representations is calculated. The consistency loss is used to update the parameters of the student model.
7. The unsupervised speaker recognition model training method based on self-distillation and offline label correction according to claim 1, characterized in that, The offline tag correction for the updated pseudo-tags includes: Each speech sample is segmented into multiple local segments, and the speaker embedding of each local segment is extracted and its pseudo-label category is predicted. The predicted category of each local segment is calculated, and the category that appears most frequently is used as the correction label for the speech sample; If the proportion of the most frequently occurring category is lower than a preset threshold, the speech sample is marked as an unstable label sample and included in the second pseudo-label sample set in the next round of training.
8. A training device for an unsupervised speaker recognition model based on self-distillation and offline label correction, characterized in that, include: The sample processing module is used to obtain an unlabeled speech sample set and extract the initial speaker embeddings corresponding to the speech samples in the unlabeled speech sample set using a preset speech representation model or an initialized speaker coding network, so as to obtain an initial embedding set composed of several initial speaker embeddings. The sample classification module is used to perform unsupervised clustering based on the initial embedding set to obtain initial pseudo-labels corresponding to each speech sample, and to perform forward prediction on the speech samples using the student model in the speaker recognition model to obtain a prediction distribution corresponding to the category of the initial pseudo-labels. Then, the speech samples are classified based on the prediction distribution to obtain a first pseudo-label sample set and a second pseudo-label sample set; the first pseudo-label sample set is a set of speech samples that meet a preset reliability condition; the second pseudo-label sample set is a set of speech samples that do not meet the preset reliability condition. The model training module is used to perform pseudo-supervised classification training on the current student model using the first pseudo-label sample set, and to generate soft targets based on the second pseudo-label sample set using the teacher model in the speaker recognition model, so as to perform self-distillation training on the student model after pseudo-supervised classification training using the soft targets. The label update module is used to re-extract the speaker embeddings corresponding to all speech samples using the current student model trained by self-distillation, so as to construct an updated embedding set, and perform unsupervised clustering based on the updated embedding set to obtain the updated pseudo-labels corresponding to each speech sample. The offline correction module is used to perform offline label correction on the updated pseudo-labels to obtain corrected labels, and to reclassify the speech samples based on the corrected labels to obtain an updated first pseudo-label sample set and a second pseudo-label sample set. Then, it jumps to the step of using the first pseudo-label sample set to perform pseudo-supervised classification training on the current student model until a preset stopping condition is met to complete the training of the speaker recognition model.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the unsupervised speaker recognition model training method based on self-distillation and offline label correction as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the unsupervised speaker recognition model training method based on self-distillation and offline label correction as described in any one of claims 1 to 7.