A three-stage semi-supervised voiceprint recognition method based on adaptive expansion strategy

Abstract

The present application relates to a kind of three-stage semi-supervised voiceprint recognition method based on adaptive expansion strategy, comprising: (1) propose the semi-supervised learning method of constructing three-stage framework;(2) propose the three-stage semi-supervised voiceprint recognition method based on adaptive expansion strategy;(3) complete the training and test of the three-stage semi-supervised voiceprint recognition method based on adaptive expansion strategy.The three-stage semi-supervised voiceprint recognition method based on adaptive expansion strategy proposed in the present application can make full use of unlabeled voiceprint data to improve voiceprint recognition performance when there is insufficient labeled voiceprint data, with low equal error rate, showing certain effectiveness.

Description

Technical Field

[0001] This invention relates to the field of semi-supervised learning technology, and in particular to a three-stage semi-supervised voiceprint recognition method based on an adaptive augmentation strategy. Background Technology

[0002] Voiceprint recognition, as a biometric authentication technology, aims to extract features from raw audio to distinguish different speakers. This technology has wide and important applications in security authentication, criminal investigation, and information identification. Most voiceprint recognition research relies on a sufficient amount of labeled data. However, when labeled data resources are very limited, deep learning models face performance degradation and severe overfitting. Semi-supervised learning can achieve better performance using only a small amount of labeled data and a large amount of unlabeled data. Domestic and international scholars have conducted in-depth research on semi-supervised learning and voiceprint recognition methods based on semi-supervised learning. Among them, the most well-known and effective semi-supervised learning methods and voiceprint recognition methods based on semi-supervised learning in the existing literature mainly include: 1. Fixed Matching Method: A simplified semi-supervised learning method with consistency and confidence: In 2020, Sohn K, Berthelot D, Carlini N, et al. FixMatch: Simplifying semi-supervised learning with consistency and confidence. Journal of Advances in neural information processing systems, 2020, 33: 596-608. This paper proposes a simplified and efficient semi-supervised learning method that uses consistency regularization and confidence thresholding to screen pseudo-labels in semi-supervised learning, achieving good performance. 2. Acoustic Embedding Method for Semi-Supervised Speaker Recognition Based on Cosine Distance Virtual Adversarial Training: In 2020, Kreyssig, Florian L, Woodland, Philip C. Cosine-distance virtual adversarial training for semi-supervised speaker-discriminative acoustic embeddings[C] / / 21st Annual Conference of the International Speech Communication Association, Shanghai, China, 2020:3241-3245. This paper proposes a cosine distance virtual adversarial training method, which is a variant of virtual adversarial training. It uses cosine distance to measure the robustness of speaker embeddings to input perturbations, thereby improving the accuracy of speaker recognition.3. Semi-supervised contrastive learning based on generalized contrastive loss and its application to speaker recognition: In 2020, Inoue N, Goto K. Semi-supervised contrastive learning with generalized contrastive loss and its application to speaker recognition[C] / / 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC).IEEE, 2020: 1641-1646. This paper proposes a semi-supervised contrastive learning method using generalized contrastive loss, which combines the losses of supervised metric learning and unsupervised contrastive learning, and achieves good recognition accuracy in speaker recognition. 4. Semi-supervised speaker identification based on graph label propagation: In 2021, Chen Long, Ravichandran, Venkatesh, Stolcke, and Andreas proposed a graph-based label propagation algorithm for semi-supervised speaker identification [C] / / 22nd Annual Conference of the International Speech Communication Association, Brno, Czech republic, 2021:2583-2587. This method effectively utilizes unlabeled data and improves the accuracy of speaker identification. 5. Flexible Matching Method: Enhancing Semi-Supervised Learning with Curriculum Pseudo-Labels: In 2021, Zhang B, Wang Y, Hou W, et al. FlexMatch: Boosting semi-supervised learning with curriculum pseudo-labeling. Journal of Advances in Neural Information Processing Systems, 2021, 34: 18408-18419. This paper proposes a curriculum pseudo-labeling method. Based on the fixed matching method, it considers the different learning difficulties of different categories of data, studies the confidence thresholds for different categories, and experimentally verifies that the proposed method further improves the performance of the model.6. Semi-Supervised Learning Method for Multi-Speaker Meeting Data Based on Graph Convolutional Networks: Tong F, Zheng S, Zhang M, et al. Graph Convolutional NetworkBased Semi-Supervised Learning on Multi-Speaker Meeting Data[C] / / ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP). IEEE, 2022: 6622-6626. This paper proposes a semi-supervised learning method based on graph convolutional networks. Given a pre-trained embedding extractor, a graph convolutional network is trained on labeled data, and unlabeled data is clustered using "pseudo-labels." Based on this, a self-correcting training mechanism is proposed, which iteratively runs the clustering training correction process on the pseudo-labels. Experimental results show that this method effectively utilizes unlabeled data and improves the accuracy of speaker recognition. Summary of the Invention

[0003] The purpose of this invention is to provide a three-stage semi-supervised voiceprint recognition method based on an adaptive expansion strategy that has both high accuracy and high utilization in selecting unlabeled data.

[0004] To achieve the above objectives, the present invention provides the following solution:

[0005] (1) A semi-supervised learning method for constructing a three-stage framework (TSF) is proposed;

[0006] (1.1) Unify contrastive learning, supervised learning, and semi-supervised learning into a three-stage semi-supervised learning framework;

[0007] (1.2) Utilize the model’s own classification ability to predict the initial confidence threshold after the second stage and carry out the third stage of semi-supervised learning;

[0008] (2) A three-stage (Three-Stage Framework with AdaptiveExpansion Strategy, TSF-AES) semi-supervised voiceprint recognition method based on an adaptive expansion strategy is proposed;

[0009] (2.1) Predict the baseline threshold using unlabeled data that was not selected by the confidence threshold;

[0010] (2.2) Based on the semi-supervised learning method based on TSF, the Adaptive Expansion Strategy (AES) is used to adaptively adjust the confidence threshold in combination with the baseline threshold according to the performance of the model, so as to reasonably expand the amount of unlabeled data selected and further improve the voiceprint recognition performance of the model.

[0011] (3) Complete the training and testing of the three-stage semi-supervised voiceprint recognition method based on the adaptive augmentation strategy;

[0012] (3.1) Train a semi-supervised voiceprint recognition model using the TSF-AES-based semi-supervised voiceprint recognition method;

[0013] (3.2) Complete the performance test of the trained model.

[0014] The aforementioned step (1.1) specifically includes:

[0015] Strong and weak data augmentation are performed on labeled and unlabeled data. Strong data augmentation involves adding music, vocals, noise, or reverberation impulse responses to the original audio and performing time-frequency domain augmentation after feature extraction. Weak data augmentation performs time-frequency domain augmentation only after feature extraction. Time-domain augmentation is represented as... ,in, Indicates time The signal below, and The start and end times of data augmentation are represented in the time domain, and the frequency domain augmentation is represented as... ,in, Indicates frequency band The signal below, and This indicates the start and end frequency domains of the data augmentation.

[0016] The aforementioned step (1.1) specifically includes:

[0017] The first stage of contrastive learning is performed using unlabeled voiceprint data. Positive and negative sample pairs are constructed using a contrastive learning method with reinforcement adversarial training. Contrastive learning enables the model to learn the similarity between data. After the contrastive learning is completed, the encoder parameters are obtained for the second stage.

[0018] The aforementioned step (1.1) specifically includes:

[0019] Based on the first-stage comparative learning using labeled data, a second-stage supervised learning is performed to improve the model's classification ability. When the model's performance approaches stability, the second-stage supervised learning ends, and the encoder parameters are obtained for the third stage.

[0020] The aforementioned step (1.1) specifically includes:

[0021] Building upon the second stage of supervised learning, a third stage of semi-supervised learning is performed. The weakly augmented unlabeled data is processed by an encoder to obtain a probability vector distribution. Samples with a probability maximum greater than a confidence threshold are used as pseudo-labels based on their predicted labels. The loss is then calculated for the strongly augmented data. ,in, This represents an unlabeled dataset. This indicates unlabeled data. Indicates encoder, This represents the probability vector distribution obtained by the encoder from weakly augmented unlabeled data. This indicates strongly augmented unlabeled data. Indicates the confidence threshold. The loss represents the normalized exponential function loss (Additive Angular Margin softmax, AAM-softmax) with additional angular boundaries, and is obtained through supervised learning by calculating the loss using labeled data and its true labels. ,in, This indicates that there is a labeled dataset. This indicates that there is labeled data. Representing the true label of the data, using total loss Conduct semi-supervised learning.

[0022] Step (1.2) specifically includes:

[0023] Based on the second stage of supervised learning, labeled data is used to predict an initial threshold through the model's own classification ability, which is then used as the confidence threshold for the third stage. The class corresponding to the maximum value of the probability vector distribution of the labeled data is compared with the true label, and the average of the maximum probabilities corresponding to the same data is used as the initial confidence threshold for the third stage. This confidence threshold can be used to filter the confidence of unlabeled data based on the current model's classification ability.

[0024] Step (2.1) specifically includes:

[0025] Unlabeled data is categorized into selected and unselected based on a confidence threshold. For the unselected unlabeled data, weak data augmentation is performed, and an encoder is used to obtain a probability vector distribution. The average of the maximum values ​​in the probability vector distribution is used as the baseline threshold. ,in, Indicates the first The bottom line threshold for each epoch. This represents the amount of unlabeled data that was not selected. This represents the total amount of unlabeled data. Indicates the first The confidence threshold for each epoch.

[0026] Step (2.2) specifically includes:

[0027] The third stage utilizes the proposed adaptive expansion strategy, which adaptively adjusts the confidence threshold based on the model's performance and the baseline threshold to increase the amount of unlabeled data selected. When the model's performance stabilizes, the confidence threshold is adaptively adjusted based on the initial threshold, the baseline threshold, the current number of training rounds, and the total number of training rounds. This allows the model to gradually increase the amount of unlabeled data selected while ensuring high-quality unlabeled data, thereby further improving the utilization rate of unlabeled data.

[0028] The aforementioned step (3.1) specifically includes:

[0029] The TSF-AES-based semi-supervised voiceprint recognition method and the voiceprint recognition model (Emphasized ChannelAttention, Propagation and Aggregation in TDNN, ECAPA-TDNN) are combined, and the semi-supervised voiceprint recognition model is trained using the voiceprint dataset.

[0030] Step (3.2) specifically includes:

[0031] The trained model was tested using a voiceprint dataset. The voiceprint recognition results were verified by the equal error rate and the quality and quantity of pseudo-labels corresponding to the selected unlabeled data. The three-stage semi-supervised voiceprint recognition task based on the adaptive augmentation strategy was completed.

[0032] The beneficial effects of this invention are as follows:

[0033] (1) Traditional semi-supervised learning methods are all derived from the field of image recognition, such as FixMatch and FlexMatch. Currently, most research on semi-supervised voiceprint recognition methods is based on graphs, which require a lot of computational resources and time to build the graph. This invention combines the semi-supervised learning idea applicable to the field of image recognition with contrastive learning and supervised learning, and applies it to the field of voiceprint recognition, proposing a three-stage semi-supervised voiceprint recognition method based on an adaptive expansion strategy. (2) This invention proposes a semi-supervised learning method for constructing TSF, unifying contrastive learning, supervised learning, and semi-supervised learning in a three-stage learning framework. In the contrastive learning stage, the model learns the similarity between data. In the supervised learning stage, the model improves its classification ability through labeled data. After supervised learning, the model predicts the initial threshold through its own classification ability to filter out high-quality unlabeled data in the third stage of semi-supervised learning. In the semi-supervised learning stage, the model uses the initial threshold as the confidence threshold for semi-supervised learning. (3) This invention proposes a semi-supervised voiceprint recognition method based on TSF-AES. On the basis of TSF, an adaptive expansion strategy is introduced. The bottom line threshold is predicted by unlabeled data that is not selected by the confidence threshold. Then, the confidence threshold is dynamically reduced by combining the initial threshold and the bottom line threshold according to the performance of the model itself. This allows the model to gradually increase the amount of unlabeled data selected while ensuring high-quality unlabeled data, thereby further improving the performance of the model. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart of the method of the present invention;

[0036] Figure 2 This is a structural diagram of the semi-supervised learning method for TSF proposed in this invention;

[0037] Figure 3 This is a structural diagram of the TSF-AES-based semi-supervised voiceprint recognition method proposed in this invention;

[0038] Figure 4 This is a comparison chart showing the quality and quantity of pseudo-labels selected by confidence thresholds on the ECAPA-TDNN model when the labeled data accounts for 2.2% of the total data. (This is based on semi-supervised learning methods such as FixMatch, FlexMatch, TSF, and TSF-AES.) Figure 4(a) in the figure is a comparison chart of the number of selected pseudo-labels. Figure 4 (b) in the figure is a comparison chart of the selected pseudo-label quality;

[0039] Figure 5 This is a comparison chart showing the quality and quantity of pseudo-labels selected by confidence thresholds on the ECAPA-TDNN model when the labeled data accounts for 5.5% of the total data. (This is based on semi-supervised learning methods such as FixMatch, FlexMatch, TSF, and TSF-AES.) Figure 5 (a) in the figure is a comparison chart of the number of selected pseudo-labels. Figure 5 (b) in the figure is a comparison chart of the selected pseudo-label quality;

[0040] Figure 6 This is a comparison chart showing the quality and quantity of pseudo-labels selected by confidence thresholds on the ECAPA-TDNN model when labeled data accounts for 11%. The comparison chart utilizes semi-supervised learning methods based on FixMatch, FlexMatch, TSF, and TSF-AES. Figure 6 (a) in the figure is a comparison chart of the number of selected pseudo-labels. Figure 6 (b) in the figure is a comparison chart of the selected pseudo-label quality;

[0041] Figure 7 This is a comparison chart showing the quality and quantity of pseudo-labels selected by semi-supervised learning methods (FixMatch, FlexMatch, TSF, and TSF-AES) on the ECAPA-TDNN model, based on confidence thresholds, when labeled data accounts for 22% of the total data. Figure 7 (a) in the figure is a comparison chart of the number of selected pseudo-labels. Figure 7 (b) in the figure is a comparison chart of the selected pseudo-label quality;

[0042] Figure 8 This is a comparison chart showing the quality and quantity of pseudo-labels selected by confidence thresholds on the ECAPA-TDNN model when labeled data accounts for 33%. The comparison charts use semi-supervised learning methods based on FixMatch, FlexMatch, TSF, and TSF-AES. Figure 8 (a) in the figure is a comparison chart of the number of selected pseudo-labels. Figure 8 (b) in the figure is a comparison chart of the selected pseudo-label quality;

[0043] Figure 9 This is a comparison chart showing the same error rate of the model in the second stage of supervised learning and the third stage of semi-supervised learning, with and without the first stage of contrastive learning. Figure 9 (a) in the figure is a comparison chart of error rates for the second-stage supervised learning model, etc. Figure 9 (b) in the figure is a comparison of error rates of the third-stage semi-supervised learning model, etc. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Combination Figure 1 The specific steps of this invention are as follows:

[0047] (1) Propose a semi-supervised learning method for constructing TSF.

[0048] This training framework unifies contrastive learning, supervised learning, and semi-supervised learning into a three-stage training system. The first stage, contrastive learning, employs an adversarial training method using unlabeled data to train the model and teach it the similarities between data points. The second stage, supervised learning, uses labeled data to enhance the model's classification ability. After the second stage, the model uses labeled data to predict an initial threshold suitable for the dataset, which serves as the confidence threshold. In the third stage, semi-supervised learning is performed. The confidence threshold obtained in the second stage is used to filter weakly enhanced unlabeled data, and the loss is calculated between the pseudo-labels predicted by the selected unlabeled data and their strongly enhanced samples. For labeled data, supervised learning is then applied. Figure 2 This is a structural diagram of the semi-supervised learning method TSF proposed in this invention.

[0049] (1.1) Preprocessing and data augmentation of raw audio data

[0050] This invention uses the training set of the large, publicly available voiceprint dataset Voxceleb2 for training, which contains 1,092,009 voiceprint data points provided by 5,994 speakers. First, the data in the training set is divided. For each class, 4, 10, 20, and 40 data points are selected as labeled data. Additionally, 33% of the data in each class is selected as labeled data, and all voiceprint data is treated as unlabeled data. To ensure consistent measurement, the percentage of labeled data is used to represent different labeled data amounts: 2.2%, 5.5%, 11%, 22%, and 33%.

[0051] Strong and weak data augmentation are performed on labeled and unlabeled data. Strong data augmentation involves adding music, human voices, noise, or reverberation impulse responses to the original audio and performing time-frequency domain augmentation after feature extraction. Weak data augmentation only performs time-frequency domain augmentation after feature extraction and does not perform any processing before feature extraction.

[0052] FBank features are extracted from the voiceprint signal, and then pre-emphasis, framing, windowing, and fast Fourier transform are performed. After passing through the Mel filter bank, the logarithmic filter bank is used for calculation to obtain the FBank features of the voice signal.

[0053] To further increase the learning difficulty and improve robustness of the model, time-domain and frequency-domain augmentation is performed on the FBank features of all data. A rectangular window is used to perform continuous-step time-domain and frequency-domain masking on the FBank features; specifically, the sampling points within the mask are set to 0. The FBank feature matrix after feature extraction is as follows: ,in Indicates frequency, Representing time, through the time range of the feature matrix. and frequency range The enhancement strategy involves randomly selecting a rectangular window for masking, and its specific representation is as follows:

[0054]

[0055] (1.2) Semi-supervised learning method of TSF

[0056] A three-stage semi-supervised learning method is adopted to learn from the processed data. ECAPA-TDNN is used as the voiceprint recognition model. First, the unlabeled data is compared and learned by reinforcement adversarial training. For each audio, two non-overlapping segments are randomly divided. Two segments of the same audio are paired, and any two segments of different audio are paired. At the same time, different data augmentations are applied to the same audio segment to encourage the model to be discriminative of voiceprint information and to be invariant to the applied augmentations. Indicates the first The first audio data Add the first segment The embedding vector obtained by the encoder after enhancement. Its prototype loss. Specifically, it is expressed as follows:

[0057]

[0058]

[0059]

[0060] In the formula, Indicates the number of unlabeled data. express and cosine similarity, and These are learnable weights and biases.

[0061] Reinforcement adversarial training assumes that two segments from the same audio data have the same channel characteristics under the same type of data augmentation, but different channel characteristics under different types of data augmentation. This method uses a binary cross-entropy loss model to classify whether two inputs belong to the same channel. To weaken the ability to correctly predict whether two audio segments have the same channel characteristics, reinforcement adversarial training also adds a gradient inversion layer between the embedding extractor and the classifier. Its loss function... Represented as:

[0062]

[0063] In the formula, It is the Sigmoid function. It is the classifier for the model.

[0064] The total loss function for reinforced adversarial training is specifically expressed as:

[0065]

[0066] The first-stage contrastive learning model learned the similarity between data points, but it didn't acquire sufficient classification ability, preventing it from selecting high-quality pseudo-labels based on a confidence threshold. The second-stage supervised learning requires the model to acquire a relatively high confidence threshold suitable for the voiceprint dataset while simultaneously developing sufficient classification ability. This invention uses labeled data as the starting point; the model improves its classification ability by learning from labeled data and predicts an initial threshold when it has sufficient classification ability. The second-stage supervised learning is performed using labeled data based on the first-stage contrastive learning. When the model's error rate doesn't decrease over four time steps, the average of the maximum values ​​of the probability vector distributions of the weakly augmented labeled data is used as the initial threshold. Since the second stage requires a sufficiently high initial threshold to ensure high-quality pseudo-labels for the selected unlabeled data, data where the predicted pseudo-labels differ from the true labels are removed when calculating the initial threshold. The probability information from these errors lowers the initial threshold. Initial Threshold The mathematical expression for the calculation is as follows:

[0067]

[0068] In the formula, This indicates that the pseudo-labels and real labels have the same amount of data. This represents the total amount of labeled data. This represents the probability vector distribution of the weakly augmented labeled data after passing through the encoder. This is an indicator function; it returns 1 if the condition is true, and 0 otherwise. This is the true label for the data.

[0069] The third stage involves semi-supervised learning. The probability vector distribution of the weakly augmented unlabeled data is obtained through the encoder. Pseudo-labels are predicted for data whose maximum value is greater than the initial threshold, and the loss is calculated with the corresponding strong augmentation results. The mathematical expression of the loss function for unlabeled data is as follows:

[0070]

[0071] In the formula, This represents an unlabeled dataset. This indicates unlabeled data. Indicates encoder, This represents the probability vector distribution obtained by the encoder from weakly augmented unlabeled data. This indicates strongly augmented unlabeled data. This represents the loss of the normalized exponential function (Additive Angular Margin softmax, AAM-softmax) with additional angular boundaries.

[0072] Supervised learning is performed on labeled data based on its true labels, and the loss function is specifically expressed as follows:

[0073]

[0074] In the formula, This indicates that there is a labeled dataset. This indicates that there is labeled data.

[0075] The third stage utilizes total losses Conduct semi-supervised learning.

[0076] (2) A semi-supervised voiceprint recognition method based on TSF-AES is proposed.

[0077] The TSF-based semi-supervised learning method predicts an initial threshold in the second stage of supervised learning and uses it as the confidence threshold for the third stage of semi-supervised learning. While a fixed confidence threshold can select sufficiently high-quality unlabeled data, it does not adequately utilize the unlabeled data. To further improve the model's utilization of unlabeled data, AES is proposed. This method adaptively adjusts the confidence threshold based on the model's performance, combining it with the baseline threshold to reasonably expand the amount of unlabeled data selected and further improve the model's voiceprint recognition performance. Figure 3 This is a structural diagram of the TSF-AES-based semi-supervised voiceprint recognition method proposed in this invention.

[0078] (2.1) Obtaining the bottom line threshold

[0079] This invention, based on a TSF-based semi-supervised learning method, employs an adaptive augmentation strategy to augment unlabeled data. Unlabeled data not selected by the confidence threshold undergoes weak augmentation, and an encoder is used to obtain a probability vector distribution. The average of the maximum values ​​in the probability vector distribution is then used as the baseline threshold. The calculation formula is as follows:

[0080]

[0081] In the formula, Indicates the first The bottom line threshold for each epoch. This represents the amount of unlabeled data that was not selected. For the first The confidence threshold for each epoch.

[0082] (2.2) Adaptive expansion strategy

[0083] The current confidence threshold is approached towards the lower threshold when the model's error rate no longer decreases over approximately four time steps. This lower threshold is clearly lower than the current confidence threshold and changes dynamically with model training. AES employs a linear descent-like method to buffer this, ensuring a slow increase in the number of unlabeled data selected. This invention achieves optimal recognition performance even with a slight decrease in accuracy as the model selects more unlabeled data, achieving both high accuracy and high utilization in unlabeled data selection. Specifically, AES is expressed as follows:

[0084]

[0085] In the formula, As the number of training epochs increases linearly from 0 to 1, Indicates the first The bottom-line threshold predicted in each epoch.

[0086] (3) Complete the training and testing of the TSF-AES-based semi-supervised voiceprint recognition method.

[0087] This invention uses the test set in Voxceleb1 for testing, which includes 4874 audio data points provided by 40 speakers, and 37611 test items formed by randomly combining these 4874 audio points, to verify the model's recognition performance. This invention combines semi-supervised learning methods based on Mean Teacher, Pseudo Label, FixMatch, FlexMatch, and SimCLRv2, as well as methods using all data for supervised learning, with the voiceprint recognition model ECAPA-TDNN. These methods are compared with the method proposed in this invention, using the equal error rate (EER) as the evaluation metric. In semi-supervised learning, since the quality and quantity of pseudo-labels selected for the confidence threshold are also important observation metrics, the quality and quantity of the selected pseudo-labels are included in the evaluation metrics.

[0088] The experiment tested each item, which consisted of two audio clips selected from the test set, labeled with whether the two clips belonged to the same speaker (1 for the same speaker, 0 for the other). The test data was encoded into embedding vectors by an encoder, and the cosine similarity between the two embedding vectors was calculated as a score. Suppose that the two embedding vectors are respectively and , The calculation formula is as follows:

[0089]

[0090] The model is based on The size of the audio clips determines whether they belong to the same speaker. Table 1 shows the statistics of the predicted and actual results:

[0091] Table 1

[0092] Predicted Results Actual Results 0 1 0 Correct assessment; they do not belong to the same speaker (TN). Incorrect judgment; speakers do not belong to the same speaker (FN). 1 Incorrect judgment; belongs to the same speaker (FP). The judgment is correct; they belong to the same speaker (TP).

[0093] The equal error rate (EER) is built upon the false acceptance rate and the false rejection rate. The false acceptance rate (FAR) represents the probability that an acceptance that should not have been made actually belongs to the same speaker, and it is expressed as follows:

[0094]

[0095] False rejection rate (FRR) represents the number of samples that should not have been rejected but were actually predicted not to belong to the same speaker. Specifically, it is expressed as follows:

[0096]

[0097] The equal error rate represents the value at which the false acceptance rate and the false rejection rate are equal. The smaller this value, the better. Its specific expression is as follows:

[0098]

[0099] The quality of pseudo-labels represents the proportion of pseudo-labels predicted from unlabeled data selected based on the confidence threshold that match the actual labels. Specifically, it is expressed as follows:

[0100]

[0101] In the formula, This represents the unlabeled data selected based on the confidence threshold. This indicates the amount of unlabeled data selected by the confidence threshold.

[0102] The number of pseudo-labels represents the ratio of the amount of unlabeled data selected for the confidence threshold to the total amount of data, specifically expressed as:

[0103]

[0104] Table 2 shows the comparison of the equal error rates (%) of semi-supervised learning methods based on Mean Teacher, PseudoLabel, FixMatch, FlexMatch, and SimCLRv2, as well as methods using all data for supervised learning, on the ECAPA-TDNN model when the labeled data percentage is 2.2%, 5.5%, 11%, 22%, and 33%. Under different labeled data amounts, the proposed TSF-AES-based semi-supervised voiceprint recognition method achieves the best performance compared to the other semi-supervised voiceprint recognition methods mentioned above, and its performance is similar to that of the voiceprint recognition method using all data for supervised learning. When the labeled percentage is 2.2%, the error rate of the TSF-AES-based method decreases from 2.19% to 1.95% compared to the TSF method, representing an 11% performance improvement. The proposed TSF-AES-based method outperforms other semi-supervised learning methods by at least 45.4%. When the labeled percentage is 5.5%, the error rate of the TSF-AES-based method decreases from 2.17% to 1.66% compared to the TSF method, representing a 23.5% performance improvement. The proposed TSF-AES-based method outperforms other semi-supervised learning methods by at least 26.9%. When the labeled percentage is 11%, the error rate of the TSF-AES-based method decreases from 1.82% to 1.59% compared to the TSF method, representing a 12.6% performance improvement. The proposed TSF-AES-based method outperforms other semi-supervised learning methods by at least 45.4%. The SF-AES method improves performance by at least 31.6% compared to the other semi-supervised learning methods mentioned above. When the labeled percentage is 22%, the error rate of the TSF-AES-based method decreases from 1.45% to 1.32% compared to the TSF method, resulting in a performance improvement of 9.0%. The proposed TSF and TSF-AES-based methods improve performance by at least 44.0% compared to the other semi-supervised learning methods mentioned above. When the labeled percentage is 33%, the error rate of the TSF-AES-based method decreases from 1.48% to 1.22% compared to the TSF method, resulting in a performance improvement of 17.6%. Although the error rate of the proposed TSF method is slightly higher than that of the FlexMatch method, the error rate of the proposed TSF-AES-based semi-supervised learning method is lower than that of FlexMatch, decreasing by 0.02%.

[0105] Table 2

[0106]

[0107] To demonstrate that the TSF-AES-based semi-supervised voiceprint recognition method proposed in this invention can select unlabeled data with better quality and quantity through a confidence threshold, the quality and quantity of pseudo-labels selected by the TSF-AES, TSF, FixMatch, and FlexMatch semi-supervised voiceprint recognition methods are analyzed for different proportions of labeled data, with 0.000351 used as the confidence threshold for FixMatch and FlexMatch methods. Figure 4 This is a comparison chart showing the quality and quantity of pseudo-labels selected by semi-supervised learning methods based on FixMatch, FlexMatch, TSF, and TSF-AES on the ECAPA-TDNN model when the labeled percentage is 2.2%. Figure 4 (a) in the figure is a comparison chart of the number of selected pseudo-labels. Figure 4 (b) in the figure is a comparison chart of the quality of the selected pseudo-labels. Figure 5 This is a comparison chart showing the quality and quantity of pseudo-labels selected by semi-supervised learning methods based on FixMatch, FlexMatch, TSF, and TSF-AES on the ECAPA-TDNN model when the labeled percentage is 5.5%. Figure 5 (a) in the figure is a comparison chart of the number of selected pseudo-labels. Figure 5 (b) in the figure is a comparison chart of the quality of the selected pseudo-labels. Figure 6 This is a comparison chart showing the quality and quantity of pseudo-labels selected by semi-supervised learning methods based on FixMatch, FlexMatch, TSF, and TSF-AES on the ECAPA-TDNN model when the labeled percentage is 11%. Figure 6 (a) in the figure is a comparison chart of the number of selected pseudo-labels. Figure 6 (b) in the figure is a comparison chart of the quality of the selected pseudo-labels. Figure 7 This is a comparison chart showing the quality and quantity of pseudo-labels selected by semi-supervised learning methods based on FixMatch, FlexMatch, TSF, and TSF-AES on the ECAPA-TDNN model when the labeled percentage is 22%. Figure 7 (a) in the figure is a comparison chart of the number of selected pseudo-labels. Figure 7 (b) in the figure is a comparison chart of the quality of the selected pseudo-labels. Figure 8 This is a comparison chart showing the quality and quantity of pseudo-labels selected by semi-supervised learning methods based on FixMatch, FlexMatch, TSF, and TSF-AES on the ECAPA-TDNN model when the labeled percentage is 33%. Figure 8 (a) in the figure is a comparison chart of the number of selected pseudo-labels. Figure 8(b) in the figure shows a comparison of the quality of the selected pseudo-labels. Regarding the number of pseudo-labels, the FixMatch and FlexMatch methods exhibit two extremes. Under different amounts of labeled data, the FixMatch method selects relatively few pseudo-labels based on the confidence threshold, with the highest selection being less than 10% of the total data. The FlexMatch method, through its course pseudo-labeling algorithm, selects almost all unlabeled data each time. The TSF-based semi-supervised learning method constructed in this invention selects a moderate number of pseudo-labels that increases slowly. The TSF-AES-based semi-supervised voiceprint recognition method proposed in this invention shows a phased increase in the number of pseudo-labels selected, with each significant increase indicating a change in the confidence threshold. Ultimately, the number of selected pseudo-labels reaches over 85%, a significant improvement compared to the TSF method. Regarding the quality of pseudo-labels, the FixMatch method, using a high confidence threshold, exhibits fluctuating quality in the early stages of model training. In most cases, it gradually stabilizes and approaches 100% in the later stages. However, when labeled data accounts for 33%, the quality of the selected pseudo-labels only reaches 70%, indicating that the FixMatch method cannot enable the model to learn sufficient classification ability, thus failing to select sufficiently high-quality pseudo-labels based on the confidence threshold. The FlexMatch method, because it selects almost all unlabeled data in the early stages of model training, generally has pseudo-label quality close to 0. When labeled data accounts for 33%, the model has more labeled data to learn from, and the quality of the selected pseudo-labels gradually increases from 0, eventually reaching 95%. This indicates that the FlexMatch method cannot guarantee the quality of the selected pseudo-labels. The TSF semi-supervised learning method constructed in this invention predicts the confidence threshold suitable for the dataset based on the model's own performance. Furthermore, it can select pseudo-labels with near 100% accuracy in the third stage of semi-supervised learning. The TSF-AES-based semi-supervised voiceprint recognition method proposed in this invention expands the selection of unlabeled data by progressively lowering the confidence threshold through an adaptive expansion strategy. While the quality of the selected pseudo-labels decreases slightly with adaptive expansion, the accuracy remains close to 100%. This demonstrates that the quality of the selected pseudo-labels in the TSF-AES-based semi-supervised voiceprint recognition method does not suffer as the number of pseudo-labels increases. This verifies that the invention achieves both high accuracy and high utilization in the selection of unlabeled data, thus reaching optimal performance.

[0108] To further verify the necessity of the first-stage contrastive learning for this invention, ablation experiments were conducted with label percentages of 2.2%, 5.5%, and 11% to verify the model's performance in the second-stage supervised learning and the third-stage semi-supervised learning, as well as the model training speed in the second-stage supervised learning, with and without the first-stage contrastive learning. Figure 9This is a comparison chart showing the same error rate of the model in the second stage of supervised learning and the third stage of semi-supervised learning, with and without the first stage of contrastive learning. Figure 9 (a) in the figure is a comparison chart of error rates for the second-stage supervised learning model, etc. Figure 9 Table (b) shows the comparison of the equal error rates of the third-stage semi-supervised learning model. Without the first-stage contrastive learning, the equal error rates of both the second and third-stage models are higher than those with contrastive learning. When the labeled data percentage is 2.2%, without the first-stage contrastive learning, the model's performance cannot be further improved during the third-stage semi-supervised learning process. Table 3 shows the number of epochs required for the second-stage supervised learning training when the labeled data percentage is 2.2%, 5.5%, and 11%, with and without the first-stage contrastive learning. Without the first-stage contrastive learning, the speed of the second-stage supervised learning is slower than or equal to that with the contrastive learning stage, and the model's equal error rate is also higher. When the labeled data is small, the absence of the contrastive learning stage prevents the model from learning sufficient classification ability in the second-stage supervised learning, thus preventing the third-stage semi-supervised learning from further improving performance. Therefore, the first-stage contrastive learning is necessary for the method proposed in this invention.

[0109] Table 3

[0110] Comparison of the proportion of labeled data during the learning phase 2.2% 5.5% 11% none 26 51 24 have 26 21 16

[0111] In summary, the three-stage semi-supervised voiceprint recognition method based on an adaptive expansion strategy proposed in this invention can achieve superior performance using only a small amount of labeled data and a large amount of unlabeled data, and can effectively complete the voiceprint recognition task.

[0112] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A three-stage semi-supervised voiceprint recognition method based on adaptive expansion strategy, characterized in that, The method comprises the steps of: (1) proposing a three-stage semi-supervised learning method for constructing a framework; (1.1) unifying contrastive learning, supervised learning and semi-supervised learning in a three-stage semi-supervised learning framework; (1.2) predicting an initial confidence threshold after the second stage based on the classification ability of the model, and performing semi-supervised learning in the third stage; (2) proposing a three-stage semi-supervised voiceprint recognition method based on an adaptive expansion strategy; (2.1) predicting a bottom threshold using unlabeled data not selected by the confidence threshold; (2.2) based on the semi-supervised learning method based on the three-stage framework, using an adaptive expansion strategy to adjust the confidence threshold in combination with the bottom threshold according to the performance of the model to reasonably expand the selection amount of unlabeled data and further improve the voiceprint recognition performance of the model; (3) training and testing the three-stage semi-supervised voiceprint recognition method based on the adaptive expansion strategy; (3.1) training a semi-supervised voiceprint recognition model using the three-stage semi-supervised voiceprint recognition method based on the adaptive expansion strategy; (3.2) testing the performance of the trained model; The step (2.2) specifically comprises: In the third stage, the adaptive expansion strategy is used to adjust the confidence threshold in combination with the bottom threshold according to the performance of the model to increase the selection amount of unlabeled data. When the performance of the model tends to be stable, the confidence threshold is adjusted adaptively according to the initial threshold, the bottom threshold, the current training round and the total training round, so that the model gradually increases the selection amount of unlabeled data under the premise of ensuring high-quality unlabeled data, and further improves the utilization rate of unlabeled data.

2. The three-stage semi-supervised voiceprint recognition method based on adaptive expansion strategy according to claim 1, characterized in that, The step (1.1) specifically comprises: Data strong enhancement and data weak enhancement are performed on the labeled data and unlabeled data, wherein the data strong enhancement is adding music, human voice, noise or reverberation impulse response in the original audio, and performing time domain-frequency domain enhancement after feature extraction, and the data weak enhancement is only performing time domain-frequency domain enhancement after feature extraction, and the time domain enhancement is represented as wherein represents a signal at time , and represent the start and end time domain of data enhancement, and the frequency domain enhancement is represented as wherein represents a signal at frequency band , and represent the start and end frequency domain of data enhancement.

3. The three-stage semi-supervised voiceprint recognition method based on adaptive expansion strategy according to claim 1, characterized in that, The step (1.1) specifically comprises: In the first stage, contrastive learning is performed using unlabeled voiceprint data. A positive and negative sample pair is constructed using a reinforced adversarial training contrastive learning method. The model learns the similarity between data using contrastive learning. After contrastive learning, the parameters of the encoder are obtained for the second stage.

4. The three-stage semi-supervised voiceprint recognition method based on adaptive expansion strategy according to claim 1, characterized in that, The step (1.1) specifically comprises: In the second stage, supervised learning is performed using labeled data based on the first-stage contrastive learning to improve the classification ability of the model. When the performance of the model tends to be stable, the second-stage supervised learning is ended, and the parameters of the encoder are obtained for the third stage.

5. The three-stage semi-supervised voiceprint recognition method based on adaptive expansion strategy according to claim 1, characterized in that, The step (1.1) specifically comprises: On the basis of the second stage of supervised learning, the third stage of semi-supervised learning is carried out, the weakly enhanced unlabeled data obtains the probability vector distribution through the encoder, the label predicted by the sample with the probability maximum value greater than the confidence threshold is regarded as the pseudo label, and the loss of the strongly enhanced data is calculated wherein, represents an unlabeled data set, represents an unlabeled data, represents an encoder, represents the probability vector distribution obtained by the weakly enhanced unlabeled data through the encoder, represents strongly enhanced unlabeled data, represents a confidence threshold, represents a normalized exponential function loss with an additional angle boundary, and the loss is calculated with the real label of the labeled data for supervised learning wherein, represents a labeled data set, represents a labeled data, represents the real label of the data, and the total loss is used for semi-supervised learning.

6. The three-stage semi-supervised voiceprint recognition method based on adaptive expansion strategy according to claim 1, characterized in that, The step (1.2) specifically comprises: Based on the second-stage supervised learning, the initial threshold is predicted as the confidence threshold of the third stage by using the classification ability of the model itself and labeled data. The class corresponding to the maximum value of the probability vector distribution of the labeled data is compared with the real label. The maximum value corresponding to the same data is averaged as the initial confidence threshold of the third stage. The confidence threshold can be used to screen the confidence of the unlabeled data according to the classification ability of the current model.

7. The three-stage semi-supervised voiceprint recognition method based on adaptive expansion strategy according to claim 5, characterized in that, The step (2.1) specifically comprises: The unlabeled data is divided into selected and unselected according to the confidence threshold, the unselected unlabeled data is weakly enhanced, and a probability vector distribution is obtained through an encoder, and the maximum value in the probability vector distribution is averaged as a bottom threshold, wherein, represents the bottom threshold of the th epoch, represents the amount of unselected unlabeled data, represents the total amount of unlabeled data, represents the confidence threshold of the th epoch.

8. The three-stage semi-supervised voiceprint recognition method based on adaptive expansion strategy according to claim 1, characterized in that, The step (3.1) specifically comprises: The adaptive expansion strategy-based three-stage semi-supervised voiceprint recognition method is combined with a voiceprint recognition model ECAPA-TDNN, and a semi-supervised voiceprint recognition model is trained by using a voiceprint dataset.

9. The three-stage semi-supervised voiceprint recognition method based on adaptive expansion strategy according to claim 1, characterized in that, The step (3.2) specifically includes: The trained model is tested for performance by using a voiceprint dataset, and voiceprint recognition results are verified by equal error rate, and quality and quantity of pseudo-labels corresponding to selected unlabeled data, so as to complete the adaptive expansion strategy-based three-stage semi-supervised voiceprint recognition task.

Images