Unsupervised speaker verification method based on maximum coding space
By adopting an unsupervised method to maximize coding space in speaker verification technology, combined with CAM++ and DINO frameworks for model training and fine-tuning, the problem of model performance degradation in the existing technology is solved, and higher adaptability and performance improvement is achieved.
Patent Information
- Application Number
- CN202411756892.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-02
- Publication Date
- 2025-03-04
AI Technical Summary
The existing speaker verification technology degrades model performance when the training data and test data mismatches, and there is a lack of effective unsupervised learning methods to improve the model's adaptability to new environments.
Unsupervised speaker verification method based on maximizing coding space is adopted, supervised pre-training is performed through the CAM++ network structure, fine-tuning is performed in combination with the DINO teacher-student framework, and the model is optimized using sample screening and loss weighting methods.
On the two evaluation sets of Vox1 and CnCeleb, the equal error rate (EER) is reduced by 4.3% and 5.9% compared with the supervised baseline model, improving the model's adaptability in the new environment.
Smart Images
Figure CN119559951A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of speech processing, and in particular to an unsupervised speaker verification method based on maximizing coding space. Background Art
[0002] With the development of the Internet and artificial intelligence, identity authentication methods have undergone major changes. Traditional identity authentication has various deficiencies and can no longer meet people's needs for security and convenience of authentication systems. Speaker verification systems have advantages such as contactless collection and low cost. Therefore, in order to meet people's needs for safe, efficient and contactless identity authentication methods, further research on voiceprint recognition technology has important practical significance and application value. Voiceprint recognition technology can extract speaker identity information from voice and is widely used in security authentication, voice assistants, smart homes, medical care, equipment monitoring and other fields. Speaker Verification (SV) is an important research direction in the field of voiceprint recognition technology. It can use the registered voice of an existing identity in the process of voice interaction to verify whether the voice to be recognized is the voice of the same speaker without limiting the content of the speech. This has important research value in application scenarios that require identity authentication, such as voice wake-up and security authentication.
[0003] The core problem that needs to be solved in speaker verification is target speaker extraction (TSE), that is, how to use deep neural networks to extract the information that best represents the speaker's identity and apply it to back-end discrimination. The existing challenge is that the mismatch between training data and test data often leads to a decline in model performance. In order to solve this problem, researchers have proposed many designs. For example, researchers use a two-stage method to train the model. First, the speaker model is pre-trained in an unsupervised manner, including contrastive learning and clustering algorithms. Pseudo-labels are generated through these methods, and then supervised fine-tuning is performed using pseudo-labels. However, this method is more dependent on prior information of the data set, which will be affected when the test environment is unknown. The design of the present invention can completely avoid this problem, and can use unlabeled speech to update the model, thereby improving the model's adaptability to new environments.
[0004] However, due to the difficulty in obtaining a large amount of labeled data, coupled with the shortcomings of existing methods, such as the impact of classification redundancy and noise in unsupervised loss, the accuracy of model evaluation has reached a bottleneck. The encoding rate of the feature space can effectively reflect the different categories in each batch of samples, and maximizing the encoding rate of the feature space can effectively solve the classification redundancy problem; the impact of noise will run through the entire model training process, and the extracted speaker embedding can be re-scored at the back end to screen samples, thus minimizing the impact of noisy samples. Summary of the invention
[0005] In view of the shortcomings existing in the above problems, the present invention provides an unsupervised speaker verification method based on maximizing coding space.
[0006] The present invention provides an unsupervised speaker verification method based on maximizing coding space, comprising the following steps:
[0007] Step 1: Acquire audios of several speakers and identity data of the corresponding speakers, wherein the data is recorded or downloaded in different scenarios;
[0008] Step 2: Preprocess the raw data obtained in step 1; divide all audio into source domain data and target domain data according to different scenarios. The source domain data is used for pre-training, and the target domain data is used for fine-tuning the model and experimental evaluation;
[0009] Step 3: Based on the CAM++ network structure, use the source domain dataset to perform supervised pre-training on the network to obtain a pre-trained model;
[0010] Step 4: Fine-tune the model using the DINO-based teacher-student framework, and use the pre-trained model obtained in step 3 to initialize both the teacher model and the student model;
[0011] Step 5: For each speech data used for fine-tuning in step 3, randomly split it into two equal-length non-overlapping speech segments, and input them into the teacher model and student model in step 4 at the same time to obtain different speaker embedding vectors;
[0012] Step 6: Calculate the cosine similarity score for the speaker embedding vector obtained in step 5, and choose whether to filter the sample by comparing it with the confidence threshold;
[0013] Step 7: Obtain the speaker embedding vector after filtering in step 6, and calculate the MSE loss, cosine loss and coding rate loss;
[0014] Step 8: Weight the three losses in step 7, calculate the gradient and pass it back, fix the parameters of the teacher model, and only update the parameters of the student model;
[0015] Step 9: For the updated student model, input each speech used for evaluation in step 3 into the student model to obtain a complete speaker vector;
[0016] Step 10: Input the speaker vector in step 9 into the linear classification layer and then use cosine similarity analysis to get the speaker classification result;
[0017] Furthermore, the specific process of step three is as follows: first, all audio is processed into 3-second sequences, and the sampling rate is set to 16,000, so each sequence contains 48,000 points. Then, 80 groups of filters are used to extract FBank features for each sequence. Then, each audio sequence becomes a two-dimensional array of shape [80, 298], which contains time-frequency information. The pre-trained model can convert each speech into a 192-dimensional speaker embedding vector.
[0018] Furthermore, the specific content of step 4 is to use the teacher model initialized by the pre-trained model as f t , the student model initialized with the pre-trained model is f s .
[0019] Furthermore, the specific process of step 5 is as follows: Assume that the batch size of the speech components used to fine-tune the model is N, where the batch samples are represented by a set X = {x 1 ,x 2 …,x i ,…,x N}, for each sample x i , split it into non-overlapping segments x of 2 seconds in length i1 and x i2 , and input them into the teacher model f as defined in claim 3 t and student model f s In the example above, we get different speaker embeddings. and
[0020]
[0021] Furthermore, the specific process of step 6 is to calculate the similarity score using the speaker embedding obtained as claimed in claim 4:
[0022]
[0023] By comparing the similarity score with the confidence threshold α, we can choose whether to retain the sample:
[0024]
[0025] Save means retaining the sample, and pass means discarding the sample.
[0026] Furthermore, the number of samples screened in step 7 will be less than or equal to the number of samples in step 6; calculate the MSE loss:
[0027]
[0028] Calculate the cosine loss:
[0029]
[0030] Calculate the coding rate loss:
[0031]
[0032] where I∈R 1×192×192 represents the identity matrix, d represents the dimension of the speaker feature 192, ε represents the transformation size and ε>0, Z i ∈R 2×N×192 is the output of the student model, It's Z i The transposed matrix of .
[0033] Furthermore, the loss weighting in step eight can be expressed as:
[0034] L=L MSE +L CS +L TCR ;
[0035] The weighted loss is fed back through the student model to calculate the gradient, and only the parameters of the student model are updated. Repeat steps 4 to 8 to fine-tune the model.
[0036] Furthermore, steps nine and ten are used to extract the speaker embedding vector (192 dimensions) representation of each speech in the evaluation set.
[0037] Compared with the prior art, the present invention has the following advantages:
[0038] Compared with the traditional supervised speaker verification technology, the unsupervised speaker verification method of maximizing the coding space proposed in this invention combines the ideas of unsupervised learning and few-sample learning, and uses the sample screening method for back-end processing. In terms of indicator evaluation, the equal error rate (EER) of the method proposed in this invention on the two evaluation sets of Vox1 and CnCeleb is reduced by 4.3% and 5.9% respectively compared with the supervised baseline model CAM++. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings, wherein:
[0040] Figure 1 It is an unsupervised speaker verification method based on maximizing the encoding space;
[0041] Figure 2 is a flow chart of sample screening;
[0042] Figure 3It is a comparison of the speaker verification performance index values of the embodiment of the present invention and the performance values of the supervised SOTA model in the current field, as well as the comparison with the performance values of other speaker verification methods. The bold part is the performance of the embodiment of the present invention;
[0043] Figure 4 This is an ablation experiment conducted by the present invention for the confidence threshold α in step 6, and the bold part is the performance of the embodiment with the best parameters; DETAILED DESCRIPTION
[0044] The embodiments of the present invention are described in detail below with reference to the accompanying drawings:
[0045] This embodiment discloses Figure 1 As shown, the following steps are included:
[0046] The audios of several speakers and the identity data of the corresponding speakers are obtained, wherein the data is recorded or downloaded in different scenarios.
[0047] The raw data obtained in the previous step is processed and preprocessed; all audio is divided into source domain data and target domain data according to different scenarios. The source domain data is used for pre-training, and the target domain data is used for fine-tuning the model and experimental evaluation.
[0048] Based on the CAM++ network structure, the source domain dataset is used to perform supervised pre-training on the network to obtain a pre-trained model. First, all audio in the training set is processed into 3-second sequences, and the sampling rate is set to 16,000, so each sequence contains 48,000 points. Then, 80 sets of filters are used to extract FBank features for each sequence, so each audio sequence becomes a two-dimensional array of shape [80, 298], which contains time-frequency information. The pre-trained model can convert each speech into a 192-dimensional speaker embedding vector.
[0049] Use the pre-trained model to initialize the teacher model in the DINO framework as f t , use the pre-trained model to initialize the student model in the DINO framework as f s .
[0050] Assume that the batch size of the speech components used to fine-tune the model is N, where the samples in the batch are represented as a set X = {x 1 ,x 2 …,x i ,…,x N}, for each sample x i , split it into non-overlapping segments x of 2 seconds in length i1 and x i2 , and input them into the teacher model f as defined in claim 3 t and student model fs In the example above, we get different speaker embeddings. and
[0051]
[0052] Calculate similarity scores for sample screening:
[0053]
[0054] By comparing the similarity score with the confidence threshold α, we can choose whether to retain the sample:
[0055]
[0056] Save means retaining the sample, and pass means discarding the sample.
[0057] The number of samples after screening will be less than or equal to the number of samples in step 6; calculate the MSE loss:
[0058]
[0059] Calculate the cosine loss:
[0060]
[0061] Calculate the coding rate loss:
[0062]
[0063] where I∈R 1×192×192 represents the identity matrix, d represents the dimension of the speaker feature 192, ε represents the transformation size and ε>0, Z i ∈R 2×N×192 is the output of the student model, It's Z i The transposed matrix of .
[0064] The loss weighting can be expressed as:
[0065] L=L MSE +L CS +L TCR ;
[0066] The weighted loss is fed back through the student model to calculate the gradient, and only the parameters of the student model are updated. Repeat steps 4 to 8 to fine-tune the model.
[0067] The fine-tuned model is used to extract speaker embedding vectors (192 dimensions) for each utterance in the evaluation set.
[0068] The above description is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, various changes, modifications, substitutions and variations can be made to these embodiments without departing from the technical principles of the present invention. These changes, modifications, substitutions and variations should also be regarded as the scope of protection of the present invention.
Claims
1. An unsupervised speaker verification method based on maximizing coding space, characterized in that: The steps include: Step 1: Acquire audios of several speakers and identity data of the corresponding speakers, wherein the data is recorded or downloaded in different scenarios; Step 2: preprocessing the raw data obtained in step 1; All audio is divided into source domain data and target domain data according to different scenarios. The source domain data is used for pre-training, and the target domain data is used for fine-tuning the model and experimental evaluation. Step 3: Based on the CAM++ network structure, use the source domain dataset to perform supervised pre-training on the network to obtain a pre-trained model; Step 4: Fine-tune the model using the DINO-based teacher-student framework, and use the pre-trained model obtained in step 3 to initialize both the teacher model and the student model; Step 5: For each speech data used for fine-tuning in step 3, randomly split it into two equal-length non-overlapping speech segments, and input them into the teacher model and student model in step 4 at the same time to obtain different speaker embedding vectors; Step 6: Calculate the cosine similarity score for the speaker embedding vector obtained in step 5, and choose whether to filter the sample by comparing it with the confidence threshold; Step 7: Obtain the speaker embedding vector after filtering in step 6, and calculate the MSE loss, cosine loss and coding rate loss; Step 8: Weight the three losses in step 7, calculate the gradient and pass it back, fix the parameters of the teacher model, and only update the parameters of the student model; Step 9: For the updated student model, input each speech used for evaluation in step 3 into the student model to obtain a complete speaker vector; Step 10: Input the speaker vector in step 9 into the linear classification layer and then use cosine similarity analysis to get the speaker classification result.
2. The unsupervised speaker verification method based on maximizing coding space according to claim 1, characterized in that: The specific process of step three is as follows: first, all audio is processed into 3-second sequences, and the sampling rate is set to 16,000, so each sequence contains 48,000 points. Then, 80 groups of filters are used to extract FBank features for each sequence. Then, each audio sequence becomes a two-dimensional array with a shape of [80, 298], which contains time-frequency information. The pre-trained model can convert each speech into a 192-dimensional speaker embedding vector.
3. The unsupervised speaker verification method based on maximizing coding space according to claim 1, characterized in that: The specific content of step 4 is to use the teacher model initialized by the pre-trained model as f t , the student model initialized with the pre-trained model is f s .
4. The unsupervised speaker verification method based on maximizing coding space according to claim 3, characterized in that: The specific process of step 5 is as follows: a batch size of speech components used for fine-tuning the model is N, where the batch samples are represented by a set X = {x1, x2…, x i ,…,x N }, for each sample x i , split it into non-overlapping segments x of 2 seconds in length i1 and x i2 , and input them into the teacher model f t and student model f s In the example above, we get different speaker embeddings. and 5. The unsupervised speaker verification method based on maximizing coding space according to claim 4, characterized in that: The specific process of step 6 is to use speaker embedding to calculate the similarity score: By comparing the similarity score with the confidence threshold α, we can choose whether to retain the sample: Save means retaining the sample, and pass means discarding the sample.
6. The unsupervised speaker verification method based on maximizing coding space according to claim 5, characterized in that: The number of samples screened in step 7 will be less than or equal to the number of samples in step 6; calculate the MSE loss: Calculate the cosine loss: Calculate the coding rate loss: where I∈R 1×192×192 represents the identity matrix, d represents the dimension of the speaker feature 192, ε represents the transformation size and ε>0, Z i ∈R 2×N×192 is the output of the student model, It's Z i The transposed matrix of .
7. The unsupervised speaker verification method based on maximizing coding space according to claim 6, characterized in that: The loss weight in step eight can be expressed as: L=L MSE +L CS +L TCR The weighted loss is fed back through the student model to calculate the gradient, and only the parameters of the student model are updated. Repeat steps 4 to 8 to fine-tune the model.
8. The unsupervised speaker verification method based on maximizing coding space according to claim 1, characterized in that: The steps nine and ten are used to extract the speaker embedding vector representation of each speech in the evaluation set.