A text-independent speaker verification method based on identity information and semantic information disentanglement
By employing a two-stream neural network architecture and a mutual information deentanglement method, the problem of the voiceprint recognition model being affected by semantic content is solved, achieving higher recognition accuracy and generalization ability, and making it suitable for speaker verification in complex environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANXI UNIV
- Filing Date
- 2023-06-01
- Publication Date
- 2026-05-19
AI Technical Summary
Existing voiceprint recognition models are easily affected by semantic content during training, resulting in poor generalization performance, especially when tested on different datasets.
A dual-stream neural network architecture is adopted to extract the identity information and semantic content features of speech separately. The mutual information mechanism is used to de-entangle the voiceprint features and semantic features. Voiceprint recognition networks such as ECAPA-TDNN, ResNet, and VGG, as well as speech recognition networks such as Deep Speech2 and Whisper, are used. The model training is optimized by combining cross-entropy loss and de-entanglement methods.
It improves the accuracy and generalization ability of the voiceprint recognition model, effectively distinguishing the speaker's identity in different voice scenarios and reducing the interference of semantic content on recognition.
Smart Images
Figure BDA0004261662910000021 
Figure BDA0004261662910000031 
Figure BDA0004261662910000033
Abstract
Description
Technical Field
[0001] This invention relates to the field of voiceprint recognition (also known as speaker recognition) technology, and in particular to a text-independent speaker identification method based on the deentanglement of identity information and semantic information. Background Technology
[0002] Speaker recognition has been widely applied in tasks such as public security telecommunications fraud prevention, criminal investigation, and mobile payment. Currently, text-independent speaker verification technology has become a research hotspot both domestically and internationally in recent years. Text-independent speaker verification is a complex voiceprint recognition problem that does not rely on specific semantic information, i.e., it does not require limiting the content of the speech, and can determine whether the registered speech and the test speech belong to the same speaker in a noisy environment.
[0003] Because speech signals simultaneously contain both speaker identity and semantic information, existing voiceprint recognition algorithms are often influenced by semantic content during the training process to obtain voiceprint features. For example, the VoxCeleb 2 dataset contains speeches by celebrities from different professional fields. The vocabulary used in these speeches is often closely related to their respective industries; for instance, athletes primarily speak about sports, while politicians primarily speak about politics. Furthermore, models trained on the predominantly English-speaking VoxCeleb 2 dataset perform poorly when tested on the Chinese-speaking Cn-Celeb dataset. This demonstrates that trained voiceprint recognition models may use semantic content as a clue to identify the speaker. While there is inevitably some coupling between identity and semantic information, over-reliance on semantic content can reduce the accuracy of voiceprint recognition models and affect their generalization performance on new data.
[0004] Therefore, separating voiceprint features from semantic information and eliminating the influence of semantic content on the voiceprint recognition model greatly helps improve the accuracy and generalization of the voiceprint recognition model. Therefore, this invention proposes a text-independent speaker verification method based on deentanglement. By simultaneously encoding speaker identity and semantic content, features representing speaker identity information and semantic content are obtained, and a mutual information mechanism is introduced to untangle the two. Summary of the Invention
[0005] To address the technical problem that the voiceprint features extracted by speaker recognition systems are affected by the semantic content of the dataset, resulting in poor generalization performance of the model, this invention provides a text-independent speaker identification method based on the deentanglement of identity information and semantic information.
[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:
[0007] A text-independent speaker identification method based on identity and semantic information deentanglement includes the following steps:
[0008] 1) Building a neural network
[0009] The neural network used is a two-stream network architecture, which simultaneously extracts features representing speaker identity information and semantic content features from a speech segment. A speaker recognition network is used to extract speaker features, and a speech recognition network is used to extract speech content features.
[0010] Among them, the voiceprint recognition network is used for training, and a classification layer is added after the speaker feature layer. The classification dimension is equal to the number of speakers in the training dataset. The speech recognition network does not participate in training and directly uses the pre-trained model of the pre-trained speech recognition network to extract semantic content features. The construction of the neural network is completed.
[0011] 2) Model training
[0012] First, the speech data in the dataset is divided into batches of equal size, each containing K speech data. For each speech data in a batch, a 3-second speech segment is randomly selected and speech enhancement is performed. Then, its spectrogram is calculated, and the Mel-scale Frequency Cepstral Coefficients (MFCC) feature is extracted as the input of the neural network model.
[0013] Then, the MFCC features are input into the speech recognition network to directly obtain the feature representation f of the semantic content. sch ;
[0014] Simultaneously, the MFCC features are input into the voiceprint recognition network to obtain the voiceprint features f. spk ;
[0015] Then, the sum of the cross-entropy losses L between the classification results of the K voiceprint features obtained from a batch of data and the true labels is calculated using the following formula (1). A The classification result is obtained by AAM-softmax calculation, and the true label needs to be processed by one-hot encoding.
[0016]
[0017] In formula (1), cos(θ) j,i )=||W j ||||f spk_i ||cos <W j ,f spk_i >, i.e., f spk_i and W jAfter L2 regularization, the dot product, where i and j are indices in a batch of data, 0 < i < K, 0 < j < K, f spk_i y represents the voiceprint features of the i-th speech. i Let W represent the label of the i-th speech, and W represent the weight matrix of the classification head. The classification result is represented by label y. i The weight parameter W corresponding to the time. j Represents y i The weight parameters of the other classification heads, s, are fixed scaling factors to prevent the gradient from becoming too small during training, through W j and f spk_i An angular margin m is added between them to maximize intra-class similarity and minimize inter-class similarity, aiming to make the distance between similar but different speakers greater.
[0018] Simultaneously, the voiceprint features f obtained by calculating batch speech data using the deentanglement method are used. spk With semantic features f sch The correlation between them yields the loss L. s To achieve deentanglement between voiceprint features and semantic features;
[0019] The unentanglement method described here uses the mutual information method as shown in formula (2).
[0020]
[0021] In formula (2), Given f sch down f spk The true posterior variational approximation can be obtained by the network θ(f) spk ,f sch Parameterization.
[0022] Next, the loss function of the entire training network is obtained through the following formula (3);
[0023] L cls =L A +λL S (3)
[0024] In formula (3), λ is the weight hyperparameter.
[0025] Finally, using all the training data, minimize the loss function L. cls The speaker confirmation model was obtained after training was completed.
[0026] 3) Result Testing
[0027] First, the last layer of the classification layer is removed from the trained model to represent the voiceprint features;
[0028] Then, for the two speech samples to be compared, their full-length spectrograms are extracted, and their MFCC features are extracted. The MFCC features are then input into the model to obtain the speaker signature representations f for the two speech samples. spk1 f spk2 ;
[0029] Finally, the similarity score of the voiceprint features of the two voices to be compared is obtained by using the following formula (4) to determine whether the two voices to be compared belong to the same person.
[0030]
[0031] In formula (4), ||·||2 refers to the second norm.
[0032] The voiceprint recognition network can be any one of the following network models: ECAPA-TDNN, ResNet, Thin-ResNet, or VGG.
[0033] The speech recognition network can be any one of Deep Speech2, Deep Speech, or Whisper network models.
[0034] Compared with the prior art, the beneficial effects of the present invention are:
[0035] By using semantic content with fixed-length features f sch The features are represented and the voiceprint features f are calculated using unentanglement methods such as mutual information. spk With semantic features f sch The correlation between them is investigated to deentangle the voiceprint features and semantic features, resulting in more accurate and robust speaker voiceprint features. This also solves the problem of poor generalization performance of the speaker voiceprint features extracted by the speaker recognition system due to interference from semantic content factors in the dataset. Detailed Implementation
[0036] The text-independent speaker identification method based on identity information and semantic information deentanglement described in this embodiment includes the following steps:
[0037] 1) Building a neural network
[0038] The neural network used is a two-stream network architecture. It simultaneously extracts features representing speaker identity information and semantic content features from a segment of speech. A voiceprint recognition network is used to extract speaker features, which can be an ECAPA-TDNN, ResNet, Thin-ResNet, or VGG network model. A speech recognition network is used to extract speech content features, which can be a Deep Speech2, Deep Speech, or Whisper network model.
[0039] Among them, the voiceprint recognition network is used for training, and a classification layer is added after the speaker feature layer. The classification dimension is equal to the number of speakers in the training dataset. The speech recognition network does not participate in training and directly uses the pre-trained model of the pre-trained speech recognition network to extract semantic content features. The construction of the neural network is completed.
[0040] 2) Model training
[0041] First, the speech data in the dataset is divided into batches of equal size, each containing K speech data points. For each speech point in a batch, a 3-second speech segment is randomly selected and speech enhancement is performed. Then, its spectrogram is calculated, and Mel-scale Frequency Cepstral Coefficients (MFCC) features are extracted as input to the neural network model. The training dataset is the VoxCeleb2 dataset, which is derived from real-world scenarios in natural environments, with audio taken from English audio on YouTube. The speech contains some realistic noise, with irregular timing, including sudden environmental noise, background voices, laughter, speech aliasing, echoes, indoor noise, and recording equipment noise. The speaking scenarios include celebrity red carpet walks, celebrity speeches, reality TV interviews, and commentary in large stadiums. The MFCC feature has a dimension of 80. Two enhancement methods are used in speech processing—additive noise and room impulse response (RIR) simulation. Additive noise uses speech segments from the MUSAN corpus.
[0042] Then, the MFCC features are input into the Deep Speech2 speech recognition network to directly obtain the feature representation f of the semantic content. sch The dimensions of the speech features are (101, 29).
[0043] Simultaneously, the MFCC features are input into the voiceprint recognition network ECAPA-TDNN to obtain the voiceprint features f. spk The voiceprint feature has 256 dimensions.
[0044] Then, the sum of the cross-entropy losses L between the classification results of the K voiceprint features obtained from a batch of data and the true labels is calculated using the following formula (1). A The classification result is obtained by AAM-softmax calculation, and the true label needs to be processed by one-hot encoding.
[0045]
[0046] In formula (1), cos(θ) j,i)=||W j ||||f spk_i ||cos <W j ,f spk_i >, i.e., f spk_i and W j After L2 regularization, the dot product, where i and j are indices in a batch of data, 0 < i < K, 0 < j < K, f spk_i y represents the voiceprint features of the i-th speech. i Let W represent the label of the i-th speech, and W represent the weight matrix of the classification head. The classification result is represented by label y. i The weight parameter W corresponding to the time. j Represents y i The weight parameters of the other classification heads, s, are fixed scaling factors to prevent the gradient from becoming too small during training, through W j and f spk_i An angular margin m is added between them to maximize intra-class similarity and minimize inter-class similarity, aiming to make the distance between similar speakers greater. Here, s is 30, m is 0.2, and K is 400.
[0047] Meanwhile, the voiceprint features f obtained from the batch speech data are calculated using the mutual information-based deentanglement method according to the following formula (2). spk With semantic features f sch The correlation between them yields the loss L. s To achieve deentanglement between voiceprint features and semantic features;
[0048]
[0049] In formula (2), Given f sch down f spk The true posterior variational approximation can be obtained by the network θ(f) spk ,f sch Parameterization.
[0050] Next, the loss function of the entire training network is obtained through the following formula (3);
[0051] L cls =L A +λL S (3)
[0052] In formula (3), λ is the weight hyperparameter. λ is set to 10.
[0053] Finally, using all the training data, minimize the loss function L. cls The speaker confirmation model was obtained after training was completed.
[0054] 3) Result Testing
[0055] First, the last layer of the classification layer is removed from the trained model to represent the voiceprint feature representation;
[0056] Then, for the two speech samples to be compared, their full-length spectrograms are extracted, and their MFCC features are extracted. The MFCC features are then input into the model to obtain the speaker signature representations f for the two speech samples. spk1 f spk2 The audio samples for comparison are from the test set, specifically the CN-Celeb dataset. CN-Celeb is a Chinese speech dataset designed by Tsinghua University, comprising approximately 130,000 speech samples from 1,000 Chinese celebrities. CN-Celeb covers 11 speech scenarios, including entertainment programs, interviews, singing, drama, movies, video blogs, live broadcasts, speeches, recitations, and advertisements. Compared to VoxCeleb2, which only contains interview-type audio, CN-Celeb offers a richer range of scenarios. The CN-Celeb dataset presents a greater challenge to current speaker recognition methods.
[0057] Finally, the similarity score of the voiceprint features of the two voices to be compared is obtained by using the following formula (4) to determine whether the two voices to be compared belong to the same person.
[0058]
[0059] In formula (4), ||·||2 refers to the second norm.
Claims
1. A text-independent speaker identification method based on identity information and semantic information deentanglement, characterized in that, Includes the following steps: 1) Building a neural network The neural network used is a two-stream network architecture. It simultaneously extracts features representing speaker identity information and semantic content features from a speech segment. A speaker recognition network is used to extract speaker features, and a speech recognition network is used to extract speech content features. The speaker recognition network is used for training, and a classification layer is added after the speaker feature layer. The classification dimension is equal to the number of speakers in the training dataset. The speech recognition network does not participate in training and directly uses the pre-trained model of the pre-trained speech recognition network to extract semantic content features. The neural network is then constructed. 2) Model training First, the speech data in the dataset is divided into batches of equal size, each containing K speech data. For each speech data in a batch, a 3-second speech segment is randomly selected and speech enhancement is performed. Then, its spectrogram is calculated, and the Mel-scale Frequency Cepstral Coefficients (MFCC) feature is extracted as the input of the neural network model. Then, the MFCC features are input into the speech recognition network to directly obtain the feature representation f of the semantic content. sch ; Simultaneously, the MFCC features are input into the voiceprint recognition network to obtain the voiceprint features f. spk ; Then, the sum of the cross-entropy losses L between the classification results of the K voiceprint features obtained from a batch of data and the true labels is calculated using the following formula (1). A The classification result is obtained by AAM-softmax calculation, and the true label needs to be processed by one-hot encoding. In formula (1), cos(θ) j,i )=||W j ||||f spk_i ||cos <W j ,f spk_i >, i.e., f spk_i and W j After L2 regularization, the dot product, where i and j are indices in a batch of data, 0 < i < K, 0 < j < K, f spk_i y represents the voiceprint features of the i-th speech. i Let W represent the label of the i-th speech, and W represent the weight matrix of the classification head. The classification result is represented by label y. i The weight parameter W corresponding to the time. j Represents y i The weight parameters of the other classification heads, s, are fixed scaling factors to prevent the gradient from becoming too small during training, through W j and f spk_i An angular margin m is added between them to maximize intra-class similarity and minimize inter-class similarity, aiming to make the distance between similar speakers greater; Simultaneously, the voiceprint features f obtained by calculating batch speech data using the deentanglement method are used. spk With semantic features f sch The correlation between them yields the loss L. s To achieve deentanglement between voiceprint features and semantic features; The unentanglement method described here uses the mutual information method as shown in formula (2); In formula (2), Given f sch down f spk The true posterior variational approximation can be obtained by the network θ(f) spk ,f sch Parameterization; Next, the loss function of the entire training network is obtained through the following formula (3); THE cls =L A +λL S (3) In formula (3), λ is the weight hyperparameter; Finally, using all the training data, minimize the loss function L. cls The speaker confirmation model is obtained after training is completed; 3) Result Testing First, the last layer of the classification layer is removed from the trained model to represent the voiceprint features; Then, for the two speech samples to be compared, their full-length spectrograms are extracted, and their MFCC features are extracted. The MFCC features are then input into the model to obtain the speaker signature representations f for the two speech samples. spk1 f spk2 ; Finally, the similarity score of the voiceprint features of the two voices to be compared is obtained by using the following formula (4) to determine whether the two voices to be compared belong to the same person. In formula (4), ||·||2 refers to the second norm.
2. The text-independent speaker identification method based on identity information and semantic information deentanglement according to claim 1, characterized in that, The voiceprint recognition network can be any one of the following network models: ECAPA-TDNN, ResNet, Thin-ResNet, or VGG.
3. The text-independent speaker identification method based on identity information and semantic information deentanglement according to claim 1, characterized in that, The speech recognition network can be any one of Deep Speech2, Deep Speech, or Whisper network models.