A speaker verification method based on distance correlation metric learning
Through distance-related measurement learning methods, automatic classification neural networks and deep neural networks are used to build a scoring model, and different measurement functions are given according to the distance embedded by the speaker, which solves the problem of confusing speech in traditional methods, and improves the problem of insufficient speech distinction. The accuracy of the speaker confirmation.
Patent Information
- Application Number
- CN202111223654.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-21
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2041-10-21
AI Technical Summary
When comparing the similarity between two speaker embeddings, the traditional speaker confirmation method treats all embeddings equally, failing to effectively distinguish and confusing speech, resulting in a high misjudgment rate.
Using distance-related metric learning methods, we build a scoring model through automatic classification neural networks and deep neural networks. Different measurement functions are given according to the distance between the speaker's embeddings, and pre-classification and distance calculation are performed to improve confusing voices. ability to distinguish.
It effectively improves the accuracy of the speaker's confirmation, reduces the occurrence of misjudgment of abnormal points and misjudgment of similar points, and enhances the ability to distinguish pronunciations that are prone to confusing.
Smart Images
Figure CN114067803B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical fields of voiceprint recognition, pattern recognition and machine learning, and in particular relates to a speaker confirmation method based on distance-related metric learning. Background Art
[0002] Speaker verification refers to confirming whether two or more voices are spoken by the same person based on the information related to the speaker contained in the voice. With the rapid development of information technology and communication technology, speaker verification technology has received more and more attention and has been widely used in many fields. Such as identity authentication, arresting telephone and telecommunications criminals, confirming identities based on telephone recordings in court, telephone voice tracking, providing anti-theft door opening functions, etc. In the field of Internet applications and communications, speaker recognition technology can be applied to voice dialing, telephone banking, telephone shopping, database access, information services, voice e-mail, security control, computer remote login and other fields.
[0003] Traditional speaker verification generally includes the following steps. First, the speech data is preprocessed: frame division and windowing are performed to extract the features of each frame (commonly used are Mel cepstral features and Fbank features); then, according to the features of each frame, a speech is converted into a speaker embedding (embedding, commonly used are i-vector, d-vector, x-vector) containing the speaker information of the speech. Speaker embedding is actually a multidimensional vector; finally, the difference between the speaker embeddings of the two speech is compared and scored to obtain the similarity, that is, the distance, so as to determine whether the two speech are spoken by the same speaker. Commonly used scoring models include cosine scoring and PLDA (probabilistic linear discriminant analysis) scoring. Cosine scoring first normalizes the length of the speaker embedding, and then calculates the cosine value of the angle between the two speaker embeddings. The larger the cosine value, the smaller the corresponding angle, that is, the smaller the distance between the two speaker embeddings, and the greater the possibility that the two speech belong to the same speaker. PLDA scoring measures the similarity of two speech sounds by calculating the ratio of the probability that the two speaker embeddings are generated by the same speaker to the probability that they are generated by different speakers. The higher the ratio, the higher the similarity, and the greater the possibility that the two speech sounds belong to the same speaker.
[0004] The scoring model used in the traditional speaker verification method treats all speaker embeddings equally when comparing the similarity of two speaker embeddings, without focusing on the distinction of easily confused speech. In fact, when the distance between speaker embeddings is relatively small, heterogeneous points are more likely to be misjudged (the distance between heterogeneous points is required to be large); and when the distance between speaker embeddings is relatively large, homogeneous points are more likely to be misjudged (the distance between homogeneous points is required to be small). The present invention proposes a speaker verification method based on distance-related metric learning, and different metric functions are given when scoring according to the different distances between speaker embeddings. When the distance between speaker embeddings is relatively small, the metric function focuses on expanding the distance between heterogeneous points; when the distance between speaker embeddings is relatively large, the metric function focuses on reducing the distance between homogeneous points. Since the present invention needs to give different metric functions according to the different distances between speaker embeddings, it is necessary to pre-classify speaker embedding pairs (two speaker embeddings form a speaker embedding pair) according to the distance, and each class corresponds to a metric function. In order to avoid the interference caused by artificial pre-classification, the present invention adopts an end-to-end training method to allow the model to actively learn the pre-classification of speaker embedding pairs. The present invention focuses on distinguishing easily confused speech and can improve the accuracy of speaker confirmation. Summary of the invention
[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a speaker verification method based on distance-related metric learning. The present invention emphasizes the distinction of easily confused speech and can effectively improve the accuracy of speaker verification.
[0006] The present invention proposes a speaker verification method based on distance-related metric learning, which is divided into two stages: model training and speaker verification, and includes the following steps:
[0007] 1) Model training stage: The model training stage is divided into two stages: preprocessing the training speech and extracting the speaker embedding of each speech and training the scoring model based on the distance-related metric learning method. Specifically, it includes the following steps:
[0008] 1-1) Preprocess the training speech and extract the speaker embedding of each speech;
[0009] Preprocess the training speech, including framing, extracting the features corresponding to each frame of speech (which can be one of the Mel cepstral features or Fbank features), and then extract the speaker embedding corresponding to the speech (which can be one of the i-vector, d-vector or x-vector) based on the features corresponding to all frames of a speech, recorded as y i ,y i is a multidimensional vector.
[0010] 1-2) Training the scoring model based on the distance-related metric learning method; including two stages: scoring model construction and scoring model training, specifically including the following steps:
[0011] 1-2-1) Scoring model construction. Since the present invention needs to give different metric functions according to the different distances between speaker embeddings, it is necessary to build an automatic classification neural network g to pre-classify speaker embedding pairs (two speaker embeddings form a speaker embedding pair), and then build a metric function for each class. Finally, according to the category to which the speaker embedding pair belongs and the metric function corresponding to the category, the distance of the speaker embedding pair is obtained. Scoring model construction includes the following steps:
[0012] A) Build an automatic classification neural network g, based on the difference y between the two speakers’ embeddings i -y j , embed the speaker into (y i ,y j ) is divided into K categories. The input of the automatic classification neural network g is the difference y between the two speaker embeddings i -y j , the activation function of the output layer is the softmax function, and the activation functions of other layers are all ReLU functions. The output is the speaker embedding pair (y i ,y j ) belongs to these K categories, and the output nodes are K in total, each node represents a category. i ,y j ) belongs to the kth category with a probability of
[0013] p(k th |y i ,y j ) = g(θ, y i -y j ) (k)
[0014] Where θ represents the parameters of the automatic classification neural network g. g(θ,y i -y j ) is the output of the neural network g, and k represents the value corresponding to the kth node.
[0015] B) Build a corresponding metric function for each category output by the automatic classification neural network g. Each metric function is represented by a deep neural network. K categories require a total of K deep neural networks. The input of all deep neural networks is the speaker embedding y i , the activation function of all layers is ReLU function, the k-th deep neural network f k The output is the vector z i =f k (θ k ,yi ), where θ k Represents the parameters of the k-th deep neural network. y is obtained by measuring the k-th neural network i and j The distance is
[0016]
[0017] C) The final scoring model obtains y i and j The distance d(y i ,y j ) is the speaker embedding pair (y i ,y j ) is the probability of belonging to the kth category multiplied by the distance d obtained according to the kth metric function k (y i ,y j ), and then accumulate, that is:
[0018]
[0019] 1-2-2) Scoring model training
[0020] A) Randomly select N pairs of speaker embeddings of the same speaker from the training speech Speaker embeddings for N pairs of different speakers Enter the scoring model built in step 1-2-1) to get the distance and In order to make the speaker embedding distance of the same speaker small and the speaker embedding distance of different speakers large, the loss function optimized by the model is:
[0021]
[0022] Using the gradient descent method, update the parameters θ of the automatic classification neural network and the parameters θ of the K deep neural networks k (k=1,…,K).
[0023] B) Repeat steps 1-2-2)A) and update the scoring model parameters until the model converges and the training is completed.
[0024] C) Randomly select M pairs of speaker embeddings of the same speaker from the training speech M speaker embeddings for different speakers Input into the scoring model trained in step 1-2-2) B) to get the distance between M embeddings of the same speaker The distance between the embeddings of M different speakers Based on these 2M distances, a DET curve is drawn, in which the distance corresponding to the equal error rate point is used as the threshold for judging whether it is the same speaker or different speakers.
[0025] 2) Speaker confirmation stage: Given two speech sounds to be tested, a scoring model trained by a distance-related metric learning method is used to determine whether they are spoken by the same person. Specifically, the following steps are included:
[0026] 2-1) Preprocess the two test voices and extract the speaker embeddings of the two voices;
[0027] The two test voices are preprocessed, including framing, extracting the features of each frame of voice corresponding to the same type as step 1-1), and then extracting the speaker embedding of each voice corresponding to the same type as step 1-1) according to the features corresponding to all frames of each voice, which are recorded as and
[0028] 2-2) Obtain the distance between the two voices according to the scoring model;
[0029] According to the scoring model obtained in step 1-2), the distance between the two voices is calculated as:
[0030]
[0031] 2-3) Determine whether the distance between the two voices is greater than the threshold value obtained in step 1-2-2) C). If it is greater than the threshold value, they are spoken by different speakers. If it is less than or equal to the threshold value, they are spoken by the same speaker. At this point, speaker confirmation is completed.
[0032] The characteristics and beneficial effects of the present invention are:
[0033] In the existing speaker verification technology, when comparing the similarity of two speaker embeddings, all speaker embeddings are treated equally, without focusing on the distinction of easily confused speech. Compared with the prior art, the present invention proposes a speaker verification method based on distance-related metric learning. When scoring speaker embeddings, different metric functions are given according to the different distances between speaker embeddings. If the distance between speaker embeddings is relatively small, heterogeneous points are more likely to be misjudged, and the corresponding metric function focuses on expanding the distance between heterogeneous points. If the distance between speaker embeddings is relatively large, similar points are more likely to be misjudged, and the corresponding metric function focuses on reducing the distance between similar points. In order to pre-classify speaker embedding pairs and avoid interference caused by manual pre-classification, the present invention adopts an end-to-end training method to allow the model to actively learn the automatic classification neural network of speaker embedding pairs. The present invention emphasizes the distinction of easily confused speech, which can improve the accuracy of speaker verification. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 is a flow chart of the method of the present invention.
[0035] Figure 2 It is a schematic diagram of the scoring model of the method of the present invention.
[0036] Figure 3 It is a schematic diagram of the automatic classification neural network structure of the scoring model of an embodiment of the present invention.
[0037] Figure 4 It is a schematic diagram of the deep neural network structure of the scoring model of an embodiment of the present invention. DETAILED DESCRIPTION
[0038] The present invention proposes a speaker verification method based on distance-related metric learning, which is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.
[0039] The present invention proposes a speaker verification method based on distance-related metric learning, such as Figure 1 As shown, it is divided into two stages: model training and speaker confirmation, including the following steps:
[0040] 1) Model training stage: The model training stage is divided into two stages: preprocessing the training speech and extracting the speaker embedding of each speech and training the scoring model based on the distance-related metric learning method. Specifically, it includes the following steps:
[0041] 1-1) Preprocess the training speech and extract the speaker embedding of each speech;
[0042] The training speech is preprocessed, including framing and extracting the features corresponding to each frame of speech (which can be one of the Mel cepstral features or Fbank features). In this embodiment, we extract 23-dimensional Mel cepstral features for each frame of speech. Then, based on the features corresponding to all frames of a speech, the speaker embedding corresponding to the speech is extracted (which can be one of the i-vector, d-vector or x-vector). In this embodiment, the speaker embedding corresponding to each speech extracted is x-vector, denoted by y i ,y i is a 512-dimensional vector.
[0043] 1-2) Training the scoring model based on the distance-related metric learning method; including two stages: scoring model construction and scoring model training, specifically including the following steps:
[0044] 1-2-1) Scoring model construction. Since the present invention needs to give different metric functions according to the different distances between speaker embeddings, Figure 2As shown, it is necessary to build an automatic classification neural network g to pre-classify speaker embedding pairs (two speaker embeddings form a speaker embedding pair), then build a metric function for each class, and finally get the distance of the speaker embedding pair based on the category to which the speaker embedding pair belongs and the metric function corresponding to the category. The scoring model building includes the following steps:
[0045] A) Build an automatic classification neural network g, based on the difference y between the two speakers’ embeddings i -y j , embed the speaker into (y i ,y j ) is divided into K categories. The input of the automatic classification neural network g is the difference y between the two speaker embeddings i -y j , the activation function of the output layer is the softmax function, and the activation functions of other layers are all ReLU functions. The output is the speaker embedding pair (y i ,y j ) belongs to these K categories, and there are K output nodes in total, each node represents a category. In this embodiment, we divide the speaker embedding pairs into K = 3 categories. Figure 3 As shown, the input of the automatic classification neural network g in this embodiment is y i -y j , because y i is a 512-dimensional vector, so the input layer has 512 nodes. The hidden layer of the automatic classification neural network g has 2 layers, the first hidden layer has 384 nodes, the second hidden layer has 256 nodes, and the output layer has 3 nodes. i ,y j ) belongs to the kth category with a probability of
[0046] p(k th |y i ,y j ) = g(θ, y i -y j ) (k)
[0047] Where θ represents the parameters of the automatic classification neural network g. g(θ,y i -y j ) is the output of the neural network g, and k represents the value corresponding to the kth node.
[0048] B) Build a corresponding metric function for each category output by the automatic classification neural network g. Each metric function is represented by a deep neural network. In this embodiment, three categories require three deep neural networks. Figure 4 As shown, the input of all deep neural networks is the speaker embedding y i, the input layer has 512 nodes, the hidden layer has 3 layers, the number of nodes are 1024, 2048, 700 respectively, and the number of nodes in the output layer is 256. The activation function of all layers is the ReLU function. The k-th deep neural network f k The output is the vector z i =f k (θ k ,y i ), where θ k Represents the parameters of the k-th deep neural network. y is obtained by measuring the k-th neural network i and j The distance is
[0049]
[0050] C) The final scoring model obtains y i and j The distance d(y i ,y j ) is the speaker embedding pair (y i ,y j ) is the probability of belonging to the kth category multiplied by the distance d obtained according to the kth metric function k (y i ,y j ), and then accumulate, that is:
[0051] This embodiment corresponds to
[0052] 1-2-2) Scoring model training
[0053] A) Randomly select N pairs of speaker embeddings of the same speaker from the training speech Speaker embeddings for N pairs of different speakers In this embodiment, N=128. Input the scoring model built in step 1-2-1) to obtain the distance and In order to make the speaker embedding distance of the same speaker small and the speaker embedding distance of different speakers large, the loss function optimized by the model is:
[0054] This embodiment corresponds to
[0055] Using the gradient descent method, update the parameters θ of the automatic classification neural network and the parameters θ of the K=3 deep neural networks k (k=1, 2, 3).
[0056] B) Repeat steps 1-2-2)A) and update the scoring model parameters until the model converges and the training is completed.
[0057] C) Randomly select M pairs of speaker embeddings of the same speaker from the training speech M speaker embeddings for different speakers Input into the scoring model trained in step 1-2-2) B) to get the distance between M embeddings of the same speaker The distance between the embeddings of M different speakers Based on the 2M distances, a DET curve is drawn, wherein the distance corresponding to the equal error rate point is used as the threshold for judging whether the speaker is the same speaker or different speakers. In this embodiment, M=1000, and the threshold obtained based on the 2M distances is 5.6.
[0058] 2) Speaker confirmation stage: Given two speech sounds to be tested, a scoring model trained by a distance-related metric learning method is used to determine whether they are spoken by the same person. Specifically, the following steps are included:
[0059] 2-1) Preprocess the two test voices and extract the speaker embeddings of the two voices;
[0060] The two test voices are preprocessed, including framing and extracting the features of the same type as step 1-1) corresponding to each frame of the voice. In this embodiment, the features correspond to the Mel cepstrum features. Then, based on the features corresponding to all frames of each voice, the speaker embedding of the same type as step 1-1) corresponding to each voice is extracted. In this embodiment, the speaker embedding corresponds to x-vector, which is recorded as and
[0061] 2-2) Obtain the distance between the two voices according to the scoring model;
[0062] According to the scoring model obtained in step 1-2), the distance between the two voices is calculated as:
[0063]
[0064] In this embodiment, the calculated distance between the two speech sounds is 10.2.
[0065] 2-3) Determine whether the distance between the two voices is greater than the threshold value obtained in step 1-2-2) C). If it is greater than the threshold value, they are spoken by different speakers. If it is less than or equal to the threshold value, they are spoken by the same speaker. In this embodiment, the threshold value is 5.6, and the distance between the two voices is 10.2. 10.2>5.6, so it is determined that the two voices are spoken by different speakers. At this point, speaker confirmation is completed.
[0066] The method of the present invention can be understood by those skilled in the art as follows: the method of speaker confirmation can be accomplished by a program, and the program can be stored in a computer-readable storage medium.
[0067] The above is only a specific embodiment of the present invention, which obviously cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.
Claims
1. A speaker verification method based on distance-related metric learning, characterized in that: It is divided into two stages: model training and speaker verification. In the model training stage, the training speech is preprocessed and the speaker embedding of each speech is extracted, and the scoring model is trained based on the distance-related metric learning method. In the speaker verification stage, given two test speech, the scoring model trained by the distance-related metric learning method is used to determine whether the two speech are spoken by the same person. The model training phase includes the following steps: 1) Model training phase; specifically includes the following steps: 1-1) Preprocess the training speech and extract the speaker embedding of each speech; Preprocess the training speech, including framing and extracting the features corresponding to each frame of speech. Then, based on the features corresponding to all frames of a speech, extract the speaker embedding corresponding to the speech, denoted as y i ,y i is a multidimensional vector; 1-2) Training the scoring model based on the distance-related metric learning method; including two stages: scoring model construction and scoring model training, specifically including the following steps: 1-2-1) Scoring model construction: Build an automatic classification neural network g to pre-classify speaker embedding pairs. Two speaker embeddings form a speaker embedding pair. Then build a metric function for each class. Finally, according to the category to which the speaker embedding pair belongs and the metric function corresponding to the category, the distance of the speaker embedding pair is obtained. Model construction includes the following steps: A) Build an automatic classification neural network g, based on the difference y between the two speakers’ embeddings i -y j , embed the speaker into (y i ,y j ) is divided into K categories; the input of the automatic classification neural network g is the difference y between the two speaker embeddings i -y j , the activation function of the output layer is the softmax function, and the activation functions of other layers are all ReLU functions. The output is the speaker embedding pair (y i ,y j ) belongs to these K categories, and there are K output nodes in total, each node represents a category; the speaker embedding pair (y i ,y j ) belongs to the kth category with a probability of p(k th |and i ,and j )=g(θ,y i -and j )(k) Where θ represents the parameters of the automatic classification neural network g; g(θ, y i -y j ) is the output of the neural network g, and k represents the value corresponding to the kth node; B) Build a corresponding metric function for each category output by the automatic classification neural network g. Each metric function is represented by a deep neural network. K categories require a total of K deep neural networks. The input of all deep neural networks is the speaker embedding y i , the activation function of all layers is ReLU function, the k-th deep neural network f k The output is the vector z i =f k (θ k ,y i ), where θ k Represents the parameters of the k-th deep neural network; y is obtained by measuring the k-th neural network i and j The distance is C) The final scoring model obtains y i and j The distance d(y i ,y j ) is the speaker embedding pair (y i ,y j ) is the probability of belonging to the kth category multiplied by the distance d obtained according to the kth metric function k (y i ,y j ), and then accumulate, that is: 1-2-2) Scoring model training; A) Randomly select N pairs of speaker embeddings of the same speaker from the training speech Speaker embeddings for N pairs of different speakers Enter the scoring model built in step 1-2-1) to get the distance and In order to make the speaker embedding distance of the same speaker small and the speaker embedding distance of different speakers large, the loss function optimized by the model is: Using the gradient descent method, update the parameters θ of the automatic classification neural network and the parameters θk (k = 1, ..., K) of the K deep neural networks; B) Repeat step A) to update the scoring model parameters until the model converges and the training is completed; C) Randomly select M pairs of speaker embeddings of the same speaker from the training speech M speaker embeddings for different speakers Input into the scoring model trained in step 1-2-2) B) to get the distance between M embeddings of the same speaker The distance between the embeddings of M different speakers Based on these 2M distances, a DET curve is drawn, where the distance corresponding to the equal error rate point is used as the threshold for judging whether it is the same speaker or different speakers; The speaker confirmation stage includes the following steps: 2) Speaker confirmation stage; specifically includes the following steps: 2-1) Preprocess the two test voices and extract the speaker embeddings of the two voices; The two test voices are preprocessed, including framing, extracting the features of each frame of voice corresponding to the same type as step 1-1), and then extracting the speaker embedding of each voice corresponding to the same type as step 1-1) according to the features corresponding to all frames of each voice, which are recorded as and 2-2) Obtain the distance between the two voices according to the scoring model; According to the scoring model obtained in step 1-2), the distance between the two voices is calculated as: 2-3) Determine whether the distance between the two voices is greater than the threshold value obtained in step C) of step 1-2-2). If it is greater than the threshold value, they are spoken by different speakers. If it is less than or equal to the threshold value, they are spoken by the same speaker. At this point, speaker confirmation is completed.