A deep neural network-based voiceprint recognition method

Abstract

The application discloses a kind of based on deep neural network's voiceprint recognition method, belong to speech processing technical field.The application includes: training voiceprint feature extraction network and voiceprint recognition scoring network, based on the voiceprint feature extraction network of trained prediction registration voice voiceprint feature vector, based on registration voiceprint feature vector constructs voiceprint feature database;Acoustic feature to be identified is obtained, based on the scoring result of voiceprint recognition scoring network with voiceprint feature database is obtained to determine recognition result.The voiceprint feature vector extracted by the application is more delicate, so that the speech feature is better preserved.The Fbank feature extracted reduces the amount of calculation in the speech preprocessing process, speeds up the feature construction speed.By setting a larger number of channels to the network, the fitting ability of the neural network model is enhanced.The scoring network uses the parameters obtained by training the PLDA algorithm to initialize the network parameters, which speeds up the network convergence speed and achieves better results.

Description

Technical Field

[0001] This invention belongs to the field of speech processing technology, specifically relating to a voiceprint recognition method based on deep neural networks. Background Technology

[0002] In the early stages of voiceprint recognition technology research, voiceprint identification was mainly achieved through manual comparison of time-frequency spectra. Subsequently, several traditional methods were proposed and achieved good results, including template matching, Gaussian mixture background model (GMM-UBM), support vector machine model (GMM-SVM), joint factor analysis (JFA), and so on. However, traditional methods suffer from problems such as being greatly affected by noise and having weak cross-channel resolution capabilities.

[0003] After 2012, with the widespread application of deep learning technology, deep learning-based voiceprint recognition emerged and became the optimal method, replacing traditional approaches. The earliest method was DNN / i-vetor, which aimed to obtain an i-vector instead of traditional methods. However, due to its requirement for large amounts of supervised data for training and its difficulty in handling cross-lingual issues, research on it gradually decreased. Subsequently, embedding methods appeared, the basic idea of ​​which is to extract weights from the neural network as speaker feature vectors, including j-vector, d-vector, and x-vector. j-vector is used to handle text-related problems. x-vector, utilizing TDNN (Time Delay Neural Network), can extract more information than d-vector and has become the mainstream method for text-independent voiceprint recognition.

[0004] Currently, voiceprint recognition technology is in the x-vector derivative stage, mainly improving x-vectors in terms of network input, network structure, pooling methods, and loss functions. EcapaTDNN, proposed in recent years, is an improvement on the x-vector structure and has achieved significant performance gains. Voiceprint recognition systems require a scoring mechanism, and common scoring algorithms include cosine and PLDA (Probabilistic Linear Discriminant Analysis). In recent years, the NerualPLDA algorithm has emerged, integrating PLDA into neural networks, facilitating the learning of voiceprint recognition systems and the construction of end-to-end systems. Therefore, it is necessary to propose a voiceprint recognition method based on the NerualPLDA algorithm to improve voiceprint recognition performance. Summary of the Invention

[0005] This invention provides a voiceprint recognition method based on deep neural networks, which can be used to improve voiceprint recognition performance.

[0006] The technical solution adopted in this invention is as follows:

[0007] A voiceprint recognition method based on a deep neural network, comprising the following steps:

[0008] Step S1: Train the voiceprint feature extraction network and the voiceprint recognition scoring network;

[0009] The input to the voiceprint feature extraction network is the acoustic features of the speech data, and the output is a voiceprint feature vector.

[0010] The voiceprint recognition scoring network takes a pair of voiceprint feature vectors as input and outputs the similarity between the voiceprint feature vector pairs. The voiceprint recognition scoring network includes two feature mapping branches with the same network structure and a regularization layer. The feature mapping branches sequentially include a first fully connected layer, a batch normalization layer, and a second fully connected network. The outputs of both feature mapping branches are input to the regularization layer, and the final scoring result is obtained based on the output of the regularization layer. The training parameters of the regularization layer include a positive definite symmetric matrix Q representing intra-class differences and a semi-positive definite symmetric matrix P representing inter-class differences.

[0011] Collect speech data from training samples, perform speech data preprocessing on the speech data of the training samples, and then extract acoustic features.

[0012] The acoustic features of the training samples are input into the voiceprint feature extraction network, and the voiceprint feature vector of each training sample is obtained based on its output. The voiceprint feature vector sample set is obtained based on the voiceprint feature vector of all training samples.

[0013] A pair of voiceprint feature vectors are randomly selected from the voiceprint feature vector sample set to form the input data of the voiceprint recognition scoring network. The scoring result of the current recognition object is obtained based on the output of the voiceprint recognition scoring network. The network parameters of the voiceprint feature extraction network and the voiceprint recognition scoring network are trained by deep learning based on the preset loss function. The training stops when the preset training termination condition is met, and the trained voiceprint feature extraction network and voiceprint recognition scoring network are obtained.

[0014] Step S2: When the registration voice sent by the registered user is received, the voiceprint feature vector of the registered voice is predicted based on the trained voiceprint feature extraction network to obtain the registered voiceprint feature vector of the registrant, and a voiceprint feature database is constructed based on the registered voiceprint feature vector.

[0015] Step S3: Perform speech data preprocessing on the speech data to be recognized, and then extract acoustic features to obtain the acoustic features to be recognized.

[0016] The acoustic features to be identified are input into the trained voiceprint feature extraction network to obtain the voiceprint feature vector to be identified.

[0017] Iterate through each registered voiceprint feature vector in the voiceprint feature database, take the current registered voiceprint feature vector and the voiceprint feature vector to be identified as the current scoring object pair, and input them into the trained voiceprint recognition scoring network to obtain the scoring result of the current scoring object pair; and take the speaker of the registered voiceprint feature vector corresponding to the highest score as the recognition result of the speech data to be identified, and check whether the scoring result corresponding to the recognition result is greater than or equal to the preset global threshold. If so, output the recognition result; otherwise, the recognition result is invalid.

[0018] Furthermore, the speech data preprocessing includes: speech format adjustment, speech activity detection, and speech fragmentation processing.

[0019] Furthermore, in step S1, the speech data preprocessing of the speech data of the training samples also includes adding noise and reverberation.

[0020] Furthermore, the acoustic features used are 80-dimensional F-bank features.

[0021] Furthermore, the network structure of the voiceprint feature extraction network is as follows: a first convolutional block, first, second and third backbone network blocks, a second convolutional block, an attention pooling network block, a fully connected layer with batch normalization operation and a Softmax layer;

[0022] The input to the second convolutional block also includes the output feature maps of the first backbone network block and the second backbone network block;

[0023] The network structure of each convolutional block includes, in sequence, a one-dimensional convolutional layer, an activation function layer, and a batch normalization layer;

[0024] Each backbone network block is a multi-scale backbone network with an SE (Squeeze-and-Excitation) network;

[0025] The attention pooling network block includes: pooling operations that introduce an attention mechanism, and batch normalization operations.

[0026] Furthermore, the scoring results of the voiceprint recognition scoring network are set as follows:

[0027] The two speakers in the input voiceprint feature vector pair of the voiceprint recognition scoring network are defined as two roles: template speaker and speaker to be identified; and the voiceprint feature vector of the template speaker is defined as x1, and the voiceprint feature vector of the speaker to be identified is defined as x2.

[0028] The scoring results of the voiceprint recognition scoring network are as follows:

[0029] Where n1 and n2 represent the feature vectors obtained by mapping x1 and x2 through two fully connected layers, respectively. Q and P are network parameters.

[0030] The technical solution provided by this invention brings at least the following beneficial effects:

[0031] The voiceprint recognition method of this invention extracts voiceprint feature vectors with more refined voiceprint information, resulting in better preservation of speech features. The Fbank feature extraction network extracted in this invention reduces the computational load in the speech preprocessing process and accelerates feature construction. This method enhances the fitting ability of the neural network model by setting a larger number of channels C. The front-end network training uses Chinese speech datasets extracted from real-world scenarios, making the method more suitable for Chinese speech recognition. The back-end scoring network uses parameters obtained from training the PLDA algorithm to initialize network parameters, accelerating network convergence and achieving better results. In this method, the scoring results are regularized using AS-norm, more accurately representing the similarity and differences between voiceprint feature vectors. Through these methods, the voiceprint recognition method implemented in this invention was tested on the CnCeleb dataset, and the obtained EER, minDCF, and other metrics are superior to existing methods. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a schematic diagram of the process for extracting Fbank acoustic features in an embodiment of the present invention;

[0034] Figure 2 This is a network structure diagram of the voiceprint feature extraction network used in this embodiment of the invention;

[0035] Figure 3 This is a schematic diagram of the processing procedure of the voiceprint recognition backend network used in an embodiment of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0037] The implementation process of the deep neural network-based voiceprint recognition method provided in this embodiment of the invention includes:

[0038] Step 1: Extract the Fbank acoustic features from the audio data. Preferably, the Fbank dimension is set to 80 for each speech segment.

[0039] Step 2: The EcapaTDNN network is used as the front-end network for voiceprint recognition (voiceprint feature extraction network) to extract voiceprint feature vectors. Preferably, the number of channels in the EcapaTDNN network is set to 1024. For each speech segment, the extracted 80-dimensional Fbank features are input into the network to obtain a 192-dimensional voiceprint vector output.

[0040] Step 3: NeuralPLDA is used as the back-end scoring network for voiceprint recognition. The voiceprint feature vectors of the two speech segments (the voiceprint feature vector to be recognized and the registered voiceprint feature vector in the voiceprint feature database) are input into the back-end scoring network for voiceprint recognition in pairs to obtain the maximum likelihood score.

[0041] Step 4: Set a global threshold and use the AS-norm method to regularize the scores.

[0042] In this invention, the first step is to extract audio features as input to the front-end network. Commonly used audio features include MFCC (Mel-frequency cepstral coefficients) and F-bank. MFCC, based on F-bank, undergoes Discrete Cosine Transform (DCT) to weaken the correlation between dimensions, and first-order and second-order differences are calculated accordingly. Therefore, it is often used in models sensitive to highly correlated signals. Neural networks are not sensitive to highly correlated signals, and extracting F-bank requires less computation. Therefore, this invention uses F-bank as the input feature of the front-end network.

[0043] like Figure 1 As shown in the embodiment of the present invention, the specific process of extracting the Fbank acoustic features of audio data is as follows:

[0044] (1) Pre-intensification.

[0045] Audio signals exhibit spectral tilt, meaning that the high-frequency components have lower energy and the low-frequency components have higher energy. To increase the energy of the high-frequency components, the audio signal needs to be pre-emphasized. The calculation method is shown in equation (1):

[0046] x′[t]=x[t]-αx[t-1] (1)

[0047] Where x[t] represents the t-th sampling point of the audio signal, and α is a constant, typically ranging from (0.95 to 0.99).

[0048] (2) Frame splitting and windowing.

[0049] To facilitate signal processing and analysis, audio signals need to be segmented into frames, which involves sliding the audio sequence along a certain length and step size to generate audio frames. For example, a frame length of 25ms and a step size of 10ms are commonly used in voiceprint recognition.

[0050] After framing, to resolve the high discontinuity between frames caused by subsequent Fourier transforms, windowing is required. The specific processing is shown in equation (2):

[0051] x′[n]=w[n]x[n] (2)

[0052] Where x[n] represents the nth sampling point, and w[nI] represents the corresponding weight. In the simplest case, w is taken as an all-one vector to generate a rectangular window. However, such a sharply angled window can lead to spectral leakage, so other windows, such as Hamming windows and Heining windows, need to be used instead.

[0053] (3) Discrete Fourier Transform.

[0054] For each frame of audio, the frequency domain information is extracted using Discrete Fourier Transform (DCT). The calculation method is shown in equation (3):

[0055]

[0056] Where X[k] is a complex number containing the amplitude and phase information of frequency band k, and N is the number of frequency bands. The graph obtained by plotting frequency on the x-axis and amplitude on the y-axis is called a spectrum graph. In practical calculations, the energy value |x[k]| of each frequency band is often taken. 2 Using the vertical axis as the ordinate, we obtain the energy spectrum.

[0057] (4) Mel filtering and logarithmic calculation.

[0058] The human ear's perception of audio frequencies is non-linear, and the Mel scale, as a mathematical transformation, describes this non-linearity. Its transformation formula is shown in equation (4):

[0059]

[0060] Mel filtering can be used to transform ordinary frequencies into Mel frequencies. Specifically, the energy spectrum of the audio frame obtained above is passed through a set of triangular filters to obtain the spectrum under the Mel scale. The number of filters is equal to the number of Mel scale values ​​obtained, which is the dimension of the Fbank feature; in this embodiment, 80 is used.

[0061] To better represent human perception of energy and enhance the robustness of the features, the logarithm of the energy values ​​obtained above is often taken. This yields the F-bank features of the final audio frame. Concatenating the features from each frame gives the F-bank features of the audio.

[0062] In extracting voiceprint feature vectors, this invention uses the EcapaTDNN network as the voiceprint recognition front-end network (i.e., the voiceprint feature extraction network), such as... Figure 2 As shown, the EcapaTDNN network consists of the following components in sequence: a convolutional block (one-dimensional convolution, activation function, and batch normalization (BN)), three SE-Res2Block layers, a convolutional block, an attention statistics module (attention statistics pool + BN), a fully connected layer (FC) with BN, and a softmax layer (using an improved AAM-Softmax). The fully connected layer with BN and the softmax layer constitute the output layer of the EcapaTDNN network. Figure 2 In this model, T represents the number of audio frames, C represents the number of channels, and S represents the number of speakers. The EcapaTDNN network introduces Res2Net (a multi-scale backbone network) to reduce the number of parameters in its network structure; adds an SE (Squeeze-and-Excitation) network; and introduces feature aggregation and accumulation mechanisms. The pooling function introduces a channel attention mechanism; the loss function uses an improved version of Softmax, AAM-Softmax. A larger value for the number of channels C between network layers results in better model performance. In this embodiment, a value of 1024 is used for C. Figure 2 In this process, the 192-dimensional vector output by the FC+BN layer is the extracted voiceprint feature vector.

[0063] In this embodiment of the invention, the backend scoring and regularization processing uses a voiceprint recognition backend network as follows: Figure 3 As shown, its input is a voiceprint feature vector, which passes through two fully connected layers (Affine, the trained network parameters include weights W and bias b) and a regularization layer (regularization requires trained network parameters including in-class parameters), and finally scores to obtain a verification score. The first fully connected layer is a vector dimensionality reduction process; the second fully connected layer is the centering and diagonalization process in PLDA. Figure 3 In the diagram, a normalization operation is also included in the two fully connected layers, as shown by x. e x represents the registered voiceprint feature vector. t n represents the voiceprint feature vector to be identified. e n t They represent x respectively e x t The feature vector after two fully connected layers of mapping is then used to obtain n. e n tThe input is fed into a regularization layer for regularization processing, and then scored based on the regularization result. Here, Q is a positive definite symmetric matrix representing intra-class differences, and P is a semi-positive definite symmetric matrix representing inter-class differences. In this embodiment, the voiceprint recognition backend network includes two branches with identical network structures: a first fully connected layer (weights and biases denoted as W1 and b1, respectively), a batch normalization layer, and a second fully connected network (weights and biases denoted as W2 and b2, respectively). The inputs of these two branches are then fed into the regularization layer to obtain... and This leads to the scoring results:

[0064] In this embodiment of the invention, the voiceprint recognition backend network uses an approximate SoftDCF of the cost detection function (DCF) as the loss function, which is defined as shown in equation (5):

[0065]

[0066] in,

[0067]

[0068]

[0069]

[0070] C FR and C FA P represents the cost corresponding to FR (False Rejection Rate) errors and FA (Acceptance Rate) errors, respectively. T t represents the prior probability of the actual speaker appearing. i Indicates whether sample i is a positive sample (i.e., a speech pair belonging to the same speaker), s i θ represents whether the model predicts a score, θ represents the model's global threshold, α is a constant, σ represents the sigmoid function, and N represents the number of samples.

[0071] In voiceprint verification tasks, discrimination is performed based on a global threshold (minDCF threshold). However, the distribution of verification scores differs for different speakers. To mitigate this impact, verification scores can be regularized. A commonly used regularization method is S-norm; in this embodiment of the invention, its adaptive version, AS-norm, is used.

[0072] The processing procedure of a voiceprint recognition method based on a deep neural network provided in this invention can be summarized as follows:

[0073] (1) Preprocess the input speech data. Common preprocessing operations include: adjusting the speech format, such as downsampling the speech, changing the encoding format, etc.; performing speech activity detection to remove silent and noise parts in the speech and improve the speech quality; fragmenting the speech and collecting speech segments of equal length for training; adding noise and reverberation to the original speech to improve the model training effect.

[0074] (2) Perform acoustic feature extraction on the input speech data, i.e. extract the Fbank acoustic features of the audio data.

[0075] (3) In the training phase of voiceprint recognition, the network parameters of the voiceprint feature extraction network and the voiceprint recognition scoring network are trained. First, the Fbank acoustic features of the training samples are input into the voiceprint feature extraction network to obtain the voiceprint feature extraction network vector of each training sample. Then, the voiceprint feature extraction network vector is input into the two branches of the voiceprint recognition scoring network to obtain the corresponding scoring results. Based on the preset loss function, the network parameters of the voiceprint feature extraction network and the voiceprint recognition scoring network are trained by deep learning. When the preset training termination condition is met (the maximum number of training times is reached, or the network loss meets the specified condition), the training is stopped, and the trained voiceprint feature extraction network and voiceprint recognition scoring network are obtained.

[0076] In the registration stage of voiceprint recognition, the trained voiceprint feature extraction network is used to predict the voiceprint feature vector of the registered speech, so as to obtain the registered voiceprint feature vector of the current registrant. Based on the registered voiceprint feature vector of each registrant, a voiceprint feature database, i.e., the speaker model library, is constructed.

[0077] In the voiceprint recognition stage, after preprocessing the speech data to be recognized, Fbank acoustic features are extracted and then input into a trained voiceprint feature extraction network to obtain the voiceprint feature vector to be recognized. Each registered voiceprint feature vector in the voiceprint feature database is traversed, and it is compared with the voiceprint feature vector to be recognized as the current scoring pair. This pair is then input into a trained voiceprint recognition scoring network to compare and score the speech with the speaker model database. The speaker with the highest score is taken as the recognition result. The confirmation decision in the process of obtaining the recognition result is as follows: a global threshold is set; scores above the threshold are accepted, otherwise rejected.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0079] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A voiceprint recognition method based on deep neural networks, characterized in that, Includes the following steps: Step S1: Train the voiceprint feature extraction network and the voiceprint recognition scoring network; The input to the voiceprint feature extraction network is the acoustic features of the speech data, and the output is a voiceprint feature vector. The voiceprint recognition scoring network takes a pair of voiceprint feature vectors as input and outputs the similarity between the voiceprint feature vector pairs. The voiceprint recognition scoring network includes two feature mapping branches with the same network structure and a regularization layer. The feature mapping branches sequentially include a first fully connected layer, a batch normalization layer, and a second fully connected network. The outputs of both feature mapping branches are input to the regularization layer, and the final scoring result is obtained based on the output of the regularization layer. The training parameters of the regularization layer include a positive definite symmetric matrix Q representing intra-class differences and a semi-positive definite symmetric matrix P representing inter-class differences. Collect speech data from training samples, perform speech data preprocessing on the speech data of the training samples, and then extract acoustic features. The acoustic features of the training samples are input into the voiceprint feature extraction network, and the voiceprint feature vector of each training sample is obtained based on its output. The voiceprint feature vector sample set is obtained based on the voiceprint feature vector of all training samples. A pair of voiceprint feature vectors are randomly selected from the voiceprint feature vector sample set to form the input data of the voiceprint recognition scoring network. The scoring result of the current recognition object is obtained based on the output of the voiceprint recognition scoring network. The network parameters of the voiceprint feature extraction network and the voiceprint recognition scoring network are trained by deep learning based on the preset loss function. The training stops when the preset training termination condition is met, and the trained voiceprint feature extraction network and voiceprint recognition scoring network are obtained. Step S2: When the registration voice sent by the registered user is received, the voiceprint feature vector of the registered voice is predicted based on the trained voiceprint feature extraction network to obtain the registered voiceprint feature vector of the registrant, and a voiceprint feature database is constructed based on the registered voiceprint feature vector. Step S3: Perform speech data preprocessing on the speech data to be recognized, and then extract acoustic features to obtain the acoustic features to be recognized. The acoustic features to be identified are input into the trained voiceprint feature extraction network to obtain the voiceprint feature vector to be identified. Iterate through each registered voiceprint feature vector in the voiceprint feature database, take the current registered voiceprint feature vector and the voiceprint feature vector to be identified as the current scoring object pair, and input them into the trained voiceprint recognition scoring network to obtain the scoring result of the current scoring object pair; and take the speaker of the registered voiceprint feature vector corresponding to the highest score as the recognition result of the speech data to be identified, and check whether the scoring result corresponding to the recognition result is greater than or equal to the preset global threshold. If so, output the recognition result; otherwise, the recognition result is invalid.

2. The method as described in claim 1, characterized in that, The speech data preprocessing includes: speech format adjustment, speech activity detection, and speech fragmentation.

3. The method as described in claim 2, characterized in that, In step S1, the preprocessing of the speech data of the training samples also includes adding noise and reverberation.

4. The method as described in claim 1, characterized in that, The acoustic features used are 80-dimensional F-bank features.

5. The method as described in claim 1, characterized in that, The network structure of the voiceprint feature extraction network is as follows: a first convolutional block, first, second and third backbone network blocks, a second convolutional block, an attention pooling network block, a fully connected layer with batch normalization operation and a Softmax layer. The input to the second convolutional block also includes the output feature maps of the first backbone network block and the second backbone network block; The network structure of each convolutional block includes, in sequence, a one-dimensional convolutional layer, an activation function layer, and a batch normalization layer; Each backbone network block is a multi-scale backbone network with SE network; The attention pooling network block includes: pooling operations that introduce an attention mechanism, and batch normalization operations.

6. The method as described in claim 1, characterized in that, The scoring results of the voiceprint recognition scoring network are set as follows: The two speakers in the input voiceprint feature vector pair to the voiceprint recognition scoring network are defined as two roles: the template speaker and the speaker to be identified; and the voiceprint feature vector of the template speaker is defined as... The voiceprint feature vector of the speaker to be identified is defined as ; The scoring results of the voiceprint recognition scoring network are as follows: + ; in, , They represent , The feature vector obtained after two fully connected layers.

7. The method according to any one of claims 1 to 6, characterized in that, The loss function used in the voiceprint recognition scoring network is as follows: ； ； ； ； in, and These represent the costs associated with the false rejection rate (FR) and the correct acceptance rate (FA), respectively. This represents the prior probability of the actual speaker appearing. Indicates whether sample i is a positive sample; when it is a positive sample... =1, negative samples =0, if the current voiceprint feature vector pair belongs to the same speaker, it is a positive sample; otherwise, it is a negative sample. Indicates whether the model predicts a score. Indicates the global threshold. This represents a preset constant. This represents the sigmoid function, and N represents the number of samples.

Images