Speaker verification method, terminal and storage medium based on deep neural network

Abstract

The present invention discloses a speaker verification method, terminal and storage medium based on a deep neural network, the method comprising: obtaining speech data of multiple speakers in a preset data set; converting the multiple speech data into a two-dimensional data set by preprocessing, and dividing the two-dimensional data set into a training set and a verification set according to a preset ratio; constructing a deep neural network according to a residual neural network and a long short-term memory network, training and verifying the deep neural network through the training set and the verification set to obtain a trained deep neural network; predicting the input audio information of several speakers to be tested through the trained deep neural network, and outputting the verification result of the speaker to be tested. The present invention makes full use of the frequency domain characteristics and time domain characteristic information of the audio data, and focuses on the audio data that is not easy to be distinguished when updating the parameters of the network model, so that the trained model can complete the speaker verification task more accurately and improve the performance of the speaker verification system.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a speaker verification method, terminal and storage medium based on a deep neural network. Background Art

[0002] With the development of artificial intelligence and the improvement of computer computing power, speaker verification methods based on deep neural networks have achieved good results in recent years and have gradually been applied in smart mobile devices, smart security and other fields.

[0003] At present, most of the speaker verification methods based on deep neural networks are based on simple convolutional neural networks or recurrent neural networks. For example, in Google's speaker verification solution based on GE2E loss [1], in order to extract the embedding vector of each speech data, the solution uses a three-layer LSTM (Long Short-Term Memory) network. The specific structure is as follows: Figure 2 shown.

[0004] For example, some speaker verification schemes have adopted a method combining convolutional neural networks and recurrent neural networks to construct an audio feature extraction network, but when training the network model, they use softmax cross entropy loss (softmax activation function is often connected to the last layer of the neural network for classification tasks; in classification tasks, the commonly used cross entropy loss (cross entropy loss) is used as the loss function) to train the model. For example, in the voiceprint recognition method based on ResNet (residual neural network)-LSTM [2], the audio feature vector extracted by ResNet+LSTM is mapped into a 340-dimensional vector (340 speakers in the training set), and the cross entropy loss of the model is calculated together with the real label after the softmax activation function, and then the model parameters are updated by back propagation.

[0005] The main disadvantage of the existing speaker verification scheme is that only convolutional neural networks or recurrent neural networks are used to extract the embedding vector of speech data. Since the spectral feature graph of speech data has two dimensions, time domain and frequency domain, if the above existing scheme is adopted, it is impossible to simultaneously consider the time domain features and frequency domain features of speech data when extracting the embedding vector of speech data, resulting in that the feature information of speech data cannot be fully utilized when completing the speaker verification task. In addition, the use of softmax cross entropy loss cannot make the feature vectors learned by samples of the same type closer and closer, and the feature vectors learned by samples of different types farther and farther, making softmax loss unsuitable for processing speaker verification tasks. As a result, the speaker verification model trained by the existing speaker verification technical scheme has high FAR (False Acceptance Rate) and FRR (False Rejection Rate) when completing the speaker verification task, and the system performance cannot reach the ideal state.

[0006] Therefore, the prior art needs to be improved. Summary of the invention

[0007] The technical problem to be solved by the present invention is that, in view of the defects of the prior art, the present invention provides a speaker verification method, a terminal and a storage medium based on a deep neural network to solve the problem that the existing speaker verification method cannot simultaneously consider the frequency domain characteristics and the time domain characteristics of the speech data.

[0008] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0009] In a first aspect, the present invention provides a speaker verification method based on a deep neural network, comprising:

[0010] Obtaining speech data of multiple speakers in a preset data set;

[0011] Converting the plurality of speech data into a two-dimensional data set by preprocessing, and dividing the two-dimensional data set into a training set and a validation set according to a preset ratio;

[0012] Constructing a deep neural network based on the residual neural network and the long short-term memory network, training and verifying the deep neural network through the training set and the verification set to obtain a trained deep neural network;

[0013] The trained deep neural network is used to predict the input audio information of several speakers to be tested, and the verification result of the speakers to be tested is output.

[0014] In one implementation, the step of obtaining speech data of a plurality of speakers in a preset data set includes:

[0015] The speech data of multiple speakers in the preset data set are read at a preset sampling rate to obtain multiple audio time series and corresponding audio sampling rates.

[0016] In one implementation, converting the plurality of speech data into a two-dimensional data set by preprocessing, and dividing the two-dimensional data set into a training set and a validation set according to a preset ratio, includes:

[0017] Convert the plurality of audio time series into corresponding complex matrices by short-time Fourier transform;

[0018] Determine whether the frame number of each complex matrix is ​​greater than a preset frame number;

[0019] If it is greater than or equal to the preset number of frames, then selecting the preset frame data arranged at the tail of the current complex matrix, and updating the corresponding audio file according to the selected preset frame data;

[0020] If it is less than the preset number of frames, the audio file corresponding to the current complex matrix is ​​ignored;

[0021] All selected complex matrices are divided into the training set and the validation set according to the preset ratio, and stored in the corresponding paths in a specified format.

[0022] In one implementation, the residual neural network includes: a convolution layer with 64 convolution kernels of size 7×7 and stride 2, a pooling layer with a pooling window size of 3×3 and stride 2, and 12 residual blocks;

[0023] The long short-term memory network includes: three long short-term memory layers with a hidden layer dimension of 128 and an output vector dimension of 64.

[0024] In one implementation, the training and verification of the deep neural network using the training set and the verification set to obtain the trained deep neural network includes:

[0025] Acquire audio features of the audio data in the training set;

[0026] Input the audio features into the deep neural network, connect them to the fully connected layer through the last output signal of the last long short-term memory layer, and map them to a preset dimension using the fully connected layer to obtain a matrix E of a specified size;

[0027] Calculate a contrast loss value based on the output of the deep neural network;

[0028] Optimizing the deep neural network according to a gradient descent algorithm and the contrast loss value;

[0029] The optimized deep neural network is evaluated according to the verification set, and the network model is saved according to the evaluation result to obtain the trained deep neural network.

[0030] In one implementation, obtaining audio features of audio data in a training set includes:

[0031] Reading an audio file in the training set and obtaining a complex matrix in the audio file;

[0032] The obtained complex matrix is ​​disrupted to obtain the complex matrix of the first M audio data of the first N speakers after the disorder;

[0033] Modulo the shuffled complex matrix to obtain the amplitude spectrum of the corresponding audio data;

[0034] Performing a square operation on the amplitude spectrum to obtain an energy spectrum of the audio data;

[0035] Constructing a Mel filter bank, and taking the logarithm of the energy spectrum after passing it through the Mel filter bank to obtain the audio features of the corresponding audio data;

[0036] The obtained audio features are used as input of the deep neural network.

[0037] In one implementation, calculating the contrast loss value according to the output of the deep neural network includes:

[0038] Calculate the embedding center vector of each speaker among the first N speakers;

[0039] Calculate the similarity between each speaker’s embedding center vector and the embedding center vectors of other speakers;

[0040] A corresponding similarity matrix is ​​obtained according to the similarity, and the contrast loss value is calculated according to the similarity matrix.

[0041] In one implementation, the trained deep neural network predicts the input audio information of the plurality of speakers to be tested, and outputs the verification result of the speakers to be tested, including:

[0042] Input several audio data of registered speakers and several audio data of speakers to be tested;

[0043] The audio data of each speaker is subjected to Fourier transform and audio feature extraction operations to obtain corresponding audio features;

[0044] Generate an embedding vector corresponding to the audio feature through the trained deep neural network, and obtain an embedding center vector of the registered speaker and an embedding center vector of the speaker to be tested;

[0045] Calculating the cosine similarity between the embedding center vector of the registered speaker and the embedding center vector of the speaker to be tested;

[0046] Determining whether the cosine similarity exceeds a preset threshold;

[0047] If yes, it is determined that the speaker to be tested is the same person as the registered speaker, and a verification result of the speaker to be tested is output.

[0048] In a second aspect, the present invention further provides a terminal, comprising: a processor and a memory, wherein the memory stores a speaker verification program based on a deep neural network, and when the speaker verification program based on a deep neural network is executed by the processor, it is used to implement the operation of the speaker verification method based on a deep neural network as described in the first aspect.

[0049] In a third aspect, the present invention further provides a storage medium, which is a computer-readable storage medium, and which stores a speaker verification program based on a deep neural network. When the speaker verification program based on a deep neural network is executed by a processor, it is used to implement the operation of the speaker verification method based on a deep neural network as described in the first aspect.

[0050] The present invention adopts the above technical solution to achieve the following effects:

[0051] The present invention constructs a novel speaker verification network model training method combining Fbank features, ResNet, LSTM and contrast loss, which can make full use of the frequency domain characteristics and time domain characteristic information of audio data when learning the speaker verification network model, and at the same time focus on audio data that is not easy to distinguish when updating the parameters of the network model. Thus, the trained speaker verification model can complete the speaker verification task more accurately and improve the performance of the speaker verification system. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0053] Figure 1 It is a flow chart of a speaker verification method based on a deep neural network in one implementation of the present invention.

[0054] Figure 2 It is a schematic diagram of the existing speaker verification scheme based on GE2E loss.

[0055] Figure 3 It is a schematic diagram of Fourier transform in one implementation of the present invention.

[0056] Figure 4 It is a schematic diagram of the structure of a deep neural network in one implementation of the present invention.

[0057] Figure 5 It is a flowchart of training and verifying a deep neural network in one implementation of the present invention.

[0058] Figure 6 It is a schematic diagram of audio feature extraction of an audio file in an implementation of the present invention.

[0059] Figure 7 It is a functional principle diagram of a terminal in one implementation of the present invention.

[0060] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0062] Exemplary Methods

[0063] At present, speaker verification methods based on deep neural networks have the following two problems: First, most speaker verification methods are based on convolutional neural networks or recurrent neural networks alone, so there is a problem that the frequency domain features and time domain features of speech data cannot be considered at the same time. Another speaker verification method uses softmax cross entropy loss (softmax loss function) as the loss function when training the speaker verification network model. However, because softmax loss cannot make the feature vectors learned by samples of the same type closer and closer, and the feature vectors learned by samples of different types farther and farther away, softmax loss is not suitable for speaker verification tasks.

[0064] In view of the above technical problems, this embodiment proposes a new speaker verification network model training method that combines Fbank features (audio features obtained by processing audio in a manner similar to the human ear), ResNet (residual neural network), LSTM (long short-term memory network) and contrast loss (contrast loss), which can make full use of the frequency domain features and time domain feature information of audio data when learning the speaker verification network model, and at the same time focus on audio data that is not easy to distinguish when updating the parameters of the network model. Thereby reducing the false acceptance rate (FAR) and false rejection rate (FRR) of the speaker verification system and improving the performance of the speaker verification system.

[0065] like Figure 1 As shown, an embodiment of the present invention provides a speaker verification method based on a deep neural network, comprising the following steps:

[0066] Step S100, obtaining speech data of multiple speakers in a preset data set.

[0067] In this embodiment, the speaker verification method based on deep neural network is applied in a terminal, and the terminal includes but is not limited to: a computer, a mobile terminal and other devices.

[0068] In this embodiment, the method can be applied to a speaker verification system based on a deep neural network, and the application scenarios include: authorization of access control of various things in personal daily life, such as smartphone lock screen, voice-controlled password locks of various network accounts, computer voice-controlled locks, voice-controlled security doors, car voice-controlled locks, etc.

[0069] Traditional speech verification schemes based on deep neural networks can often only take into account information in one dimension of the time domain features or the frequency domain features in the speech data. By introducing the scheme of this embodiment, the speaker verification system can consider both the time domain features and the frequency domain features of the speech data when performing speaker verification, thereby completing the speaker verification task more accurately.

[0070] In this embodiment, the main process is divided into four main stages: data preprocessing stage, embedding vector extraction network construction stage, network model training stage and model prediction stage.

[0071] Among the four main stages in this embodiment, the data preprocessing stage refers to receiving the speech data of different speakers in the data set, converting the speech data into a two-dimensional array after preprocessing, and saving it in a training set and a validation set at a ratio of 9:1. The embedding vector extraction network construction stage refers to building a deep neural network based on ResNet+LSTM to extract the embedding features of the speech data. The network model training stage refers to training the network model using the embedding vectors and real labels of the speech data extracted by the ResNet+LSTM deep neural network; the model prediction stage refers to when the speaker recognition system is in use, it gives several pieces of speech data of the registered speaker and several pieces of speech data of the speaker to be tested, and the system uses the model prediction results to determine whether the speaker to be tested is the same person as the registered speaker.

[0072] Specifically, in an implementation of this embodiment, step S100 includes the following steps:

[0073] Step S101 , reading speech data of multiple speakers in the preset data set at a preset sampling rate to obtain multiple audio time series and corresponding audio sampling rates.

[0074] In this embodiment, it is necessary to input a preset data set in the speaker verification system based on a deep neural network, where the preset data set is an audio data set, which includes speech audio files of different speakers; in the data preprocessing stage, the speech audio files of different speakers in the preset data set can be read at a sampling rate of 16000, and the audio time series corresponding to the read speech audio file and the audio sampling rate can be returned in the system.

[0075] In this embodiment, by inputting a preset data set into the speaker verification system based on a deep neural network, the speech audio files of different speakers in the preset data set can be used to train and test the constructed deep neural network based on ResNet+LSTM, so as to use the trained deep neural network to perform audio verification on the speaker to be verified.

[0076] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the speaker verification method based on the deep neural network also includes the following steps:

[0077] Step S200: converting the plurality of speech data into a two-dimensional data set through preprocessing, and dividing the two-dimensional data set into a training set and a validation set according to a preset ratio.

[0078] In this embodiment, after obtaining multiple audio time series and corresponding audio sampling rates, the multiple voice data can be converted into a two-dimensional data group through short-time Fourier transform to obtain the training data required for the deep network model; and, divided into a training set and a validation set in a ratio of 9:1, and saved to the corresponding path in the .npy file format.

[0079] Specifically, in an implementation of this embodiment, step S200 includes the following steps:

[0080] Step S201, converting the plurality of audio time series into corresponding complex matrices by short-time Fourier transform;

[0081] Step S202, determining whether the number of frames of each complex matrix is ​​greater than a preset number of frames;

[0082] Step S203, if it is greater than or equal to the preset number of frames, selecting the preset frame data arranged at the tail of the current complex matrix, and updating the corresponding audio file according to the selected preset frame data;

[0083] Step S204: if the number of frames is less than the preset number of frames, the audio file corresponding to the current complex matrix is ​​ignored;

[0084] Step S205 , dividing all selected complex matrices into the training set and the validation set according to the preset ratio, and storing them in corresponding paths in a specified format.

[0085] In this embodiment, during the short-time Fourier transform process, the audio sequence is converted into a corresponding complex matrix by short-time Fourier transform (STFT), and the size of the complex matrix is:

[0086]

[0087] Among them, n_fft is the FFT window size 512, hop_length is the frame shift 160, and win_length is the window length 320.

[0088] Specifically, Figure 3 As shown, the required audio file is first obtained, and then the sample audio data is obtained through a sampling rate of 16000, and then a plurality of sub-audio data are obtained through frame processing; Fourier transform is performed using windowing technology, and the complex matrix, i.e., the Fourier transform (STFT) matrix, is obtained in a stacked and spliced ​​manner.

[0089] Furthermore, after obtaining the complex matrix, it is then determined whether the number of frames of the generated complex matrix meets the predetermined number of frames of 80 frames. If it does not meet the predetermined number of frames of 80 frames, the audio file is ignored; if it meets the predetermined number of frames of 80 frames, the last 80 frames of the complex matrix generated by the audio file are taken to represent the audio file, that is, the 80 frames of data arranged at the tail of the current complex matrix are selected, and the corresponding audio file is updated according to the selected 80 frames of data; finally, all the obtained complex matrices are divided into a training set and a verification set in a ratio of 9:1, and saved in the corresponding path in the .npy file format.

[0090] In this embodiment, the sample audio data set is preprocessed by short-time Fourier transform, and then the constructed deep neural network is trained, so that the speaker verification system based on the deep neural network can consider both the time domain characteristics and the frequency domain characteristics of the speech data when performing speaker verification, thereby completing the speaker verification task more accurately.

[0091] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the speaker verification method based on the deep neural network also includes the following steps:

[0092] Step S300, constructing a deep neural network based on the residual neural network and the long short-term memory network, training and verifying the deep neural network through the training set and the verification set, and obtaining a trained deep neural network.

[0093] In existing speaker recognition solutions, using convolutional neural networks or recurrent neural networks alone to extract embedded vectors of speech data often misses information on one dimension of the time domain features or frequency domain features of the speech data.

[0094] In order to solve this defect, a new deep neural network based on ResNet+LTSM is constructed in this embodiment to extract the embedding vector of audio data. The new deep neural network combines the characteristics of ResNet and LSTM networks, and can consider the time domain characteristics and frequency domain characteristics of audio data at the same time when extracting the embedding vector of audio data.

[0095] Specifically, Figure 4 As shown in the figure, the ResNet network part includes: a convolution layer with 64 convolution kernels of size 7×7 and stride 2, a pooling layer with a pooling window size of 3×3 and stride 2, and 12 residual blocks; the LSTM network part includes: three long short-term memory layers with a hidden layer dimension of 128 and an output vector dimension of 64.

[0096] In this embodiment, after constructing a new deep neural network, the new deep neural network is trained using the training set; after each round of training is completed, the new deep neural network is scored using the test set, and the next round of training is performed based on the test score feedback until the model training is completed.

[0097] Specifically, in an implementation of this embodiment, training and verifying the deep neural network using the training set and the verification set includes the following steps:

[0098] Step S301, obtaining audio features of the audio data in the training set;

[0099] Step S302, inputting the audio features into the deep neural network, and connecting to a fully connected layer through the last output signal of the last long short-term memory layer, and mapping to a preset dimension by using the fully connected layer to obtain a matrix E of a specified size;

[0100] Step S303, calculating a contrast loss value according to the output of the deep neural network;

[0101] Step S304, optimizing the deep neural network according to the gradient descent algorithm and the contrast loss value;

[0102] Step S305, evaluating the optimized deep neural network according to the verification set, and saving the network model according to the evaluation result to obtain the trained deep neural network.

[0103] like Figure 5 As shown, in this embodiment, each round of training of the deep neural network model includes five steps: obtaining input audio Fbank features, extracting audio embedding vectors, calculating model loss, back-propagating to update model parameters, and evaluating model performance.

[0104] Since the human ear responds nonlinearly to the sound spectrum, the audio features in the audio data, namely the Fbank features, can be obtained through the Fbank algorithm; the Fbank algorithm is a front-end processing algorithm that processes the audio in a manner similar to the human ear, which can improve the performance of speech recognition. In general, the steps to obtain the Fbank features of the speech signal are: pre-emphasis, framing, windowing, short-time Fourier transform (STFT), mel filtering, de-averaging, etc. The mfcc features can be obtained by performing discrete cosine transform (DCT) on the Fbank.

[0105] Specifically, in an implementation of this embodiment, step S301 includes the following steps:

[0106] Step S301a, reading the audio file in the training set and obtaining the complex matrix in the audio file;

[0107] Step S301b, shuffling the order of the obtained complex matrix to obtain the complex matrix of the first M audio data of the first N speakers after the order is changed;

[0108] Step S301c, taking the modulus of the shuffled complex matrix to obtain the amplitude spectrum of the corresponding audio data;

[0109] Step S301d, performing a square operation on the amplitude spectrum to obtain an energy spectrum of the audio data;

[0110] Step S301e, constructing a Mel filter bank, passing the energy spectrum through the Mel filter bank and taking the logarithm to obtain the audio features of the corresponding audio data;

[0111] Step S301f, using the obtained audio features as input of the deep neural network.

[0112] like Figure 6 As shown in the figure, in the process of obtaining the Fbank feature, the .npy file of the training set is first read to obtain the STFT matrix of the training set audio file, which is then shuffled to obtain the STFT matrix of the first M audio data of the first N speakers after shuffling; after taking the modulus of the STFT matrix, the amplitude spectrum of the audio data can be obtained; then, the amplitude spectrum is squared to obtain the energy spectrum of the audio data; finally, a Mel filter bank is constructed, and the energy spectrum is passed through the Mel filter bank and the logarithm is taken to obtain the Fbank feature of the audio data, and this feature is used as the input of the network model.

[0113] In this embodiment, the Fbank feature is input into the deep neural network based on ResNet+LSTM, and the last output signal of the last layer of LSTM is mapped to a low dimension through the fully connected layer, and the final matrix E of (N*M)×(DenseDim) size is obtained; that is, the embedding vector of this batch of audio data is obtained through the output of the deep neural network based on ResNet+LSTM, which can be used ji The embedding vector representing the i-th audio of the j-th speaker.

[0114] After obtaining the output matrix E of the deep neural network, the contrast loss value can be calculated based on the output matrix E.

[0115] Specifically, in an implementation of this embodiment, step S303 includes the following steps:

[0116] Step S303a, calculating the embedding center vector of each speaker among the first N speakers;

[0117] Step S303b, calculating the similarity between the embedding center vector of each speaker and the embedding center vectors of other speakers;

[0118] Step S303c: obtaining a corresponding similarity matrix according to the similarity, and calculating the contrast loss value according to the similarity matrix.

[0119] In this embodiment, the contrast loss of the model is calculated according to the output of the deep neural network model, and the process is as follows:

[0120] First, calculate the embedding center vector of each speaker in this batch of data: With the embedding center vector of each speaker, we can calculate the similarity between the embedding vector of the batch of audio data and the embedding center vector of each speaker: ji,k =w·cos(e ji ,c k )+b, and obtain a similarity matrix S; finally calculate the model loss:

[0121]

[0122] After obtaining the loss value of the deep neural network model, the entire model can be iteratively optimized based on the model's loss Loss based on the gradient descent optimization algorithm Adam or SGD; finally, the model is evaluated using the validation set, and the model is saved if the model performance improves.

[0123] In this embodiment, in the process of using the validation set to evaluate the model, the validation set is first processed by the deep neural network model to obtain a loss value (this process is the same as the processing process of the training set); then, the optimal performance of the model is refreshed according to the obtained loss value, that is, the model parameters are updated according to the loss value; the refreshed performance is used to determine whether the deep neural network model can be saved; if the requirements for model saving are not met, it is further determined whether the current number of training times reaches the set number of training times; if the set number of training times is reached, the training is terminated; if the set number of training times is not reached, the next round of training process is performed.

[0124] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the speaker verification method based on the deep neural network also includes the following steps:

[0125] Step S400: predicting the input audio information of several speakers to be tested by using the trained deep neural network, and outputting the verification result of the speakers to be tested.

[0126] In this embodiment, in the model prediction stage, several audio data of registered speakers and several audio data of speakers to be tested are input, and the speaker is predicted by the trained deep neural network model, so as to determine whether the speaker to be tested and the registered speaker are the same person; it can be understood that this process is a process of using an application such as voice matching to verify whether the speech (i.e., the audio data of the registered speaker) belongs to a specific speaker based on the known voice of the speaker (i.e., the audio data of the registered speaker).

[0127] Specifically, in an implementation of this embodiment, step S400 includes the following steps:

[0128] Step S401, inputting a plurality of audio data of registered speakers and a plurality of audio data of speakers to be tested;

[0129] Step S402, performing Fourier transform and audio feature extraction operations on the audio data of each speaker to obtain corresponding audio features;

[0130] Step S403, generating an embedding vector corresponding to the audio feature through the trained deep neural network, and obtaining an embedding center vector of the registered speaker and an embedding center vector of the speaker to be tested;

[0131] Step S404, calculating the cosine similarity between the embedding center vector of the registered speaker and the embedding center vector of the speaker to be tested;

[0132] Step S405, determining whether the cosine similarity exceeds a preset threshold;

[0133] Step S406: If yes, determine that the speaker to be tested and the registered speaker are the same person, and output the verification result of the speaker to be tested.

[0134] In this embodiment, in the model prediction stage, several audio data of registered speakers and several audio data of speakers to be tested are input. Figure 3 As shown in Figure 6 The process shown in is converted into Fbank features, and then the embedded vector of the audio is generated by the above-mentioned trained deep neural network model, and the embedded center vectors of the registered speaker and the speaker to be tested can be calculated. Finally, the cosine similarity between the embedded center vectors of the registered speaker and the speaker to be tested is calculated. If it exceeds a preset threshold (for example, the cosine similarity threshold can be set to 95%), the system determines that the speaker to be tested is the same person as the registered speaker.

[0135] This embodiment achieves the following technical effects through the above technical solution:

[0136] This embodiment constructs a new speaker verification network model training method that combines Fbank features, ResNet, LSTM, and contrast loss, which can make full use of the frequency domain and time domain feature information of audio data when learning the speaker verification network model, and at the same time focus on audio data that is not easy to distinguish when updating the parameters of the network model. Therefore, the trained speaker verification model can complete the speaker verification task more accurately and improve the performance of the speaker verification system.

[0137] Exemplary Devices

[0138] Based on the above embodiments, the present invention also provides a terminal, comprising: a processor, a memory, an interface, a display screen and a communication module connected via a system bus; wherein the processor is used to provide computing and control capabilities; the memory includes a storage medium and an internal memory; the storage medium stores an operating system and a computer program; the internal memory provides an environment for the operation of the operating system and the computer program in the storage medium; the interface is used to connect external devices, such as mobile terminals and computers; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or a mobile terminal.

[0139] When the computer program is executed by the processor, it is used to implement the operation of a speaker verification method based on a deep neural network.

[0140] It can be understood by those skilled in the art that Figure 7 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0141] In one embodiment, a terminal is provided, comprising: a processor and a memory, wherein the memory stores a speaker verification program based on a deep neural network, and when the speaker verification program based on the deep neural network is executed by the processor, it is used to implement the operation of the speaker verification method based on the deep neural network as described above.

[0142] In one embodiment, a storage medium is provided, wherein the storage medium stores a speaker verification program based on a deep neural network, and when the speaker verification program based on the deep neural network is executed by the processor, it is used to implement the operation of the speaker verification method based on the deep neural network as described above.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory.

[0144] In summary, the present invention provides a speaker verification method, terminal and storage medium based on a deep neural network, the method comprising: obtaining speech data of multiple speakers in a preset data set; converting the multiple speech data into a two-dimensional data set by preprocessing, and dividing the two-dimensional data set into a training set and a verification set according to a preset ratio; constructing a deep neural network according to a residual neural network and a long short-term memory network, training and verifying the deep neural network through the training set and the verification set to obtain a trained deep neural network; predicting the input audio information of several speakers to be tested through the trained deep neural network, and outputting the verification result of the speaker to be tested. The present invention makes full use of the frequency domain characteristics and time domain characteristic information of the audio data, and focuses on the audio data that is not easy to distinguish when updating the parameters of the network model, so that the trained model can complete the speaker verification task more accurately and improve the performance of the speaker verification system.

[0145] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A speaker verification method based on deep neural network, characterized in that: The speaker verification method based on deep neural network includes: Obtaining speech data of multiple speakers in a preset data set; Converting the plurality of speech data into a two-dimensional data set by preprocessing, and dividing the two-dimensional data set into a training set and a validation set according to a preset ratio; Constructing a deep neural network based on the residual neural network and the long short-term memory network, training and verifying the deep neural network through the training set and the verification set to obtain a trained deep neural network; Predicting the input audio information of the plurality of speakers to be tested by using the trained deep neural network, and outputting the verification result of the speakers to be tested; The step of training and verifying the deep neural network by using the training set and the verification set to obtain a trained deep neural network includes: Acquire audio features of the audio data in the training set; Input the audio features into the deep neural network, connect them to the fully connected layer through the last output signal of the last long short-term memory layer, and map them to a preset dimension using the fully connected layer to obtain a matrix E of a specified size; The contrast loss value is calculated according to the output of the deep neural network; the contrast loss value is calculated as follows: ； Optimizing the deep neural network according to a gradient descent algorithm and the contrast loss value; Evaluating the optimized deep neural network according to the validation set, and saving the network model according to the evaluation result to obtain the trained deep neural network; The obtaining of audio features of the audio data in the training set includes: Reading an audio file in the training set and obtaining a complex matrix in the audio file; The obtained complex matrix is ​​disrupted to obtain the complex matrix of the first M audio data of the first N speakers after the disorder; Modulo the shuffled complex matrix to obtain the amplitude spectrum of the corresponding audio data; Squaring the amplitude spectrum to obtain an energy spectrum of the corresponding audio data; Constructing a Mel filter bank, and taking the logarithm of the energy spectrum after passing the Mel filter bank to obtain the audio features of the corresponding audio data; Using the obtained audio features as input of the deep neural network; The calculating the contrast loss value according to the output of the deep neural network includes: Calculate the embedding center vector of each speaker in the first N speakers; the embedding center vector is: ; Calculate the similarity between each speaker's embedding center vector and the embedding center vectors of other speakers; the similarity is: ; The embedding vector representing the i-th audio of the j-th speaker; A corresponding similarity matrix is ​​obtained according to the similarity, and the contrast loss value is calculated according to the similarity matrix.

2. The speaker verification method based on deep neural network according to claim 1, characterized in that: The step of obtaining speech data of multiple speakers in a preset data set includes: The speech data of multiple speakers in the preset data set are read at a preset sampling rate to obtain multiple audio time series and corresponding audio sampling rates.

3. The speaker verification method based on deep neural network according to claim 1, characterized in that: The preprocessing of converting the plurality of speech data into a two-dimensional data set, and dividing the two-dimensional data set into a training set and a verification set according to a preset ratio, comprises: Convert multiple audio time series into corresponding complex matrices through short-time Fourier transform; Determine whether the frame number of each complex matrix is ​​greater than a preset frame number; If it is greater than or equal to the preset number of frames, then selecting the preset frame data arranged at the tail of the current complex matrix, and updating the corresponding audio file according to the selected preset frame data; If it is less than the preset number of frames, the audio file corresponding to the current complex matrix is ​​ignored; All selected complex matrices are divided into the training set and the validation set according to the preset ratio, and stored in the corresponding paths in a specified format.

4. The speaker verification method based on deep neural network according to claim 1, characterized in that: The residual neural network includes: a convolution layer with 64 convolution kernels of size 7×7 and stride 2, a pooling layer with a pooling window size of 3×3 and stride 2, and 12 residual blocks; The long short-term memory network includes: three long short-term memory layers with a hidden layer dimension of 128 and an output vector dimension of 64.

5. The speaker verification method based on deep neural network according to claim 1, characterized in that: The trained deep neural network predicts the input audio information of the multiple speakers to be tested, and outputs the verification result of the speakers to be tested, including: Input several audio data of registered speakers and several audio data of speakers to be tested; The audio data of each speaker is subjected to Fourier transform and audio feature extraction operations to obtain corresponding audio features; Generate an embedding vector corresponding to the audio feature through the trained deep neural network, and obtain an embedding center vector of the registered speaker and an embedding center vector of the speaker to be tested; Calculating the cosine similarity between the embedding center vector of the registered speaker and the embedding center vector of the speaker to be tested; Determining whether the cosine similarity exceeds a preset threshold; If yes, it is determined that the speaker to be tested is the same person as the registered speaker, and a verification result of the speaker to be tested is output.

6. A terminal, characterized in that: include: A processor and a memory, wherein the memory stores a speaker verification program based on a deep neural network, and when the speaker verification program based on the deep neural network is executed by the processor, it is used to implement the operation of the speaker verification method based on the deep neural network as described in any one of claims 1 to 5.

7. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and the storage medium stores a speaker verification program based on a deep neural network. When the speaker verification program based on a deep neural network is executed by a processor, it is used to implement the operation of the speaker verification method based on a deep neural network as described in any one of claims 1 to 5.

Images