Voiceprint recognition method and system based on variational autoencoder and adversarial generative network

Abstract

The present invention discloses a voiceprint recognition method and system based on variational autoencoder and adversarial generative network, which belongs to the technical field of voiceprint recognition, and includes: S1, collecting M voice files of each speaker, preprocessing each voice file, and obtaining a spectrogram; S2, constructing a paired data set of the spectrogram; S3, constructing a voiceprint feature extraction network, and extracting the voiceprint features of the spectrogram; specifically: S301, extracting the implicit features of the spectrogram by constructing the condition of variational autoencoder and adversarial generative model; S302, constructing a voiceprint recognition network; S4, training the voiceprint recognition network; S5, converting the first spectrum data X into a speech pattern; S6, and extracting the speech pattern of the speech pattern; S7, and extracting the speech pattern of the speech pattern; S8, and extracting the speech pattern of the speech pattern; S9, and extracting the speech pattern of the speech pattern; S10, and extracting the speech pattern of the speech pattern; S11, and extracting the speech pattern of the speech pattern; S12, and extracting the speech pattern of the speech pattern; S13, and extracting the speech pattern of the speech pattern; S14, and extracting the speech pattern of the speech pattern; S15, and extracting the speech pattern of the speech pattern; S16, and extracting the speech pattern of the speech pattern; S17, and extracting the speech pattern of the speech pattern; S18, and extracting the speech pattern of the speech pattern; S19, and extracting the speech pattern of the speech pattern; S20, and extracting the speech pattern of the speech pattern; S21, and extracting the speech pattern of the speech pattern; S22, and extracting the speech pattern of the speech pattern; S23, and extracting the speech pattern of the speech pattern; S24, and extracting the speech pattern of the speech pattern; S25, and extracting the speech pattern of the speech pattern; S26, and extracting the speech pattern of the speech pattern; S27, and extracting the speech pattern of the speech pattern; S 1 and the second spectrum data X 2 The voiceprint recognition network is respectively imported to obtain two scalars Ds; the binary norm D of the two scalars Ds is obtained, and the judgment result is obtained through the value of the binary norm D.

Description

Technical Field

[0001] The present invention belongs to the technical field of voiceprint recognition, and in particular, relates to a voiceprint recognition method and system based on variational autoencoder and adversarial generative network. Background Art

[0002] As we all know, the so-called voiceprint is the spectrum of sound waves that carry speech information displayed by electroacoustic instruments. The production of human language is a complex physiological and physical process between the human body's language center and the vocal organs. The vocal organs used by people when speaking - tongue, teeth, larynx, lungs, and nasal cavity - vary greatly in size and shape, so the voiceprints of any two people are different. The acoustic characteristics of each person's voice are both relatively stable and variable, and are not absolute and unchanging. This variation can come from physiology, pathology, psychology, simulation, disguise, and is also related to environmental interference. Despite this, since everyone's vocal organs are different, in general, people can still distinguish the voices of different people or judge whether they are the same person's voice. Voiceprint Recognition (VPR), also known as Speaker Recognition, has two categories, namely Speaker Identification and Speaker Verification. The former is used to determine which of several people said a certain speech, which is a "multiple-choice" problem; while the latter is used to confirm whether a certain speech was said by a specified person, which is a "one-to-one discrimination" problem. Different tasks and applications will use different voiceprint recognition technologies. Whether it is recognition or confirmation, the speaker's voiceprint needs to be modeled first, which is the so-called "training" or "learning" process. Obviously, voiceprint recognition often requires the collection of a large amount of voice data, and the number of data items for each speaker under various semantic conditions (including various voices and dialects) in various background environments needs to be balanced. If a speaker fails to collect enough data for a speech uttered in a special dialect (or text semantics) in a special context, the voiceprint recognition model may fail to detect accurately or cannot be recognized when used under such conditions.

[0003] Most traditional methods to solve this problem are oversampling or undersampling, that is, first divide the collected data into several categories according to the speaker, background environment, semantic dialect and other conditions, and extract data with replacement within the category. Similar algorithms include SMOTE, Bagging, bootstrap, etc. On the one hand, this kind of extraction is likely to extract duplicate data, while other data is not extracted. In addition, since the extracted data within the class is highly correlated with the specific data in the class, the general classification logic contained in all classes is not shared with each class. Therefore, it is very likely that for some speakers and their voice scenes with more data collection, their voiceprint recognition rate is high, while for speakers and their voice scenes with less data collection, the model often has model failure phenomena such as reduced accuracy and inability to recognize. Summary of the invention

[0004] Technical purpose: The present invention provides a voiceprint recognition method and system based on variational autoencoder and adversarial generative network; a large amount of unbalanced speech semantic data is obtained by using the background, and the adversarial generative network combined with the variational autoencoder algorithm is used to extract the features in the background data that are only related to the general differences of the speaker's voiceprint but not related to the specific speaker's voiceprint data, and a specially designed generated data sampling strategy is used to achieve data balance, and all generated data are used for voice comparison, thereby improving the recognition accuracy of voiceprint recognition.

[0005] Technical Solution

[0006] The first object of the present invention is to provide a voiceprint recognition method based on variational autoencoder and adversarial generative network, comprising the following steps:

[0007] S1. Collect M voice files of each speaker, pre-process each voice file, and obtain a spectrogram; M is a natural number greater than 1;

[0008] S2, constructing a paired data set of the spectrogram; the paired data set includes N triples (X 1 ,X 2 ,y); where: X 1 is the first grammatical data of the triple, X 2 is the second piece of spectrum data of the triple, the label y is defined as 0 for the same pairing, and the label y is defined as 1 for the different pairing; each piece of data in the same pairing set and each piece of data in the different pairing set are composed of two spectrograms; the two spectrograms of each piece of data in the same pairing set are spectrograms of the same speaker; the two spectrograms of each piece of data in the different pairing set are spectrograms of different speakers;

[0009] S3. Construct a voiceprint feature extraction network to extract the voiceprint features of the spectrogram; specifically:

[0010] S301, extracting implicit features of the spectrogram by constructing a conditional and adversarial generation model of variational autoencoder;

[0011] S302, building a voiceprint recognition network;

[0012] S4, training the voiceprint recognition network;

[0013] S5. X the first spectrum data 1 and the second spectrum data X 2 The voiceprint recognition network is respectively imported to obtain two scalars Ds; the binary norm D of the two scalars Ds is obtained, and the judgment result is obtained through the value of the binary norm D.

[0014] Preferably, S2 is specifically:

[0015] First, two spectrograms are extracted from the spectrogram of the same speaker and paired to obtain a similar pair, and the label y of the similar pair is defined as 0; one spectrogram is extracted from each of the spectrograms of two different speakers and paired to obtain a heterogeneous pair, and the label y of the heterogeneous pair is defined as 1;

[0016] Then, according to the total number of pairs N, the number of speakers is defined as the number of classes k, and the same-class and different-class sampling ratio is , according to the limiting condition: s1 + s2 = N / k, calculate the number of similar pairs s1 and the number of heterogeneous pairs s2 to be extracted for each category:

[0017] ;

[0018] Finally, the same type of pairs, different type of pairs and label y form a data set, which includes N triplets (X 1 ,X 2 ,y).

[0019] Preferably, the S301 is specifically:

[0020] S3011, construct an encoder, which takes the spectrogram data X obtained by processing the original data with the spectrogram data as input, and outputs the implicit feature vector Z through the first convolutional neural network 0 ;

[0021] S3012, construct a sampling layer, the sampling layer is based on the implicit feature vector Z 0 As input, generate m pairs of outputs through a multilayer perceptron. For each pair of outputs, define the first parameter as the mean and the second parameter as the variance, and sample a value t from the standard normal distribution. 0 , using the formula: , where: i is a natural number from 1 to m; get the output and form the sampling implicit feature vector Z in the original order 1 ;

[0022] S3013, construct a decoder, the decoder is to sample the hidden feature vector Z 1 As input, a second convolutional neural network is used to obtain a generated spectrogram data with a dimension consistent with the spectrogram data X. ;

[0023] S3014, generating spectrogram data As input, a second convolutional neural network is used to obtain a hidden feature vector Z 0 Different numerical vectors Z 0 ';

[0024] S3015, constructing a real data and generated data discriminator, the real data and generated data discriminator will Marked as y 0 =1,(X,Z 0 ) is marked as y 0 =0, 、(X,Z 0 ) and the mark are taken as input, and pass through the third convolutional neural network and the fully connected layer in sequence, and the output is a scalar Ds;

[0025] S3016. Combine and construct a voiceprint feature extraction network;

[0026] The encoder, decoder, real data and generated data discriminator are connected in sequence to form a voiceprint feature extraction network; the input of the voiceprint feature extraction network is the spectrogram data X, and the output is (Z0, Z0', Z1, , Ds).

[0027] Preferably, the S302 is specifically:

[0028] During training, the input of the voiceprint recognition network includes the first spectral data X 1 and the second spectrum data X 2 and the same and different class labels, after passing through the voiceprint feature extraction network, the output includes the first spectrum data X 1 Characteristics (Z 0 , Z 0 ',Z 1 , , Ds) 1 and the second spectrum data X 2 Characteristics (Z 0 , Z 0 ', Z 1 , , Ds) 2 and predicted same and different class labels;

[0029] When applied, the input of the voiceprint recognition network includes the first spectrum data X 1 and the second spectrum data X 2 , the output is the predicted same or different class labels.

[0030] Preferably, S4 is specifically:

[0031] S401, initializing the voiceprint recognition network;

[0032] S402, triple (X 1 ,X 2 ,y) input into the voiceprint recognition network;

[0033] S403, Order ;

[0034] definition: is the first loss function, where Indicates the value obtained by processing the first piece of data in the same pair of data. The value obtained by processing the second data in the same paired data is represented by using the mini-batch gradient descent method to update the weights of the third convolutional neural network in the real data and generated data discriminator based on the first loss function;

[0035] S404. Order: ;

[0036] Where: n is the one-dimensional dimension of the spectrogram data X after flattening; X i represents the i-th number after the spectrogram data X is flattened, Spectrogram data representing the output of the voiceprint feature extraction network The i-th value of ;

[0037] Get the RMSE of the first paired data, recorded as RMSE (1) , the RMSE of the second data is recorded as RMSE (2) ;

[0038] make: ;

[0039] ;

[0040] Where D is the bi-norm of the first spectrogram output feature and the second spectrogram output feature of the paired data; m 0 is the marginal value, m 0 Greater than or equal to 1; y is the label when inputting paired data during training;

[0041] The second loss function is defined as:

[0042] ;

[0043] Where: L d is the Euclidean space distance loss between two sample points in the latent space, and β and γ are adjusted according to the experimental results;

[0044] Based on the second loss function, use the mini-batch gradient descent method to update the network weights other than the real data and generated data discriminator range;

[0045] S405, replace the previous network with the network after updating the network weights, define the total number of iterations, re-execute S402-S404, and each execution is defined as an iteration; after the iteration is completed, the trained voiceprint recognition network is obtained.

[0046] The second object of the present invention is to provide a voiceprint recognition system based on variational autoencoder and adversarial generative network, comprising:

[0047] Preprocessing module: collect M voice files of each speaker, preprocess each voice file to obtain a spectrogram; M is a natural number greater than 1;

[0048] Paired dataset construction module: constructs a paired dataset of the spectrogram; the paired dataset includes N triples (X 1 ,X 2 ,y); where: X 1 is the first grammatical data of the triple, X 2 is the second piece of spectrum data of the triple, the label y is defined as 0 for the same pairing, and the label y is defined as 1 for the different pairing; each piece of data in the same pairing set and each piece of data in the different pairing set are composed of two spectrograms; the two spectrograms of each piece of data in the same pairing set are spectrograms of the same speaker; the two spectrograms of each piece of data in the different pairing set are spectrograms of different speakers;

[0049] Feature extraction and voiceprint recognition network construction module: Construct a voiceprint feature extraction network to extract the voiceprint features of the spectrogram; specifically, extract the implicit features of the spectrogram by constructing the conditional and adversarial generation model of the variational autoencoder; and construct a voiceprint recognition network;

[0050] Training module: training the voiceprint recognition network;

[0051] Judgment module: the first spectrum data X 1 and the second spectrum data X 2 The voiceprint recognition network is respectively imported to obtain two scalars Ds; the binary norm D of the two scalars Ds is obtained, and the judgment result is obtained through the value of the binary norm D.

[0052] Preferably, the construction process of the paired data set construction module is:

[0053] First, two spectrograms are extracted from the spectrogram of the same speaker and paired to obtain a similar pair, and the label y of the similar pair is defined as 0; one spectrogram is extracted from each of the spectrograms of two different speakers and paired to obtain a heterogeneous pair, and the label y of the heterogeneous pair is defined as 1;

[0054] Then, according to the total number of pairs N, the number of speakers is defined as the number of classes k, and the same-class and different-class sampling ratio is , according to the limiting condition: s1 + s2 = N / k, calculate the number of similar pairs s1 and the number of heterogeneous pairs s2 to be extracted for each category:

[0055] ;

[0056] Finally, the same type of pairs, different type of pairs and label y form a data set, which includes N triplets (X 1 ,X 2 ,y).

[0057] Preferably, the specific process of extracting implicit features of the spectrogram by constructing a conditional and adversarial generative model of a variational autoencoder includes:

[0058] Construct an encoder, which takes the spectrogram data X obtained by processing the original data with the spectrogram data as input, and outputs the implicit feature vector Z through the first convolutional neural network 0 ;

[0059] Construct a sampling layer, which uses the hidden feature vector Z 0 As input, generate m pairs of outputs through a multilayer perceptron. For each pair of outputs, define the first parameter as the mean and the second parameter as the variance, and sample a value t from the standard normal distribution. 0 , using the formula: , where: i is a natural number from 1 to m; get the output and form the sampling implicit feature vector Z in the original order 1 ;

[0060] Construct a decoder that samples the latent feature vector Z 1 As input, a second convolutional neural network is used to obtain a generated spectrogram data with a dimension consistent with the spectrogram data X. ;

[0061] Generate spectrogram data as described As input, a second convolutional neural network is used to obtain a hidden feature vector Z 0 Different numerical vectors Z 0 ';

[0062] Construct a real data and generated data discriminator, which will Marked as y 0 =1,(X,Z 0 ) is marked as y 0 =0, 、(X,Z 0 ) and the mark are taken as input, and pass through the third convolutional neural network and the fully connected layer in sequence, and the output is a scalar Ds;

[0063] Combine and construct a voiceprint feature extraction network;

[0064] The encoder, decoder, real data and generated data discriminator are connected in sequence to form a voiceprint feature extraction network; the input of the voiceprint feature extraction network is the spectrogram data X, and the output is (Z0, Z0', Z1, , Ds).

[0065] Preferably, the step of constructing a voiceprint recognition network specifically includes:

[0066] During training, the input of the voiceprint recognition network includes paired spectrogram data X 1 and paired spectrogram data X 2 and the same and different class labels, after passing through the voiceprint feature extraction network, the output includes the first spectrum data X 1 Characteristics (Z 0 , Z 0 ',Z 1 , , Ds) 1 and the second spectrum data X 2 Characteristics (Z 0 , Z 0 ', Z 1 , , Ds) 2 and predicted same and different class labels;

[0067] When applied, the input of the voiceprint recognition network includes the first spectral data X 1 and the second spectrum data X 2 , the output is the predicted same or different class labels.

[0068] Preferably, the training process of the training module is:

[0069] S401, initializing the voiceprint recognition network;

[0070] S402, triple (X 1 ,X 2 ,y) input into the voiceprint recognition network;

[0071] S403, Order ;

[0072] definition: is the first loss function, where Indicates the value obtained by processing the first piece of data in the same pair of data. The value obtained by processing the second data in the same paired data is represented by using the mini-batch gradient descent method to update the weights of the third convolutional neural network in the real data and generated data discriminator based on the first loss function;

[0073] S404. Order: ;

[0074] Where: n is the one-dimensional dimension of the spectrogram data X after flattening; X i represents the i-th number after the spectrogram data X is flattened, Spectrogram data representing the output of the voiceprint feature extraction network The i-th value of ;

[0075] Get the RMSE of the first paired data, recorded as RMSE (1) , the RMSE of the second data is recorded as RMSE (2) ;

[0076] make: ;

[0077] ;

[0078] Where D is the bi-norm of the first spectrogram output feature and the second spectrogram output feature of the paired data; m 0 is the marginal value, m 0 Greater than or equal to 1; y is the label when inputting paired data during training;

[0079] The second loss function is defined as:

[0080] ;

[0081] Where: L d is the Euclidean space distance loss between two sample points in the latent space, and β and γ are adjusted according to the experimental results;

[0082] Based on the second loss function, use the mini-batch gradient descent method to update the network weights other than the real data and generated data discriminator range;

[0083] S405, replace the previous network with the network after updating the network weights, define the total number of iterations, re-execute S402-S404, and each execution is defined as an iteration; after the iteration is completed, the trained voiceprint recognition network is obtained.

[0084] The advantages and positive effects of the present invention are:

[0085] 1. The paired data set constructed by the present invention combines a fair sampling strategy, so that the probability of each data being sampled is equal, and the result is more reliable and true.

[0086] 2. The present invention combines variational autoencoder and generative adversarial network to design a unique spectrogram feature extraction artificial neural network, which can extract a set of features that are not related to the specific speaker but are related to the differences between speakers.

[0087] 3. The present invention combines the voiceprint feature extraction network to build a unique voiceprint recognition network. Through its balanced sampling strategy, it avoids the problem of unbalanced sample collection between speakers, and achieves the effect of facilitating identification in the implicit space that the feature distance of the samples of the same speaker is close, while the feature distance of the samples of different speakers is far, thereby achieving speaker voiceprint recognition with high accuracy.

[0088] 4. The present invention designs an artificial neural network training method that can be used for voiceprint recognition network. During the training process, the data feature extraction network and the data generation network are trained simultaneously, and the data generation network and the data discrimination network are trained alternately, and the voiceprint recognition goal is achieved by designing a reasonable loss function.

[0089] 5. Compared with other voiceprint recognition artificial neural networks, the present invention extracts the common and different features of all speakers in the data set as the basis for recognition, while ignoring the individual characteristics of the specific speakers. Therefore, for the trained artificial neural network in the present invention, the result judgment accuracy is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 is a flow chart of preprocessing in a preferred embodiment of the present invention;

[0091] Figure 2 is a flow chart of constructing a data set in a preferred embodiment of the present invention;

[0092] Figure 3 is a flow chart of creating an encoder in a preferred embodiment of the present invention;

[0093] Figure 4 is a flow chart of creating a sampling layer in a preferred embodiment of the present invention;

[0094] Figure 5 is a flowchart of creating a decoder in a preferred embodiment of the present invention;

[0095] Figure 6 is a generating diagram of a numerical vector in a preferred embodiment of the present invention;

[0096] Figure 7is a structural diagram of creating real data and generating a data discriminator in a preferred embodiment of the present invention;

[0097] Figure 8 is a structural diagram of a voiceprint feature extraction network in a preferred embodiment of the present invention;

[0098] Fig. 9 is a flow chart of a real data and generated data discriminator in a preferred embodiment of the present invention;

[0099] Fig.10 is a flow chart of a voiceprint feature extraction network in a preferred embodiment of the present invention;

[0100] Fig.11 It is an overall flow chart of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0101] In order to further understand the content, features and effects of the present invention, the following embodiments are given as examples and described in detail with reference to the accompanying drawings.

[0102] See also Figures 1 to 11 , a voiceprint recognition method based on variational autoencoder and adversarial generative network, comprising:

[0103] S1. Collect M voice files of each speaker, preprocess each voice file to obtain a spectrogram; M is a natural number greater than 1; the specific process of preprocessing is as follows Figure 1 As shown:

[0104] The audio file is pre-emphasized, framed, and windowed in sequence to obtain a spectrogram;

[0105] Since the preprocessing technology is an existing technology, it will not be described in detail here;

[0106] S2, constructing a paired data set of the spectrogram; the paired data set includes N triples (X 1 ,X 2 ,y); where: X 1 is the first grammar data, X 2 is the second piece of spectrum data. When the same pairing is used, the label y is defined as 0, and when the heterogeneous pairing is used, the label y is defined as 1. Each piece of data in the same pairing set and each piece of data in the heterogeneous pairing set are composed of two spectrograms. The two spectrograms of each piece of data in the same pairing set are spectrograms of the same speaker; the two spectrograms of each piece of data in the heterogeneous pairing set are spectrograms of different speakers. Please refer to Figure 2 , the construction process of the paired data set is as follows:

[0107] exist Figure 2In order to illustrate the problem, three speakers are taken as an example for detailed description. The original spectrogram samples on the left include spectrogram samples of three speakers, and the spectrograms of each speaker are distinguished by different fillings;

[0108] First, two spectrograms are extracted from the spectrogram of the same speaker and paired to obtain a similar pair, and the label y of the similar pair is defined as 0; one spectrogram is extracted from each of the spectrograms of two different speakers and paired to obtain a heterogeneous pair, and the label y of the heterogeneous pair is defined as 1;

[0109] Then, according to the total number of pairs N, the number of speakers is defined as the number of classes k, and the same-class and different-class sampling ratio is , for example, to meet the purpose of fair sampling, that is, the probability of each speaker being extracted is the same, set the following limiting condition: s1 + s2 = N / k, calculate the number of similar pairs s1 and the number of heterogeneous pairs s2 to be extracted for each category:

[0110] ;

[0111] Finally, the same type of pairs, different type of pairs and label y form a data set, which includes N triplets (X 1 ,X 2 ,y); where: X 1 is the first grammatical data of the triple, X 2 is the second grammatical data of the triple; y is the label.

[0112] The present invention uses replacement sampling pairing for these spectrograms to construct a data set:

[0113] From the spectrogram samples of the same speaker, two samples are extracted each time to complete a pairing, and they are marked as y=0.

[0114] From the spectrogram samples of different speakers, randomly select one from a certain speaker and randomly select one from the spectrograms of other speakers to complete a pairing and mark it with y=1.

[0115] In this way, we extract s1 rounds of similar pairs and s2 rounds of heterogeneous pairs, and we get s1+s2 pairs, which are combined into a data set for training and testing. Sampling with replacement does not have the problem of insufficient number of spectra in each type of spectra.

[0116] S3. Construct a voiceprint feature extraction network to extract the voiceprint features of the spectrogram; specifically:

[0117] S301, extracting implicit features of the spectrogram by constructing a conditional and adversarial generation model of variational autoencoder;

[0118] S3011, build the encoder, see Figure 3The encoder takes the spectrogram data X obtained by processing the original data with the spectrogram data as input, and outputs the implicit feature vector Z through the first convolutional neural network. 0 ;

[0119] S3012, build sampling layer, see Figure 4 , for an input hidden feature vector Z 0 , so that it passes through a custom multi-layer perceptron to generate m pairs of outputs. In each pair of outputs, the first one is defined as the mean and the second one is defined as the variance, and the order in which they are located is the subscript.

[0120] For each pair of outputs, the system samples a value t from the standard normal distribution. 0 , using the formula:

[0121]

[0122] Get its output and form the sampled hidden feature vector Z in its original order 1 .

[0123] That is, the sampling layer uses the implicit feature vector Z 0 As input, generate m pairs of outputs through a multilayer perceptron. For each pair of outputs, define the first parameter as the mean and the second parameter as the variance, and sample a value t from the standard normal distribution. 0 , using the formula:

[0124] ;

[0125] Where: i is a natural number from 1 to m; get the output and form the sampled implicit feature vector Z in the original order 1 ;

[0126] S3013, build a decoder, see Figure 5 , for each input sampled latent feature vector Z 1 , design a custom convolutional neural network to sample the latent feature vector Z 1 After processing, a dimension is formed that is consistent with the input spectrum Figure 1 Consistently generate spectrogram data .

[0127] That is, the decoder uses the sampled implicit feature vector Z 1 As input, a dimension is obtained through the second convolutional neural network with the input spectrum Figure X Consistently generate spectrogram data ;

[0128] S3014, please refer to Figure 6 , so that the spectrogram data is generated After the previous decoder processing, a new implicit feature vector Z should be obtained. 0 Different numerical vectors, named Z 0 '.

[0129] That is, the generated spectrogram data As input, a second convolutional neural network is used to obtain a hidden feature vector Z 0 Different numerical vector, named Z 0 ';

[0130] S3015, please refer to Figure 7 ,Will Mark y 0 =1,(X,Z 0 ) is marked as y 0 = 0 as input, followed by a custom designed convolutional neural network (named as convolutional neural network 3), and its back end is connected to a fully connected layer, so that its output is a scalar, which is defined as Ds. This is used to determine whether the input is a real spectrogram or a generated spectrogram;

[0131] That is, construct a real data and generated data discriminator; the real data and generated data discriminator will Mark y 0 =1,(X,Z 0 ) is marked as y 0 =0, as input, passes through the third convolutional neural network and the fully connected layer in sequence, and the output is a scalar, defined as Ds;

[0132] S3016, please refer to Figure 8 , connect the above encoder, decoder, real data and generated data discriminator to form Figure 8 The structure is named voiceprint feature extraction network, which extracts the Z obtained by the intermediate calculation 0 , Z 0 ', Z 1 , X, , Ds (here D S There are two values, one is the Ds obtained by the X input real data and the generated data discriminator, and the other is Input the real data and generate data discriminator to get Ds);

[0133] That is, a voiceprint feature extraction network is constructed by combining: the encoder, decoder, real data and generated data discriminator are connected in sequence to form a voiceprint feature extraction network; the input of the voiceprint feature extraction network is the spectrogram data X and the output is (Z0, Z0', Z1, , Ds);

[0134] S302, building a voiceprint recognition network;

[0135] See also Fig. 9 ,for Figure 2 Each pair in , after passing through the voiceprint feature extraction network, can generate a set of features, such as Fig. 9 As shown, this figure is named voiceprint recognition network;

[0136] That is, the input of the voiceprint recognition network training (including paired spectrogram data X 1 and paired spectrogram data X 2 and the same and different class labels) pass through the voiceprint feature extraction network, and output includes the first paired spectrogram data X 1 Characteristics (Z 0 ,Z 0 ', Z 1 , , Ds) 1 and the second paired spectrogram data X 2 Characteristics (Z 0 , Z 0 ', Z 1 , , Ds) 2 And the predicted same and different class labels; when applied, only X1 and X2 are input, and the output is the predicted same and different class labels.

[0137] S4, training the voiceprint recognition network; the specific steps are:

[0138] 1) First, the voiceprint recognition network is initialized, that is, the weights of each node in the network are randomly set.

[0139] 2) The training data consists of N paired data and their same and different class labels, which are used as input one by one and imported into the voiceprint recognition network.

[0140] 3) Order ;

[0141] definition: ;

[0142] is the loss function 1, where Indicates the value obtained by processing the first piece of data in the same pair of data. Represents the value obtained by processing the second data in the same paired data, and is defined as follows. Based on this loss function, the mini-batch gradient descent method is used to update the neural network weights in the real data and generated data discriminators.

[0143] 4) Command:

[0144] ;

[0145] Then we can get the RMSE of the first paired data, recorded as RMSE (1) , the RMSE of the second data is recorded as RMSE (2) .

[0146] make:

[0147] ;

[0148] ;

[0149] Where D is the bi-norm of the first spectrogram output feature and the second spectrogram output feature of the paired data. 0 is the marginal value, M 0 Greater than or equal to 1; can be customized in the experiment.

[0150] Based on the above definition, the loss function 2 can be defined as:

[0151] ;

[0152] Among them, β and γ are adjusted automatically according to the experimental results.

[0153] Based on this loss function, the mini-batch gradient descent method is used to update the network weights other than the real data and generated data discriminator range.

[0154] 5) Replace the previous network with the network after updating the network weights, and re-execute steps 2)-4). Each execution is defined as an iteration, and the total number of iterations is defined. If the number is exceeded, it means that the training is successful.

[0155] S5, X the first spectrum 1 And the second spectrum X 2 The voiceprint recognition network is respectively imported to obtain two scalars Ds; the binary norm D of the two scalars Ds is obtained, and the judgment result is obtained through the value of the binary norm D.

[0156] A voiceprint recognition system based on variational autoencoder and adversarial generative network, comprising:

[0157] Preprocessing module: collect M voice files of each speaker, preprocess each voice file to obtain a spectrogram; M is a natural number greater than 1;

[0158] Paired dataset construction module: constructs a paired dataset of the spectrogram; the paired dataset includes N triples (X 1 ,X 2 ,y); where: X 1 is a triple (X 1 ,X 2,y) of the first grammar data, X 2 is a triple (X 1 ,X 2 ,y), the label y is defined as 0 for the same pairing and 1 for the different pairing; each data of the same pairing set and each data of the different pairing set are composed of two spectrograms; the two spectrograms of each data of the same pairing set are spectrograms of the same speaker; the two spectrograms of each data of the different pairing set are spectrograms of different speakers;

[0159] Feature extraction and voiceprint recognition network construction module: Construct a voiceprint feature extraction network to extract the voiceprint features of the spectrogram; specifically, extract the implicit features of the spectrogram by constructing the conditional and adversarial generation model of the variational autoencoder; and construct a voiceprint recognition network;

[0160] Training module: training the voiceprint recognition network;

[0161] Judgment module: the first spectrum data X 1 and the second spectrum data X 2 The voiceprint recognition network is respectively imported to obtain two scalars Ds; the binary norm D of the two scalars Ds is obtained, and the judgment result is obtained through the value of the binary norm D.

[0162] The construction process of the paired data set construction module is as follows:

[0163] First, two spectrograms are extracted from the spectrogram of the same speaker and paired to obtain a similar pair, and the label y of the similar pair is defined as 0; one spectrogram is extracted from each of the spectrograms of two different speakers and paired to obtain a heterogeneous pair, and the label y of the heterogeneous pair is defined as 1;

[0164] Then, according to the total number of pairs N, the number of speakers is defined as the number of classes k, and the same-class and different-class sampling ratio is , according to the limiting condition: s1 + s2 = N / k, calculate the number of similar pairs s1 and the number of heterogeneous pairs s2 to be extracted for each category:

[0165] ;

[0166] Finally, the same type of pairs, different type of pairs and label y form a data set, which includes N triplets (X 1 ,X 2 ,y).

[0167] The specific process of extracting implicit features of the spectrogram by constructing the conditional and adversarial generation model of the variational autoencoder includes:

[0168] Construct an encoder, which takes the spectrogram data X obtained by processing the original data with the spectrogram data as input, and outputs the implicit feature vector Z through the first convolutional neural network 0 ;

[0169] Construct a sampling layer, which uses the hidden feature vector Z 0 As input, generate m pairs of outputs through a multilayer perceptron. For each pair of outputs, define the first parameter as the mean and the second parameter as the variance, and sample a value t from the standard normal distribution. 0 , using the formula: , where: i is a natural number from 1 to m; get the output and form the sampling implicit feature vector Z in the original order 1 ;

[0170] Construct a decoder that samples the latent feature vector Z 1 As input, a second convolutional neural network is used to obtain a generated spectrogram data with a dimension consistent with the spectrogram data X. ;

[0171] Generate spectrogram data as described As input, a second convolutional neural network is used to obtain a hidden feature vector Z 0 Different numerical vectors Z 0 ';

[0172] Construct a real data and generated data discriminator, which will Marked as y 0 =1,(X,Z 0 ) is marked as y 0 =0, 、(X,Z 0 ) and the mark are taken as input, and pass through the third convolutional neural network and the fully connected layer in sequence, and the output is a scalar Ds;

[0173] Combine and construct a voiceprint feature extraction network;

[0174] The encoder, decoder, real data and generated data discriminator are connected in sequence to form a voiceprint feature extraction network; the input of the voiceprint feature extraction network is the spectrogram data X, and the output is (Z0, Z0', Z1, , Ds).

[0175] The construction of the voiceprint recognition network specifically includes:

[0176] During training, the input of the voiceprint recognition network includes the first spectral data X 1 and the second spectrum data X 2and the same and different class labels, after passing through the voiceprint feature extraction network, the output includes paired spectrogram data X 1 Characteristics (Z 0 , Z 0 ',Z 1 , , Ds) 1 and paired spectrogram data X 2 Characteristics (Z 0 , Z 0 ', Z 1 , , Ds) 2 and predicted same and different class labels;

[0177] When applied, the input of the voiceprint recognition network includes the first spectral data X 1 and the second spectrum data X 2 , the output is the predicted same or different class labels.

[0178] The training process of the training module is as follows:

[0179] S401, initializing the voiceprint recognition network;

[0180] S402, triple (X 1 ,X 2 ,y) Input is imported into the voiceprint recognition network;

[0181] S403, Order ;

[0182] definition: is the first loss function, where Indicates the value obtained by processing the first piece of data in the same pair of data. The value obtained by processing the second data in the same paired data is represented by using the mini-batch gradient descent method to update the weights of the third convolutional neural network in the real data and generated data discriminator based on the first loss function;

[0183] S404. Order: ;

[0184] Where: n is the one-dimensional dimension of the spectrogram data X after flattening; X i represents the i-th number after the spectrogram data X is flattened, Spectrogram data representing the output of the voiceprint feature extraction network The i-th value of ;

[0185] Get the RMSE of the first paired data, recorded as RMSE (1) , the RMSE of the second data is recorded as RMSE (2) ;

[0186] make: ;

[0187] ;

[0188] Where D is the bi-norm of the first spectrogram output feature and the second spectrogram output feature of the paired data; m 0 is the marginal value, m 0 Greater than or equal to 1; y is the label when inputting paired data during training;

[0189] The second loss function is defined as:

[0190] ;

[0191] Where: L d is the Euclidean space distance loss between two sample points in the latent space, and β and γ are adjusted according to the experimental results;

[0192] Based on the second loss function, use the mini-batch gradient descent method to update the network weights other than the real data and generated data discriminator range;

[0193] S405, replace the previous network with the network after updating the network weights, define the total number of iterations, re-execute S402-S404, and each execution is defined as an iteration; after the iteration is completed, the trained voiceprint recognition network is obtained.

[0194] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When the use is implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL) or wireless (e.g., infrared, wireless, microwave, etc.) mode) to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc.

[0195] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are within the scope of the technical solution of the present invention.

Claims

1. A voiceprint recognition method based on variational autoencoder and adversarial generative network, characterized in that: The steps include: S1. Collect M voice files of each speaker, pre-process each voice file, and obtain a spectrogram; M is a natural number greater than 1; S2. Construct a paired data set of the spectrogram; the paired data set includes N triples (X1, X2, y); wherein: X1 is the first piece of spectrogram data of the triple, X2 is the second piece of spectrogram data of the triple, label y is defined as 0 for the same type of pairing, and label y is defined as 1 for the heterogeneous pairing; each piece of data in the same type of pairing set and each piece of data in the heterogeneous pairing set are composed of two spectrograms; the two spectrograms of each piece of data in the same type of pairing set are spectrograms of the same speaker; the two spectrograms of each piece of data in the heterogeneous pairing set are spectrograms of different speakers; S3. Construct a voiceprint feature extraction network to extract the voiceprint features of the spectrogram; specifically: S301, extracting implicit features of the spectrogram by constructing a conditional and adversarial generation model of variational autoencoder; specifically: S3011, constructing an encoder, wherein the encoder takes the spectrogram data X obtained by processing the original data with the spectrogram data as input, and outputs a hidden feature vector Z0 through a first convolutional neural network; S3012, construct a sampling layer, wherein the sampling layer takes the implicit feature vector Z0 as input, generates m pairs of outputs through a multilayer perceptron, and for each pair of outputs, defines the previous parameter as the mean μ ii , the latter parameter is the variance σ i , sample a value t0 from the standard normal distribution, using the formula: i =t o ·σ i +μ i , where: i is a natural number from 1 to m; get the output and form the sampled latent feature vector Z1 in the original order; S3013, constructing a decoder, wherein the decoder takes the sampled implicit feature vector Z1 as input, and obtains a generated spectrogram data with a dimension consistent with the spectrogram data X through a second convolutional neural network S3014, generating spectrogram data As input, a numerical vector Z0' different from the latent feature vector Z0 is obtained through the second convolutional neural network; S3015, construct a real data and generated data discriminator, the real data and generated data discriminator will ( Z0') is marked as y0 = 1, (X,,Z0) is marked as y0 = 0, and ( Z0'), (X, Z0) and the mark are taken as input, and pass through the third convolutional neural network and the fully connected layer in sequence, and the output is a scalar Ds; S3016. Combine and construct a voiceprint feature extraction network; The above encoder, decoder, real data and generated data discriminator are connected in sequence to form a voiceprint feature extraction network; the input of the voiceprint feature extraction network is the spectrogram data X, and the output is (Z0, Z0', Z1, Ds); S302, building a voiceprint recognition network; S4, training the voiceprint recognition network; S5. Import the first piece of spectral data X1 and the second piece of spectral data X2 into the voiceprint recognition network respectively to obtain two scalars Ds; obtain the binary norm D of the two scalars Ds, and obtain the judgment result through the value of the binary norm D.

2. The voiceprint recognition method based on variational autoencoder and adversarial generative network according to claim 1 is characterized in that: The S2 is specifically: First, two spectrograms are extracted from the spectrogram of the same speaker and paired to obtain a similar pair, and the label y of the similar pair is defined as 0; one spectrogram is extracted from each of the spectrograms of two different speakers and paired to obtain a heterogeneous pair, and the label y of the heterogeneous pair is defined as 1; Then, according to the total number of pairs N, the number of speakers is defined as the number of classes k, the same-class and different-class sampling ratio is α, and according to the limiting condition: s1+s2=N / k, the number of same-class pairs s1 and different-class pairs s2 to be extracted for each class are calculated: Finally, the same type of pairs, different type of pairs and label y are combined into a data set, and the data set includes N triplets (X1, X2, y).

3. The voiceprint recognition method based on variational autoencoder and adversarial generative network according to claim 2 is characterized in that: The S302 is specifically as follows: During training, the input of the voiceprint recognition network includes the first grammatical data X1 and the second grammatical data X2 as well as the same and different class labels. After passing through the voiceprint feature extraction network, the output includes the features (Z0, Z0', Z1, Ds)1 and the features (Z0, Z0', Z1, Ds)2 and predicted same and different class labels; When applied, the input of the voiceprint recognition network includes the first piece of spectrogram data X1 and the second piece of spectrogram data X2, and the output is the predicted same and different class labels.

4. The voiceprint recognition method based on variational autoencoder and adversarial generative network according to claim 3 is characterized in that: The S4 is specifically: S401, initializing the voiceprint recognition network; S402, inputting the triplet (X1, X2, y) into the voiceprint recognition network; S403. Let L dis = -y0 log Ds - (1 - y0) log(1 - Ds); Definition: L1 = L dis (1) +L dis (2) is the first loss function, where L dis (1) Indicates the value obtained by processing the first data in the same pair of data, L dis (2) The value obtained by processing the second data in the same paired data is represented by using the mini-batch gradient descent method to update the weights of the third convolutional neural network in the real data and generated data discriminator based on the first loss function; S404. Order: Where: n is the one-dimensional dimension of the spectrogram data X after flattening; X i represents the i-th number after the spectrogram data X is flattened, Spectrogram data representing the output of the voiceprint feature extraction network The i-th value of ; Get the RMSE of the first paired data, recorded as RMSE (1) , the RMSE of the second data is recorded as RMSE (2) ; make: Where D is the bi-norm of the first spectrogram output feature and the second spectrogram output feature of the paired data; m0 is the marginal value, m0 is greater than or equal to 1; y is the label when the paired data is input during training; The second loss function is defined as: L2=L d +β·(RMSE (1) +RMSE (2) )-γ·L1; Where: L d is the Euclidean space distance loss between two sample points in the latent space, and β and γ are adjusted according to the experimental results; Based on the second loss function, use the mini-batch gradient descent method to update the network weights other than the real data and generated data discriminator range; S405, replace the previous network with the network after updating the network weights, define the total number of iterations, re-execute S402-S404, and each execution is defined as an iteration; after the iteration is completed, the trained voiceprint recognition network is obtained.

5. A voiceprint recognition system based on variational autoencoder and adversarial generative network, characterized in that: include: Preprocessing module: collect M voice files of each speaker, preprocess each voice file to obtain a spectrogram; M is a natural number greater than 1; Paired data set construction module: construct a paired data set of the spectrogram; the paired data set includes N triples (X1, X2, y); wherein: X1 is the first piece of spectrogram data of the triple (X1, X2, y), X2 is the second piece of spectrogram data of the triple (X1, X2, y), label y is defined as 0 for the same pairing, and label y is defined as 1 for the heterogeneous pairing; each piece of data in the same pairing set and each piece of data in the heterogeneous pairing set are composed of two spectrograms; the two spectrograms of each piece of data in the same pairing set are spectrograms of the same speaker; the two spectrograms of each piece of data in the heterogeneous pairing set are spectrograms of different speakers; Feature extraction and voiceprint recognition network construction module: Construct a voiceprint feature extraction network to extract the voiceprint features of the spectrogram; specifically, extract the implicit features of the spectrogram by constructing the conditional and adversarial generation model of the variational autoencoder; construct a voiceprint recognition network; the specific process includes: Construct an encoder, wherein the encoder takes the spectrogram data X obtained by processing the original data with the spectrogram data as input, and outputs a hidden feature vector Z0 through a first convolutional neural network; Construct a sampling layer, which takes the implicit feature vector Z0 as input and generates m pairs of outputs through a multi-layer perceptron. For each pair of outputs, define the previous parameter as the mean μ i , the latter parameter is the variance σ i , sample a value t0 from the standard normal distribution, using the formula: i =t o ·σ i +μ i , where: i is a natural number from 1 to m; get the output and form the sampled latent feature vector Z1 in the original order; Construct a decoder, which takes the sampled implicit feature vector Z1 as input and obtains a generated spectrogram data with a dimension consistent with the spectrogram data X through a second convolutional neural network. Generate spectrogram data as described As input, a numerical vector Z0' different from the latent feature vector Z0 is obtained through the second convolutional neural network; Construct a real data and generated data discriminator, which will ( Z0') is marked as y0 = 1, (X, Z0) is marked as y0 = 0, and ( Z0'), (X, Z0) and the mark are taken as input, and pass through the third convolutional neural network and the fully connected layer in sequence, and the output is a scalar Ds; Combine and construct a voiceprint feature extraction network; The above encoder, decoder, real data and generated data discriminator are connected in sequence to form a voiceprint feature extraction network; the input of the voiceprint feature extraction network is the spectrogram data X, and the output is (Z0, Z0', Z1, Ds); Training module: training the voiceprint recognition network; Judgment module: import the first spectral data X1 and the second spectral data X2 into the voiceprint recognition network respectively to obtain two scalars Ds; obtain the binary norm D of the two scalars Ds, and obtain the judgment result through the value of the binary norm D.

6. The voiceprint recognition system based on variational autoencoder and adversarial generative network according to claim 5 is characterized in that: The construction process of the paired data set construction module is as follows: First, two spectrograms are extracted from the spectrogram of the same speaker and paired to obtain a similar pair, and the label y of the similar pair is defined as 0; one spectrogram is extracted from each of the spectrograms of two different speakers and paired to obtain a heterogeneous pair, and the label y of the heterogeneous pair is defined as 1; Then, according to the total number of pairs N, the number of speakers is defined as the number of classes k, the same-class and different-class sampling ratio is α, and according to the limiting condition: s1+s2=N / k, the number of same-class pairs s1 and different-class pairs s2 to be extracted for each class are calculated: Finally, the same type of pairs, different type of pairs and label y are combined into a data set, and the data set includes N triplets (X1, X2, y).

7. The voiceprint recognition system based on variational autoencoder and adversarial generative network according to claim 6 is characterized in that: The construction of the voiceprint recognition network specifically includes: During training, the input of the voiceprint recognition network includes the first spectral data X1 and the second spectral data X2 as well as the same and different class labels. After passing through the voiceprint feature extraction network, the output includes the features (Z0, Z0', Z1, Ds)1 and the features (Z0, Z0', Z1, Ds)2 and predicted same and different class labels; When applied, the input of the voiceprint recognition network includes the first spectral data X1 and the second spectral data X2, and the output is the predicted same and different class labels.

8. The voiceprint recognition system based on variational autoencoder and adversarial generative network according to claim 7 is characterized in that: The training process of the training module is as follows: S401, initializing the voiceprint recognition network; S402, importing the triplet (X1, X2, y) into the voiceprint recognition network; S403. Let L dis = -y0 log Ds - (1 - y0) log(1 - Ds); Definition: L1 = L dis (1) +L dis (2) is the first loss function, where L dis (1) Indicates the value obtained by processing the first data in the same pair of data, L dis (2) The value obtained by processing the second data in the same paired data is represented by using the mini-batch gradient descent method to update the weights of the third convolutional neural network in the real data and generated data discriminator based on the first loss function; S404. Order: Where: n is the one-dimensional dimension of the spectrogram data X after flattening; X i Represents the spectrogram data X after flattening Spectrogram data representing the output of the voiceprint feature extraction network The i-th value of The RMSE of the first paired data is obtained, recorded as RMSE(1), and the RMSE of the second paired data is recorded as RMSE(2); make: Where D is the bi-norm of the first spectrogram output feature and the second spectrogram output feature of the paired data; m0 is the marginal value, m0 is greater than or equal to 1; y is the label when the paired data is input during training; The second loss function is defined as: L2=L d +β·(RMSE (1) +RMSE (2) )-γ·L1; Where: L d is the Euclidean space distance loss between two sample points in the latent space, and β and γ are adjusted according to the experimental results; Based on the second loss function, use the mini-batch gradient descent method to update the network weights other than the real data and generated data discriminator range; S405, replace the previous network with the network after updating the network weights, define the total number of iterations, re-execute S402-S404, and each execution is defined as an iteration; after the iteration is completed, the trained voiceprint recognition network is obtained.

Images