Adversarial network optimization method and system for short-utterance speaker verification

Abstract

The embodiment of the specification provides a generative adversarial network optimization method and system for short speech speaker verification, wherein the method comprises the following steps: acquiring a plurality of pairs of long and short speech acoustic feature samples; inputting the short speech acoustic feature sample into a generator for splicing to obtain a generated pseudo long speech acoustic feature sample; inputting the pseudo long speech acoustic feature sample and the acquired long speech acoustic feature sample into a speaker verification model respectively, outputting a pseudo identity feature sample and a true identity feature sample through the speaker verification model; inputting the true identity feature sample and the pseudo identity feature sample into a discriminator and a classifier, calculating the loss of the discriminator and the classifier through a loss function, and updating the parameters of the discriminator, the classifier and the generator through back propagation optimization. The problem that the discrimination effect of a speaker verification system becomes poor as the speech duration becomes shorter is solved.

Description

TECHNICAL FIELD

[0001] The present document relates to the technical field of speech processing, and in particular to a short speech speaker verification optimization method and system based on a generative adversarial network. BACKGROUND

[0002] Voiceprint recognition is a task of determining the identity of a speaker through speech, and speaker verification is a branch of voiceprint recognition research. Speaker verification models can be divided into end-to-end models and staged models, and the most widely used and researched model today is the staged model. The staged speaker verification is divided into two stages of front-end and back-end, the front-end is used for speaker identity feature extraction, and the speaker voice data is generated into a vector, and the back-end is used for similarity calculation on the generated pair of vectors to determine whether the two speeches belong to the same speaker.

[0003] The generative adversarial network generally includes a generator and a discriminator, and the two are alternately trained, the goal of the generator is to generate good data as much as possible to deceive the discriminator, and the discriminator can improve its discrimination ability through the generated data, and the two balance and promote each other, and finally obtain a better generator.

[0004] Experiments show that the performance of the speaker verification system decreases with the decrease of the speech length, and the relatively optimal effect obtained at the present stage is based on the longest speech test, but it is difficult to obtain enough long speech data in reality, so the effect of the system in actual use is much worse than expected, therefore, it is necessary to optimize the performance of the speaker verification system under the condition of short speech. SUMMARY

[0005] One or more embodiments of the present specification provide a generative adversarial network optimization method for short speech speaker verification, comprising:

[0006] S1. Obtain a plurality of pairs of long and short speech acoustic feature samples;

[0007] S2. input the short speech acoustic feature sample into the generator for splicing to obtain a generated pseudo long speech acoustic feature sample;

[0008] S3. input the pseudo long speech acoustic feature sample and the obtained long speech acoustic feature sample into the speaker verification model respectively, and output pseudo identity feature samples and true identity feature samples through the speaker verification model;

[0009] S4. input the true identity feature sample and the pseudo identity feature sample into the discriminator and the classifier, calculate the loss of the discriminator and the classifier through the loss function, and update the parameters of the discriminator, the classifier and the generator through back propagation optimization.

[0010] The one or more embodiments of the specification provide a generative adversarial network optimization system for short speech speaker verification, comprising:

[0011] A data acquisition module is configured to acquire a plurality of pairs of long and short speech acoustic feature samples.

[0012] A data processing module is configured to input the short speech acoustic feature sample into a generator for splicing to obtain a generated pseudo long speech acoustic feature sample.

[0013] A sample generation module is configured to input the pseudo long speech acoustic feature sample and the acquired long speech acoustic feature sample into a speaker verification model respectively, and output a pseudo identity feature sample and a true identity feature sample through the speaker verification model.

[0014] A training optimization module is configured to input the true identity feature sample and the pseudo identity feature sample into a discriminator and a classifier, calculate the loss of the discriminator and the classifier through a loss function, and update the parameters of the discriminator, the classifier and the generator through back propagation optimization.

[0015] Compared with the prior art, the beneficial effects of the present application are: using the speech feature distribution learned by the generative model to generate long speech features from short speech features, improving the performance of the system under short time conditions and the discrimination effect of the system; better generalization, applicable to multiple scenarios; applicable to multiple speaker verification systems, simple to train. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the one or more embodiments of the specification or the prior art, the drawings needed to be used in the embodiment or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the specification, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0017] Figure 1 The flowchart of the generative adversarial network optimization method for short speech speaker verification of the one or more embodiments of the specification;

[0018] Figure 2 The generative adversarial network architecture diagram for short speech speaker verification of the one or more embodiments of the specification;

[0019] Figure 3 The schematic diagram of the generative adversarial network optimization system for short speech speaker verification of the one or more embodiments of the specification. DETAILED DESCRIPTION

[0020] In order for those skilled in the art to better understand the technical solutions in the one or more embodiments of the present specification, the technical solutions in the one or more embodiments of the present specification will be clearly and completely described in the following with reference to the drawings in the one or more embodiments of the present specification. Obviously, the described embodiments are only a part of the embodiments of the present specification, rather than all the embodiments. Based on the one or more embodiments of the present specification, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present document.

[0021] Method embodiments

[0022] According to the embodiments of the present application, a generative adversarial network optimization method for short speech speaker verification is provided, Figure 1 is a flowchart of the generative adversarial network optimization method for short speech speaker verification of the one or more embodiments of the present specification, as Figure 1 shown, the generative adversarial network optimization method for short speech speaker verification according to the embodiments of the present application specifically includes:

[0023] S1. Obtain a plurality of pairs of long and short speech acoustic feature samples;

[0024] S2. Input the short speech acoustic feature sample into the generator for splicing to obtain a generated pseudo long speech acoustic feature sample;

[0025] S3. Input the pseudo long speech acoustic feature sample and the obtained long speech acoustic feature sample into the speaker verification model respectively, and output the pseudo identity feature sample and the true identity feature sample through the speaker verification model;

[0026] S4. Input the true identity feature sample and the pseudo identity feature sample into the discriminator and the classifier, calculate the loss of the discriminator and the classifier through the loss function, and update the parameters of the discriminator, the classifier and the generator through back propagation optimization.

[0027] Wherein, the obtained long and short speech acoustic features are obtained by linear transformation of original long and short speech, in the embodiments of the present application, the short speech refers to the speech less than 4 seconds, and the long speech refers to the speech greater than or equal to 4 seconds.

[0028] Figure 2 For the generative adversarial network architecture of the one or more embodiments of the present specification for short speech speaker verification, as Figure 2 shown, the generative adversarial network architecture includes a generator, a speaker verification model, a discriminator and a classifier. When training and optimizing, input the short speech acoustic feature sample into the generator for splicing to obtain a generated pseudo long speech acoustic feature sample, specifically:

[0029] S21. inputting the short speech acoustic feature sample into a generator to generate additional acoustic feature samples through linear transformation;

[0030] wherein the network structure of the generator is shown in Table 1:

[0031] Table 1

[0032]

[0033] wherein in_channel represents the dimension of the speech acoustic feature used by the speaker verification model, k is the size of the convolution kernel, d is the size of the convolution kernel expansion, s is the convolution step, and the activation function uses ReLU. The generator is divided into two parts by the Reshape layer, the first part is fixed and does not change, and 5 one-dimensional convolution (Conv1d) is used to extract deeper features, and then the mean and variance are calculated through the pooling layer (Pool) and spliced, and then sent to two fully connected layers (FC). The second part is determined according to the length of the additional features to be generated, and the following table shows the configuration for generating 200 additional frames. After 3 one-dimensional deconvolution (Deconv1d) operations, 200 frames (about 2 seconds) of additional acoustic features are finally output.

[0034] S22. splicing the short speech acoustic feature sample and the additional acoustic feature sample to generate a pseudo long speech acoustic feature sample;

[0035] S23. performing cepstral mean normalization processing on the pseudo long speech acoustic feature sample.

[0036] wherein the length of the additional acoustic feature sample is equal to the length of the short speech acoustic feature sample.

[0037] The pseudo long speech acoustic feature after cepstral mean normalization processing and the long speech acoustic feature corresponding to the short speech for generating the pseudo long speech acoustic feature obtained in step S1 are input into the speaker verification model trained in advance to obtain speaker identity features, i.e., pseudo identity features and true identity features corresponding to the pseudo long speech acoustic feature and the long speech acoustic feature, respectively.

[0038] wherein the speaker verification model used includes but is not limited to Resnet-based x-vector, E-TDNN, ECAPA-TDNN, etc., the dimension of the speech acoustic feature used needs to be consistent with that used by the trained speaker verification model, the trained speaker verification model is parameter frozen during the training and optimization of the adversarial generation network, and if linear discriminant analysis is used for dimension reduction, the speaker identity features output during the training of the adversarial generation network also need to be dimension reduced.

[0039] After the trained speaker verification model extracts the pseudo identity features and true identity features corresponding to the pseudo long speech acoustic features and long speech acoustic features, the pseudo identity features and true identity features are sent to the discriminator and the classifier for repeated training and optimization, and the specific process is as follows:

[0040] S41. The true identity feature sample and the pseudo identity feature sample are input into the discriminator and the classifier, the loss of the discriminator and the classifier is calculated through the loss function, and the parameters of the discriminator and the classifier are updated through back propagation;

[0041] S42. The pseudo identity feature sample is sent to the discriminator and the classifier, and when the pseudo identity feature sample is sent to the discriminator, the label of the pseudo identity feature sample is forged as true, the loss of the discriminator and the classifier is calculated through the loss function, and the parameters of the generator are updated through back propagation.

[0042] The loss function adopts a cross-entropy loss function, adopts an RMSprop optimizer, and the learning rate is set to 0.0001.

[0043] The structure of the discriminator is shown in Table 2:

[0044] Table 2

[0045]

[0046] The structure of the classifier is shown in Table 3:

[0047] Table 3

[0048]

[0049] Wherein, embedding_dim represents the dimension of the identity feature, speakers represents the total number of speakers in the training set, and the discriminator and the classifier share all layers except the last layer during training.

[0050] In this embodiment, the test is performed once every 40 iterations during training, and if the discrimination accuracy of the speaker verification model is greater than 0.75, the iterative training of the generative adversarial network can be ended.

[0051] When the training is completed and put into use, the discriminator and the classifier in the network are discarded, the generator is retained, the acoustic features of the short speech to be identified are input into the generator, the short speech acoustic features and the generator output features are spliced and processed, and then input into the speaker verification model, and the speaker identity features are output. The identity features are sent to the back end for discrimination.

[0052] By using the speech feature distribution learned by the generator, a long speech capable of containing more speaker identity information is generated from the short speech, the performance of the system under the short-time condition is improved, the discrimination effect of the system is improved, the generalization is better, the additional speech acoustic features generated by the generator in the embodiment of the application are essentially speech data subjected to linear transformation, and therefore can be applied to various scenes, the additional speech feature distribution generated in the embodiment of the application is close to the speech acoustic feature distribution of the actual speaker, the interpretability is stronger, and the embodiment of the application is easy to understand, the training is simple, and the embodiment of the application can be applied to various speaker verification systems.

[0053] System embodiment

[0054] According to the embodiment of the application, a generative adversarial network optimization system for short speech speaker verification is provided, Figure 3 is a schematic diagram of the generative adversarial network optimization system for short speech speaker verification of one or more embodiments of the present specification, as Figure 3 shown, the generative adversarial network optimization system for short speech speaker verification according to the embodiment of the application specifically comprises:

[0055] The data acquisition module 30 is configured to acquire a plurality of pairs of long and short speech acoustic feature samples.

[0056] The data processing module 32 is configured to input the short speech acoustic feature sample into the generator for splicing to obtain a generated pseudo long speech acoustic feature sample.

[0057] The data processing module 32 is specifically configured to:

[0058] input the short speech acoustic feature sample into the generator to generate additional acoustic feature samples through linear transformation;

[0059] splice the short speech acoustic feature sample and the additional acoustic feature sample to generate a pseudo long speech acoustic feature sample;

[0060] perform cepstrum mean normalization processing on the pseudo long speech acoustic feature sample.

[0061] The sample generation module 34 is configured to input the pseudo long speech acoustic feature sample and the acquired long speech acoustic feature sample into a speaker verification model respectively, and output pseudo identity feature samples and true identity feature samples through the speaker verification model.

[0062] The speaker verification model is a trained speaker verification model, including a Resnet-based x-vector model, an E-TDNN model and an ECAPA-TDNN model.

[0063] The training optimization module 36 is configured to input the real identity feature samples and the fake identity feature samples into the discriminator and the classifier, calculate the loss of the discriminator and the classifier through a loss function, and update the parameters of the discriminator, the classifier and the generator through back propagation optimization.

[0064] The training optimization module 36 is configured to:

[0065] The training optimization module 36 is configured to input the real identity feature samples and the fake identity feature samples into the discriminator and the classifier, calculate the loss of the discriminator and the classifier through a loss function, and update the parameters of the discriminator and the classifier through back propagation.

[0066] The training optimization module 36 is configured to input the real identity feature samples and the fake identity feature samples into the discriminator and the classifier, calculate the loss of the discriminator and the classifier through a loss function, and update the parameters of the discriminator and the classifier through back propagation.

[0067] The loss function is a cross-entropy loss function, and the RMSprop optimizer is used, with a learning rate of 0.0001.

[0068] The embodiment performs a test after every 40 iterations of training, and if the correct discrimination rate of the speaker verification model is greater than 0.75, the iteration training of the generative adversarial network can be ended.

[0069] The embodiment of the present application is a system embodiment corresponding to the above-mentioned method embodiment, and the specific operations of each module can be understood with reference to the description of the method embodiment, which will not be repeated here.

[0070] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the above embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A generative adversarial network optimization method for short-utterance speaker verification, characterized in that, The method comprises: S1. obtaining a plurality of pairs of long and short speech acoustic feature samples; S2. inputting the short speech acoustic feature samples into a generator for splicing to obtain generated pseudo long speech acoustic feature samples; Step S2 specifically comprises: inputting the short speech acoustic feature samples into the generator to generate additional acoustic feature samples through linear transformation; Specifically, the network structure of the generator comprises a network layer, an activation layer and an output layer, the activation function is ReLU, and the generator is divided into two parts by a Reshape layer, the first part is fixed and unchanged, 5 one-dimensional convolutions are used to extract deeper features, then the mean and variance are calculated through a pooling layer and spliced, and then sent to two fully connected layers; the second part is determined according to the length of the additional features to be generated, and finally a certain frame of additional acoustic features is output through three one-dimensional deconvolution operations; The short speech acoustic feature samples and the additional acoustic feature samples are spliced to generate pseudo long speech acoustic feature samples; The pseudo long speech acoustic feature samples are subjected to cepstrum mean normalization processing; S3. inputting the pseudo long speech acoustic feature samples and the obtained long speech acoustic feature samples into a speaker verification model respectively, and outputting pseudo identity feature samples and true identity feature samples through the speaker verification model; S4. inputting the true identity feature samples and the pseudo identity feature samples into a discriminator and a classifier, calculating the loss of the discriminator and the classifier through a loss function, and updating the parameters of the discriminator, the classifier and the generator through back propagation optimization.

2. The method of claim 1, wherein, The method of step S4 specifically comprises: inputting the true identity feature samples and the pseudo identity feature samples into the discriminator and the classifier, calculating the loss of the discriminator and the classifier through a loss function, and updating the parameters of the discriminator and the classifier through back propagation; The pseudo identity feature samples are sent into the discriminator and the classifier, the pseudo identity feature sample label is forged as true when sent into the discriminator, the loss of the discriminator and the classifier is calculated through a loss function, and the parameters of the generator are updated through back propagation.

3. The method of claim 1, wherein, The speaker verification model is a trained speaker verification model, comprising a Resnet-based x-vector model, an E-TDNN model and an ECAPA-TDNN model.

4. The method of claim 1, wherein: The length of the additional acoustic feature samples is equal to the length of the short speech acoustic feature samples; The short speech of the short speech acoustic feature samples is a speech with a length of 1-3 seconds.

5. The method of claim 1, wherein, The loss function adopts a cross-entropy loss function, the optimizer adopts an RMSprop optimizer, and the learning rate is set to 0.0001.

6. A generative adversarial network optimization system for short-utterance speaker verification, the system comprising: The method comprises: a data acquisition module for acquiring a plurality of pairs of long and short speech acoustic feature samples; a data processing module for inputting the short speech acoustic feature samples into a generator for splicing to obtain generated pseudo long speech acoustic feature samples; The data processing module is specifically configured to input the short speech acoustic feature samples into the generator to generate additional acoustic feature samples through linear transformation; Specifically, the network structure of the generator includes a network layer, an activation layer and an output layer, the activation function uses ReLU, the generator is divided into two parts by a Reshape layer, the first part is fixed and unchanged, more deep features are extracted through five one-dimensional convolutions, then the mean and variance are calculated through a pooling layer and spliced, and then sent to two fully connected layers; the second part is determined according to the length of the additional feature to be generated, and finally outputs a certain frame of additional acoustic features through three one-dimensional deconvolution operations; The short speech acoustic feature sample and the additional acoustic feature sample are spliced to generate a pseudo long speech acoustic feature sample; The pseudo long speech acoustic feature sample is subjected to cepstrum mean normalization processing; The sample generation module is configured to input the pseudo long speech acoustic feature sample and the obtained long speech acoustic feature sample into a speaker verification model respectively, and output a pseudo identity feature sample and a true identity feature sample through the speaker verification model; The training optimization module is configured to input the true identity feature sample and the pseudo identity feature sample into a discriminator and a classifier, calculate the loss of the discriminator and the classifier through a loss function, and update the parameters of the discriminator, the classifier and the generator through back propagation optimization.

7. The system of claim 6, wherein, The data processing module is specifically configured to: input the short speech acoustic feature sample into the generator to generate an additional acoustic feature sample through linear transformation; splice the short speech acoustic feature sample and the additional acoustic feature sample to generate a pseudo long speech acoustic feature sample; perform cepstrum mean normalization processing on the pseudo long speech acoustic feature sample.

8. The system of claim 6, wherein, The training optimization module is specifically configured to: input the true identity feature sample and the pseudo identity feature sample into the discriminator and the classifier, calculate the loss of the discriminator and the classifier through a loss function, and update the parameters of the discriminator and the classifier through back propagation; input the pseudo identity feature sample into the discriminator and the classifier, fake the label of the pseudo identity feature sample as true when inputting into the discriminator, calculate the loss of the discriminator and the classifier through a loss function, and update the parameters of the generator through back propagation.

9. The system of claim 6, wherein, The speaker verification model is a trained speaker verification model, including a Resnet-based x-vector model, an E-TDNN model and an ECAPA-TDNN model.

Images