Speech conversion method based on bidirectional loss function of variational auto-encoding network

Abstract

The application discloses a speech conversion method based on a bidirectional loss function of a variational auto-encoding network, and the process is as follows: selecting and dividing speech data; extracting acoustic features from each speech sample; constructing a text encoder, a speaker encoder and a decoder; inputting speech into the text encoder and the speaker encoder, respectively extracting text information and speaker information, and then obtaining reconstructed speech through the decoder; constructing a bidirectional loss function and training the network, so that the speaker pronunciation characteristic information of different speech samples of the same speaker is consistent and the divergence value of the extracted text information is as small as possible; and using a speech test set to evaluate the speech conversion performance of the variational auto-encoding network. The application separates the speaker pronunciation characteristic information and the text information by constructing the text encoder and the speaker encoder, and realizes one-time speech conversion. The bidirectional loss function is used to train the variational auto-encoding network, so that the speech conversion performance is improved without increasing the algorithm complexity.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech signal processing, and particularly relates to a speech conversion method based on a bidirectional loss function of a variational auto-encoding network. BACKGROUND

[0002] Like fingerprints, irises and other biological characteristics, the speaking characteristics of a person are also unique. Voice conversion aims to convert the speech of a source speaker into speech with the pronunciation characteristics of a target speaker, and the converted speech content is consistent with the speech content of the source speaker. Voice conversion technology has a wide range of application scenarios, such as personalized voice navigation, personalized voice broadcast of intelligent voice terminals, personalized conversion of various media voices, speaker voice disguise in military security and other fields.

[0003] Voice conversion has become an important branch of the field of intelligent speech. Current voice conversion methods mainly include methods based on Gaussian mixture models and methods based on deep neural networks. These methods can only achieve high-quality voice conversion when the size of the speech data is large enough. Therefore, their practicality is not high. In addition, these methods also have the following disadvantages: the pronunciation characteristic information of the source speaker is not effectively separated from the text information, the training data set is large, and the training is time-consuming. SUMMARY

[0004] The purpose of the present application is to solve the above-mentioned defects in the prior art, and to provide a voice conversion method based on a bidirectional loss function of a variational auto-encoding network.

[0005] The purpose of the present application can be achieved by adopting the following technical solutions:

[0006] A voice conversion method based on a bidirectional loss function of a variational auto-encoding network, the voice conversion method comprising the following steps:

[0007] S1, dividing a speech data set into a training set and a test set;

[0008] S2, extracting a mel-scale spectrogram from each speech sample of the speech data set and performing normalization processing;

[0009] S3, designing a variational auto-encoding network, the variational auto-encoding network comprising a text encoder, a speaker encoder and a decoder, wherein the text encoder and the speaker encoder are connected in parallel and then sequentially connected to the decoder.

[0010] S4, constructing a reconstruction loss function, a KL divergence loss function, a same speaker information loss function and a different speaker information loss function to obtain a total loss function, and iteratively training the variational auto-encoding network multiple times under the guidance of the total loss function;

[0011] S5, randomly select a test sample from the test set, input the mel-scale spectrum feature of the test sample into the trained variational auto-encoding network to obtain the reconstructed speech after voice conversion of the test sample.

[0012] Further, the step S1 is as follows:

[0013] S1.1, selecting a voice data set: selecting CSTR VTCK, LibriSpeech and / or TED-LIUM as the voice data set;

[0014] S1.2, dividing the voice data set: randomly selecting the voice of several speakers as the test set, and the voice of the remaining speakers as the training set, wherein the proportion of the training set is 90%, and the proportion of the test set is 10%.

[0015] Further, the step S2 is as follows:

[0016] S2.1, performing endpoint detection and amplitude normalization: performing endpoint detection on the voice samples of the training set, removing silence, and then performing amplitude normalization on the voice samples;

[0017] S2.2, framing: dividing the voice samples into short-time voice frames of fixed length, usually 10ms-40ms;

[0018] S2.3, windowing: multiplying each frame of voice with a window function ω(n) to obtain a windowed voice frame, and selecting a Hamming window as the window function:

[0019]

[0020] wherein L represents the frame length, and n represents the sampling frequency point;

[0021] S2.4, short-time Fourier transform: performing short-time Fourier transform on the selected voice sample to obtain a frequency spectrum, and the short-time Fourier transform formula is as follows:

[0022] wherein x(lH-n) is a voice sample, l is the serial number of the current filter, represents the current time period, k is the serial number of the current frequency, represents that the kth frequency is being processed, n is a variable, and the value range of n is 0≤n≤N-1, ω(n) is a window function, H is a sampling interval, and N represents sampling at N equally spaced frequencies;

[0023] S2.5, normalizing the spectrum graph: setting the frequency size to a uniform scale, and then using an approximate inverse linear transformation to restore the linear scale spectrum graph, and converting the mel-scale spectrum graph into a voice waveform.

[0024] ​Further, the step S3 comprises:

[0025] S3.1, constructing a speaker encoder, referred to as E S , the process is as follows:

[0026] S3.1.1, constructing a ConBank: the input speech waveform is convolved, the convolution kernel size, input dimension, output dimension, sliding step of the convolution layer are 3x3, 128, 128, 1 respectively, denoted as Conv1, then the convolution output result is padded and expanded, and output after splicing;

[0027] S3.1.2, the ConBank output result is convolved, then passed through a ReLu function, and then convolved and passed through a ReLu function, wherein the convolution kernel size, input dimension, output dimension, sliding step of the convolution layer are 3x3, 128, 128, 2 respectively, denoted as Conv2, and the expression of the ReLu function is:

[0028] Where x in is the input of the ReLu function;

[0029] S3.1.3, the output result of the previous step is input to an average pooling layer, referred to as AvgPool, which is a two-dimensional average pooling.

[0030] S3.1.4, the output result of the previous step is input to a fully connected layer, denoted as Dense, then passed through a ReLu function to obtain the output of E S , usually to prevent information overfitting, the output of the previous step can also be added to obtain the final output, i.e. speaker pronunciation characteristic information, denoted as z s , wherein the number of nodes of Dense is 128.

[0031] S3.2, constructing a text encoder, referred to as E c , the process is as follows:

[0032] S3.2.1, inputting the input speech waveform into ConBank;

[0033] S3.2.2, inputting the output result of the previous step into Conv1, then passing through a ReLu function, then inputting into a normalization layer, denoted as IN, inputting the output of the normalization layer into Conv2, then passing through a ReLu function, and finally inputting into IN again to obtain the final output of E c , i.e. text information, denoted as z c , wherein IN is to independently normalize each channel frequency along the channel direction, and the calculation of each channel is as follows:

[0034] where a and b are normalized weight parameters, a and b are non-zero constants, W is the data matrix on each channel in the normalization layer, E(W) is the expectation of W, Var(W) is the variance of W, and Y is the normalized value;

[0035] S3.3, construct the decoder, referred to as D, the process is as follows:

[0036] S3.3.1, z c is input to Conv1 first, then through the ReLu function, input to the adaptive normalization layer, referred to as AdaIN, at the same time, z s is input to Dense and ReLu function, and then input to AdaIN through the affine layer, wherein the affine layer is referred to as Affine, and the expression is:

[0037] Affine(x b ) = f(wx b +c)

[0038] where x b is the input of the previous layer, w is the weight parameter, c is the bias vector, w and c are constant values, f(wx b +c) represents a nonlinear activation function about x b , which takes ReLu, Sigmoid or Tanh function;

[0039] For AdaIN, the expression is: where μ(z c ) is the mean of the text information frequency, σ(z c ) is the variance of the text information frequency, μ(z s ) is the mean of the speaker information frequency, and σ(z s ) is the variance of the speaker information frequency;

[0040] S3.3.2, the output of the AdaIN in the previous step is input to Conv1 and ReLu function again, and then input to AdaIN again through upsampling to obtain the output of D. In order to prevent information overfitting, the output of the AdaIN in the previous step is directly upsampled and added to the output of D to obtain the final output, i.e. the reconstructed speech.

[0041] Further, the step S4 comprises:

[0042] S4.1, construct the loss function: let x i, i = 1, 2, 3 are speech samples in the training speech data set, assuming that x1 and x2 are speech samples of the same speaker, but the speech texts are different, and x3 is a speech sample of another speaker, wherein the speech texts of x1 and x3 are the same;

[0043] x1 is input into E c and E s respectively to obtain speaker pronunciation characteristic information z s1 and text information z c1 ;

[0044] x2 is input into E c and E s respectively to obtain speaker pronunciation characteristic information z s2 and text information z c2 ;

[0045] x3 is input into E c and E s respectively to obtain speaker pronunciation characteristic information z s3 and text information z c3 ;

[0046] z s1 and z c2 are input into D to obtain reconstructed speech x2';

[0047] z s2 and z c1 are input into D to obtain reconstructed speech x1';

[0048] z s3 and z c3 are input into D to obtain reconstructed speech x3';

[0049] S4.1.1, construct a speech reconstruction loss function: let p(z c |x i ) be a Gaussian distribution subject to conditional independence, expressed as: p(z c |x i ) = G(z c (x i ), I)

[0050] wherein z c (x i ) represents the mean of text information x i , I represents the unit variance of text information x i , G(z c (x i ), I) represents a Gaussian distribution with mean z c (x i ) and unit variance I, and the expression of the probability density function f(t) is:

[0051] Thus G(z c (x i ),I) is expressed as:

[0052] Wherein, t is a random variable of Gaussian distribution probability density function;

[0053] Since p(z c |x i ) obeys conditional independent Gaussian distribution, the speech reconstruction loss function L rec is defined as: L rec =||x1'-x1||1+||x2'-x2||1+||x3'-x3||1

[0054] Wherein, ||x i '-x i ||1, i = 1, 2, 3 represents the 1-norm of the difference between x i ' and x i ;

[0055] S4.1.2, construct KL divergence loss function:

[0056] In order to make the posterior distribution p(z c |x i ) match the Gaussian distribution G(0,I), it is necessary to make its KL divergence loss function L kl minimum, the KL divergence loss function L kl is defined as:

[0057] L kl =||z c (x i ) 2 ||2

[0058] Wherein, z c (x i ) 2 represents the square value of the frequency of text information x i , ||z c (x i ) 2 ||2 represents the 2-norm of z c (x i ) 2 ;

[0059] S4.1.3, construct bidirectional loss function:

[0060] Based on the idea of contrastive learning, a bidirectional loss function is constructed, that is, maximizing the speaker information similarity between different speeches of the same speaker and minimizing the speaker information similarity of different speakers, so that two or more structure same variational auto-encoding networks can be input with different data to obtain different outputs, and all outputs are trained together, so that two loss functions can be reconstructed: the same speaker loss function L spk and the different speaker loss function L diff , which are defined as follows, respectively:

[0061] L spk =||z s2 -z s1 ||1

[0062] L diff =-[||z s2 -z s3 ||1+||z s1 -z s3 ||1]

[0063] S4.1.4, the combined loss function:

[0064] The above several loss functions are combined to construct the total loss function L, and the total loss function L is defined as:

[0065]

[0066] Wherein alpha, gamma, are the weight coefficients of the KL divergence loss function L kl , the same speaker loss function L spk and the different speaker loss function L diff , and the value range of these coefficients is usually 0-1.

[0067] S4.2, training the variational auto-encoding network: using the Adam optimizer to optimize the network, and training until the total loss function L converges.

[0068] Further, the step S5 process is as follows:

[0069] The mel-scale spectrum features of the speech samples of the test data set are extracted and input into the above trained network to obtain the reconstructed speech after conversion, the similarity between the reconstructed speech and the original speech is calculated to obtain the objective evaluation result of the speech conversion, such as the word error rate WER, the sentence error rate SER and the character error rate CER. At the same time, the two kinds of speeches are listened to by the human ear to obtain the subjective evaluation result of the speech conversion, such as the speech similarity and the speech quality.

[0070] The present application has the following advantages and effects relative to the prior art:

[0071] (1) The method of the present application can separate the speaker pronunciation characteristic information and the text information in the speech, eliminate the aliasing of the above two kinds of information, and realize one-time speech conversion, i.e. without the speech of the target speaker in the training set, only a target speaker speech sample needs to be input in the test stage to complete the speech conversion, thereby reducing the cost of preparing speech data and the dependence of the model on the speech data.

[0072] (2) The method of the present application can improve the speech conversion performance without increasing the algorithm complexity by constructing a bidirectional loss function to train the network. BRIEF DESCRIPTION OF DRAWINGS

[0073] The accompanying drawings, which are included to provide a further understanding of the present application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0074] Figure 1 is a flow chart of a speech conversion method based on a bidirectional loss function of a variational auto-encoding network according to an embodiment of the present application;

[0075] Figure 2 is a specific step diagram of the speech conversion method according to an embodiment of the present application;

[0076] Figure 3 is a structure diagram of the speaker pronunciation characteristic information according to an embodiment of the present application;

[0077] Figure 4 is a structure diagram of the text information encoder according to an embodiment of the present application;

[0078] Figure 5 is a structure diagram of the decoder according to an embodiment of the present application. DETAILED DESCRIPTION

[0079] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0080] Embodiment 1

[0081] The embodiment discloses a speech conversion method based on a bidirectional loss function of a variational auto-encoding network.

[0082] The flowchart of the speech conversion method based on the bidirectional loss function of the variational auto-encoding network is shown in Figure 1 The flowchart of the speech conversion method based on the bidirectional loss function of the variational auto-encoding network is shown in

[0083] S1, divide a speech data set into a training set and a test set;

[0084] In the embodiment, the process of step S1 is specifically as follows:

[0085] S1.1, select a speech data set: select CSTR VTCK as the speech data set of the embodiment, which is composed of the speeches of 109 English speakers with different accents;

[0086] S1.2, divide the speech data set: randomly select the speeches of 15 speakers as the test set, and the rest of the speeches as the training set.

[0087] S2, extract a mel-scale spectrogram from each speech sample of the speech data set and perform normalization processing;

[0088] In the embodiment, the process of step S2 is specifically as follows:

[0089] S2.1, perform endpoint detection and amplitude normalization: perform endpoint detection on the speech samples of the training set, remove the silence, and then perform amplitude normalization processing on the speech samples, and the normalization interval is 0-1 here;

[0090] S2.2, frame: divide the speech samples into short-time speech frames with a fixed length, and the time length is usually 10-40 ms.

[0091] S2.3, windowing: multiply each frame of speech with a window function omega(n) to obtain a windowed speech frame, and a Hamming window is selected as the window function:

[0092]

[0093] Wherein, L represents the frame length, and n represents the sampling frequency point;

[0094] S2.4, short-time Fourier transform: perform short-time Fourier transform on the selected speech sample to obtain a frequency spectrum, and the short-time Fourier transform formula is as follows:

[0095]

[0096] Wherein, x(lH-n) is a speech sample, l is the serial number of the current filter, represents the current time period, k is the serial number of the current frequency, represents that the kth frequency is currently being processed The variable n is in the range of 0≤n≤N-1, ω(n) is a window function, H is a sampling interval, and N represents sampling at N equally spaced frequencies.

[0097] S2.5, normalize the spectrogram: set the frequency size to a uniform scale, this embodiment is 512 bins, and then restore the linear scale spectrogram by using an approximate inverse linear transformation to convert the mel scale spectrogram into a speech waveform.

[0098] S3, design a variational auto-encoding network, the variational auto-encoding network includes a text encoder, a speaker encoder, and a decoder, wherein the text encoder and the speaker encoder are connected in parallel and then sequentially connected with the decoder.

[0099] In this embodiment, the step S3 process is specifically as follows:

[0100] S3.1, construct a speaker encoder, referred to as E S , the process is as follows:

[0101] S3.1.1, construct a ConBank layer: convolve the input speech waveform, the convolution kernel size, input dimension, output dimension, and sliding step of the convolution layer are 3×3, 128, 128, and 1 respectively, denoted as Conv1, then the convolution output result is padded and expanded, and then output after splicing;

[0102] S3.1.2, convolve the ConBank output result, then pass it through a ReLu function, and then output it again through a convolution and a ReLu function, wherein the convolution kernel size, input dimension, output dimension, and sliding step of the convolution layer are 3×3, 128, 128, and 2 respectively, denoted as Conv2, and the expression of the ReLu function is:

[0103]

[0104] Wherein x in is the input of the ReLu function;

[0105] S3.1.3, input the output result of the previous step to an average pooling layer, referred to as AvgPool, this embodiment is a two-dimensional average pooling.

[0106] S3.1.4, input the output result of the previous step to a fully connected layer, denoted as Dense, and then pass it through a ReLU function to obtain the final output of E S , that is, the speaker pronunciation characteristic information, denoted as z s, where the number of nodes of Dense is 128;

[0107] S3.2, constructing a text encoder, abbreviated as E c , the process is as follows:

[0108] S3.2.1, input the input speech waveform into ConBank;

[0109] S3.2.2, input the output result of the previous step into Conv1, then through the ReLu function, and then input into the normalization layer, denoted as IN, input the output of the normalization layer into Conv2, then through the ReLu function, and finally input into IN again to obtain E c The final output of E, that is, the text information, denoted as z c , wherein IN is to independently normalize the frequency of each channel along the channel direction, and the calculation of each channel is as follows:

[0110]

[0111] Where a and b are normalization weight parameters, which are non-zero constants, W is a data matrix on each channel in the normalization layer, E(W) is the expectation of W, Var(W) is the variance of W, and Y is the normalized value.

[0112] S3.3, constructing a decoder, abbreviated as D, the process is as follows:

[0113] S3.3.1, input z c to Conv1 first, then through the ReLu function, input into the adaptive normalization layer, abbreviated as AdaIN, and at the same time, input z s to Dense and ReLu function, and then input into AdaIN through the affine layer, wherein the affine layer is abbreviated as Affine, and the expression is:

[0114] Affine(x b )=f(wx b +c)

[0115] Where x b is the input of the previous layer, w is the weight parameter, c is the bias vector, w, c are constants, f(wx b +c) represents a nonlinear activation function about x b , which takes ReLu, Sigmoid or Tanh function;

[0116] For AdaIN, the expression is:

[0117]

[0118] Where μ(zc is the mean of the text information frequency, σ(z c is the variance of the text information frequency, μ(z s is the mean of the speaker information frequency, σ(z s is the variance of the speaker information frequency;

[0119] S3.3.2, input the output of the previous step AdaIN into Conv1 and ReLu functions again, pass through upsampling, and finally input AdaIN again to obtain the final output of D.

[0120] S4, construct a reconstruction loss function, a KL divergence loss function, a same speaker information loss function and a different speaker information loss function, thereby obtaining a total loss function, and iteratively train the variational auto-encoding network multiple times under the guidance of the total loss function;

[0121] In the embodiment, the step S4 process is specifically as follows:

[0122] S4.1, construct a loss function: let x i , i=1, 2, 3 be a speech sample in a training speech data set, and assume that x1 and x2 are speech samples of the same speaker but different speech texts, and x3 is a speech sample of another speaker, wherein the speech texts of x1 and x3 are the same;

[0123] Input x1 into E c and E s respectively to obtain speaker pronunciation characteristic information z s1 and text information z c1 ;

[0124] Input x2 into E c and E s respectively to obtain speaker pronunciation characteristic information z s2 and text information z c2 ;

[0125] Input x3 into E c and E s respectively to obtain speaker pronunciation characteristic information z s3 and text information z c3 ;

[0126] Input z s1 and z c2 into D to obtain reconstructed speech x2';

[0127] Input z s2 and z c1 into D to obtain reconstructed speech x1';

[0128] Input z s3 and z c3Input D to get reconstructed speech x3';

[0129] S4.1.1, construct speech reconstruction loss function: let p(z c |x i ) be a conditional independent Gaussian distribution, denoted as:

[0130] p(z c |x i ) = G(z c (x i ), I)

[0131] Where z c (x i ) represents the mean of the text information x i , I represents the unit variance of the text information x i , G(z c (x i ), I) represents the Gaussian distribution with mean z c (x i ) and unit variance I, and the expression of the probability density function f(t) is:

[0132]

[0133] Therefore, the expression of G(z c (x i ), I) is:

[0134]

[0135] Where t is a random variable of the Gaussian distribution probability density function;

[0136] Since p(z c |x i ) is a conditional independent Gaussian distribution, the speech reconstruction loss function L rec is defined as:

[0137] L rec = ||x1'-x1||1+||x2'-x2||1+||x3'-x3||1

[0138] Where ||x i '-x i ||1, i = 1, 2, 3 represents the 1-norm of the difference between x i ' and x i ;

[0139] S4.1.2, construct KL divergence loss function:

[0140] In order to make the posterior distribution p(z c |x i) matches the Gaussian distribution G(0, I) while requiring to minimize its KL divergence loss function L kl The minimum KL divergence loss function L kl is defined as:

[0141] L kl = ||z c (x i ) 2 ||2

[0142] where z c (x i ) 2 represents the square value of the frequency of the text information x i , ||z c (x i ) 2 ||2 represents the 2-norm of z c (x i ) 2 ;

[0143] S4.1.3, construct a bidirectional loss function:

[0144] Based on the idea of contrast learning, a bidirectional loss function is constructed, that is, maximizing the speaker information similarity between different speeches of the same speaker and minimizing the speaker information similarity of different speakers, so that two or more structure identical variational auto-encoding networks can be trained at the same time by inputting different data to get different outputs, and all outputs are trained together, so two loss functions can be reconstructed: the same speaker loss function L spk and the different speaker loss function L diff , which are defined as:

[0145] L spk = ||z s2 -z s1 ||1

[0146] L diff = -[||z s2 -z s3 ||1+||z s1 -z s3 ||1]

[0147] S4.1.4, merge the loss function:

[0148] The above several loss functions are merged to construct the total loss function L, and the total loss function L is defined as:

[0149]

[0150] where α, γ, are the KL divergence loss function Lkl Loss function for the same speaker L spk Loss functions for different speakers L diff The weighting coefficients are, in this embodiment, taken as 0.1, 0.99, and 0.99 in sequence.

[0151] S4.2 Training the Variational Autoencoder Network: Optimize the network using the Adam optimizer and train until the total loss function L converges. In this embodiment, the batch size is 2, the total number of training epochs is 1000, the learning rate is 0.01, and the learning rate reduction strategy is to adjust it to one-tenth of its original value every 100 training epochs.

[0152] S5. Randomly select test samples from the test set, input the Mel-scale spectral features of the test samples into the trained variational autoencoder network to obtain the reconstructed speech after speech conversion of the test samples.

[0153] In this embodiment, step S5 is as follows:

[0154] Mel-scale spectral features are extracted from the speech samples of the test dataset and input into the trained network to obtain the reconstructed speech. The similarity between the reconstructed speech and the original speech is calculated to obtain an objective evaluation result of the speech conversion. In this embodiment, the text recognition error rate (WER), sentence recognition error rate (SER), and character recognition error rate (CER) are 2.1%, 1.7%, and 3.1%, respectively. 。 Simultaneously, both speech samples were listened to by human ears to obtain subjective evaluations of the speech conversion. This example used 50 adult men and women aged 18-50 from different age groups. Regarding the speech conversion results, 45 selected "excellent," 3 selected "good," and 2 selected "average." Based on the combined objective and subjective evaluations, the speech conversion result is relatively ideal.

[0155] Example 2

[0156] Referring to the steps of Example 1, the speech dataset in step S1 is changed to the LibriSpeech speech dataset, and the α, γ values ​​in sub-step S4.1.4 of step S1 are changed. The values ​​were set to 0.05, 0.999, and 0.999 respectively, with the remaining steps and parameters unchanged. Finally, the results of this embodiment were obtained, with the text recognition error rate WER, sentence recognition error rate SER, and character recognition error rate CER being 1.2%, 0.6%, and 2.3% respectively.

[0157] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and principles of the present application should be equivalent replacement manners and should be included in the protection scope of the present application.

Claims

1. A voice conversion method based on a bidirectional loss function of a variational auto-encoding network, characterized by, The voice conversion method comprises the following steps: S1, dividing the voice data set into a training set and a test set; S2, extracting the Mel scale spectrogram from each voice sample of the voice data set, and performing normalization processing; S3, designing a variational auto-encoding network, the variational auto-encoding network comprising a text encoder, a speaker encoder and a decoder, wherein the text encoder and the speaker encoder are connected in parallel and then sequentially connected with the decoder; step S3 comprises: S3.1, constructing an encoder for a speaker, E S The process is as follows: S3.1.1, constructing a ConBank: performing convolution on the input voice waveform, the convolution kernel size, input dimension, output dimension and sliding step of the convolution layer being 3*3, 128, 128 and 1 respectively, denoted as Conv1, then filling and expanding the convolution output result, and outputting after splicing; S3.1.2, performing convolution on the ConBank output result, then passing through a ReLu function, outputting again, and then performing convolution and passing through a ReLu function again, wherein the convolution kernel size, input dimension, output dimension and sliding step of the convolution layer are 3*3, 128, 128 and 2 respectively, denoted as Conv2, and the expression of the ReLu function is: where x in is the input to the ReLu function; S3.1.3, inputting the output result of the previous step into an average pooling layer, denoted as AvgPool; S3.1.4, input the output result of the previous step into a full connection layer, denoted as Dense, and then pass it through a ReLU function to obtain E S the final output, i.e., the speaker pronunciation characteristic information, denoted as z s wherein the number of nodes of Dense is 128; S3.2, construct a text encoder, E c The process is as follows: S3.2.1, inputting the input voice waveform into the ConBank; S3.2.2, input the output result of the previous step to Conv1, then pass it through a ReLu function, and then input it to a normalization layer, denoted as IN, input the output of the normalization layer to Conv2, then pass it through a ReLu function, and finally input it to IN again to obtain E c the final output, i.e., text information, denoted as z c , wherein IN is a normalization calculation of the frequency of each channel independently along the channel direction, and the calculation of each channel is as follows: Wherein a and b are normalized weight parameters, a and b are non-zero constants, W is a data matrix on each channel in the normalization layer, E(W) is the expectation of W, Var(W) is the variance of W, and Y is the normalized value; S4, constructing a reconstruction loss function, a KL divergence loss function, a same speaker information loss function and a different speaker information loss function, thereby obtaining a total loss function, and iteratively training the variational auto-encoding network under the guidance of the total loss function; S5, randomly selecting a test sample from the test set, inputting the Mel scale spectrogram feature of the test sample into the trained variational auto-encoding network, and obtaining the reconstructed voice after voice conversion of the test sample.

2. The speech conversion method based on a variational auto-encoding network bidirectionalization loss function according to claim 1, characterized in that, The step S1 process is as follows: S1.1, selecting a voice data set: selecting CSTR VTCK, LibriSpeech and / or TED-LIUM as the voice data set; S1.2, dividing the voice data set: randomly selecting the voice of a plurality of speakers as the test set, and the voice of the remaining speakers as the training set. 3.The voice conversion method based on the variational auto-encoding network bidirectional loss function according to claim 1, wherein, The step S2 process is as follows: S2.1, performing endpoint detection and amplitude normalization: performing endpoint detection on the voice samples of the training set, removing the silence, and then performing amplitude normalization processing on the voice samples; S2.2, frame division: dividing the voice samples into fixed-length short-time voice frames; S2.3, windowing: multiplying each frame of voice with a window function ω(n) to obtain a windowed voice frame, and selecting a Hamming window as the window function: Wherein n represents the sampling frequency point, and N represents sampling at N equally spaced frequencies; S2.4, short-time Fourier transform: performing short-time Fourier transform on the selected voice sample to obtain an amplitude spectrum, and the short-time Fourier transform formula is as follows: wherein x(lH-n) is a speech sample, I is the serial number of the current filter, represents the current time period, k is the serial number of the current frequency, represents that the kth frequency is being processed signal, the variable n has a value range of 0≤n≤N-1, ω(n) is a window function, and H is a sampling interval; S2.5, normalized spectrogram: frequency size is set to uniform scale, and then an approximate inverse linear transformation is used to restore the linear scale spectrogram, so as to convert the mel scale spectrogram into a speech waveform.

4. The speech conversion method based on a variational auto-encoding network bidirectionalization loss function according to claim 1, characterized in that, The step S3 further comprises: S3.3, constructing a decoder, referred to as D, as follows: S3.3.1, z c is input to Conv1, then through the ReLu function, input to the adaptive normalization layer, simply referred to as AdaIN, and z s is input to Dense and the ReLu function, and then through the affine layer, input to AdaIN, where the affine layer is simply referred to as Affine, and the expression is: Affine(x b ) = f(wx b +c) wherein x b is the input of the previous layer, w is a weight parameter, c is a bias vector, w, c are constants, and f(w b x b +c) represents a nonlinear activation function on x b , which is ReLu, Sigmoid or Tanh function. For AdaIN, the expression is: wherein μ(z c ) is the mean of the text information frequency, σ(z c ) is the variance of the text information frequency, μ(z s ) is the mean of the speaker information frequency, and σ(z s ) is the variance of the speaker information frequency; S3.3.2, the output of the above step AdaIN is input again into Conv1 and ReLu functions, and finally input again into AdaIN through upsampling to obtain the final output of D.

5. The speech conversion method based on a variational auto-encoding network bidirectionalization loss function according to claim 1, characterized in that, The step S4 comprises: S4.1, construct loss function: let x i , i = 1, 2, 3 be speech samples in the training speech data set, assume that x1 and x2 are speech samples of the same speaker, but the speech texts are different, x3 is a speech sample of another speaker, wherein the speech texts of x1 and x3 are the same; Input x1 into E c and E s to obtain speaker pronunciation characteristic information z s1 and text information z c1 ; input x2 into E c and E s to obtain speaker pronunciation characteristic information z s2 and text information z c2 ; Input x3 into E c and E s to obtain speaker pronunciation characteristic information z s3 and text information z c3 ; z s1 and z c2 input D to obtain reconstructed speech x2'; z s2 and z c1 input D to obtain reconstructed speech x1'; z s3 and z c3 input D to obtain reconstructed speech x3'; S4.1.1, Constructing the speech reconstruction loss function: Let p(z c |x i ) be a conditional independent Gaussian distribution, denoted as: p(z c |x i ) = G(z c (x i ), I) where z c (x i ) denotes the mean of the text information x i , I denotes the unit variance of the text information x i , G(z c (x i ), I) denotes a Gaussian distribution with the mean z c (x i ) and the unit variance I, and the expression of the probability density function f(t) is: Thus G(z c (x i ),I) is given by: Wherein t is a random variable of a Gaussian distribution probability density function; Since p(z c | x i ) follows a conditionally independent Gaussian distribution, the speech reconstruction loss function L rec is defined as: L rec = ||x1'-x1||1+||x2'-x2||1+||x3'-x3||1 where ||x i '-x i ||1, i = 1, 2, 3 represent the 1-norm of the difference between x i 'and x i ; S4.1.2, constructing a KL divergence loss function: To match the posterior distribution p(z c |x i ) to the Gaussian distribution G(0, I), while requiring the KL divergence loss function L kl to be minimized, the KL divergence loss function L kl is defined as: L kl =||z c (x i ) 2 ||2 where z c (x i ) 2 denotes the text information x i the square of the frequency; ||z c (x i ) 2 ||2denotes the 2-norm of z c (x i ) 2 ; S4.1.3, constructing a bidirectional loss function: Constructing the same speaker loss function L spk and different speaker loss function L diff , respectively, as L spk =||z s2 -z s1 ||1 L diff = -[||z s2 -z s3 ||1+||z s1 -z s3 ||1] S4.1.4, merging the loss functions: The above loss functions are merged to construct a total loss function L, and the total loss function L is defined as: wherein a, g, respectively KL divergence loss function L kl , same speaker loss function L spk , different speaker loss function L diff weighting coefficients; S4.2, training the variational auto-encoder network: the network is optimized by using an Adam optimizer, and the training is performed until the total loss function L converges.

6. The speech conversion method based on a variational autoencoder network bidirectionalization loss function according to claim 1, characterized in that, The step S5 process is as follows: The mel scale spectral features of the speech samples of the test data set are extracted and input into the trained variational auto-encoder network to obtain the reconstructed speech after conversion, the similarity between the reconstructed speech and the original speech is calculated, and the objective evaluation result of the speech conversion is obtained; at the same time, the two kinds of speeches are listened to by the human ear to obtain the subjective evaluation result of the speech conversion.

Images