Speech synthesis method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a voice synthesis method and device, equipment and a storage medium. The method comprises the steps of receiving a text of a to-be-synthesized voice input by a user; inputting a phoneme sequence corresponding to the text into a preprocessing part in the speech synthesis model to obtain an intermediate vector; inputting the intermediate vector after dimension regularization into a multiphase orthogonal mirror image analysis filter to obtain an analysis signal; inputting the analysis signal into a coupling layer in a speech synthesis model so as to perform inverse transformation on the analysis signal to obtain an inverse transformation signal; inputting the inverse transformation signal into a multiphase orthogonal mirror image synthesis filter to obtain a synthesis signal; and inputting the synthetic signal after the dimension inverse regularization into a decoder in the speech synthesis model, and synthesizing target speech according to the speech waveform. According to the embodiment of the invention, the multiphase orthogonal mirror filter is introduced into the coupling layer of the normalized stream, so that the overall stability and robustness of the speech synthesis model can be remarkably improved, and the stability of the quality of speech synthesized by applying the normalized stream technology is improved.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and in particular to a speech synthesis method and device, an electronic device and a storage medium. Background Art

[0002] In the field of speech generation, the use of the VITS system (Variational Inference with Transformer-based Stochastic Speech Generation, a speech synthesis technology based on variational inference and Transformer architecture) to synthesize speech is a classic example of the application of normalized flow technology. The core structure of the flow module in the VITS system is the normalized flow. The normalized flow is usually composed of multiple coupling layers in series, which can realize the transformation of the signal from x to z and the inverse transformation from z to x. Specifically, if the dimensions of the signal x and the signal z are both D, then by presetting a constant d (d is less than D), the D-dimensional signal can be divided into two parts. In the coupling layer transformation, the upper half remains unchanged, while the lower half is affine transformed, and the coefficients and biases of the affine transformation are calculated by the upper half.

[0003] Therefore, each time a coupling layer transformation is passed, only a part of the dimension will change, while the rest of the dimensions remain unchanged. To solve this problem, the current VITS system introduces a dimension flipping operation after each coupling layer transformation, swapping the changed part with the unchanged part before entering the next coupling layer. In this way, the coupling layer transformations of an even number of layers in series can make the two dimensions undergo roughly the same number of transformations.

[0004] However, this method of alternating the two dimensions is not uniform enough. For example, it is like jumping one step with the left foot and then jumping one step with the right foot, which is different from jumping one step with both feet at the same time. This alternating transformation method may lead to insufficient stability of the speech synthesis model built by the VITS system, which in turn affects the quality stability of the generated speech. Summary of the invention

[0005] The embodiment of the present application provides a speech synthesis method to solve the problem of poor stability of speech quality synthesized by using normalized stream technology in the prior art.

[0006] Correspondingly, the embodiments of the present application also provide a speech synthesis device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, the present application discloses a speech synthesis method, which includes:

[0008] Receive text of speech to be synthesized input by a user;

[0009] Inputting the phoneme sequence corresponding to the text of the speech to be synthesized into the preprocessing part of the speech synthesis model to obtain an intermediate vector;

[0010] After dimensionally regularizing the intermediate vector, the dimensionally regularized intermediate vector is input into a multi-phase orthogonal mirror analysis filter to obtain an analysis signal;

[0011] Inputting the analysis signal into a coupling layer in the speech synthesis model to inversely transform the analysis signal to obtain an inverse transformation signal;

[0012] Inputting the inverse transformed signal into a polyphase quadrature mirror synthesis filter to obtain a synthesized signal;

[0013] After performing dimension denormalization on the synthesized signal, the dimension denormalized synthesized signal is input into a decoder in the speech synthesis model to output a speech waveform;

[0014] A target speech corresponding to the text of the speech to be synthesized is synthesized according to the speech waveform.

[0015] Optionally, the dimensionally regularizing the intermediate vector includes:

[0016] Dividing the intermediate vector into two sub-vectors according to a median value of the length of the intermediate vector;

[0017] Cross-combining the elements of the two sub-vectors one by one to obtain the intermediate vector after the dimension is regularized;

[0018] The step of performing dimensionality dewarping on the synthesized signal comprises:

[0019] According to the element arrangement order of the intermediate vector, the element arrangement order of the synthetic signal is restored to obtain the synthetic signal after the dimension is inversely regularized.

[0020] Optionally, inputting the inverse transformed signal into a multi-phase orthogonal mirror synthesis filter to obtain a synthesized signal comprises:

[0021] After upsampling the inverse transformation signal by a preset multiple, an upsampled inverse transformation signal is obtained;

[0022] Inputting the up-sampled inverse transform signal into the multi-phase orthogonal mirror synthesis filter to obtain a preliminary synthesized signal;

[0023] The preliminary composite signals are added to obtain the composite signal.

[0024] Optionally, the preprocessing part in the speech synthesis model includes at least a text encoder, a projection layer, and a random duration predictor, and the step of inputting the phoneme sequence corresponding to the text of the speech to be synthesized into the preprocessing part in the speech synthesis model to obtain the intermediate vector includes:

[0025] Inputting the phoneme sequence corresponding to the text of the speech to be synthesized into the text encoder to obtain a matrix of a preset size;

[0026] Input the matrix into the projection layer to obtain a mean and a variance, wherein the mean and the variance are used to shape a corresponding Gaussian distribution model;

[0027] Performing random sampling on the Gaussian distribution model to obtain a random sampling result;

[0028] According to a replication parameter, a time replication operation is performed on the random sampling result to obtain the intermediate vector, and the replication parameter is obtained by inputting the matrix into the random duration predictor.

[0029] Optionally, the speech synthesis model further includes at least a posterior encoder and a stream module. Before receiving the text of the speech to be synthesized input by the user, the method further includes:

[0030] Collect training recordings and corresponding training texts;

[0031] Inputting the phoneme sequence corresponding to the training text into the text encoder to obtain a training matrix of a preset size;

[0032] Inputting the training matrix into the projection layer to obtain a first training mean and a first training variance of a first preset size, wherein the first training mean and the first training variance are used to shape a corresponding first training Gaussian distribution model;

[0033] Inputting the linear spectrum corresponding to the training recording into the posterior encoder to obtain a second training mean and a second training variance of a second preset size, wherein the second training mean and the second training variance are used to shape a corresponding second training Gaussian distribution model;

[0034] Performing random sampling on the second training Gaussian distribution model to obtain a training random sampling result;

[0035] Input the training random sampling results into the decoder and the stream module respectively to obtain the generation loss and the training vector;

[0036] Performing time-series alignment on the training vector and the first training Gaussian distribution model to obtain an alignment result and a relative entropy between the training vector and the first training Gaussian distribution model;

[0037] Comparing the alignment result with a training replication parameter to obtain a duration loss, wherein the training replication parameter is obtained by inputting the training matrix into the random duration predictor;

[0038] Summarizing the generation loss, the duration loss, and the relative entropy to obtain a loss function value;

[0039] The speech synthesis model is updated using the loss function value.

[0040] Optionally, the stream module includes the polyphase orthogonal mirror analysis filter, the coupling layer, and the polyphase orthogonal mirror synthesis filter, and the training random sampling result is input into the decoder and the stream module respectively to obtain the generation loss and the training vector, including:

[0041] After dimensionally regularizing the training random sampling results, the dimensionally regularized training random sampling results are input into the multi-phase orthogonal mirror analysis filter to obtain a training analysis signal;

[0042] Inputting the training analysis signal into the coupling layer to perform forward transformation on the training analysis signal to obtain a forward transformation signal;

[0043] Inputting the forward transformed signal into the multi-phase orthogonal mirror synthesis filter to obtain a training synthesis signal;

[0044] The training synthetic signal is dimensionally inversely regularized to obtain the training vector.

[0045] Optionally, the step of inputting the training random sampling result into the decoder and the stream module respectively to obtain a generation loss and a training vector comprises:

[0046] Inputting the training random sampling result into the decoder and outputting a training speech waveform;

[0047] The error between the training speech waveform and the training recording is calculated to obtain the generation loss.

[0048] The present application also discloses a speech synthesis device, which includes:

[0049] A text input module, used to receive text to be synthesized speech input by a user;

[0050] A preprocessing module, used for inputting the phoneme sequence corresponding to the text of the speech to be synthesized into the preprocessing part of the speech synthesis model to obtain an intermediate vector;

[0051] An analysis signal acquisition module is used to input the dimensionally regularized intermediate vector into a multi-phase orthogonal mirror analysis filter to obtain an analysis signal after dimensionally regularizing the intermediate vector;

[0052] An inverse transformation module, used for inputting the analysis signal into a coupling layer in the speech synthesis model to perform an inverse transformation on the analysis signal to obtain an inverse transformation signal;

[0053] A synthetic signal acquisition module, used for inputting the inverse transformation signal into a multi-phase orthogonal mirror synthesis filter to obtain a synthetic signal;

[0054] A dimensionality deregulation module, configured to, after performing dimensionality deregulation on the synthesized signal, input the synthesized signal after dimensionality deregulation into a decoder in the speech synthesis model, and output a speech waveform;

[0055] The speech synthesis module is used to synthesize a target speech corresponding to the text of the speech to be synthesized according to the speech waveform.

[0056] The embodiment of the present application also discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the speech synthesis method as described in one or more of the embodiments of the present application.

[0057] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes the speech synthesis method as described in one or more of the embodiments of the present application.

[0058] Compared with the prior art, the embodiments of the present application have the following advantages:

[0059] In an embodiment of the present application, a text of speech to be synthesized input by a user is received; the phoneme sequence corresponding to the text of the speech to be synthesized is input into the preprocessing part of the speech synthesis model to obtain an intermediate vector; after the intermediate vector is dimensionally regularized, the dimensionally regularized intermediate vector is input into a polyphase orthogonal mirror analysis filter to obtain an analysis signal; the analysis signal is input into a coupling layer in the speech synthesis model to inversely transform the analysis signal to obtain an inverse transformation signal; the inverse transformation signal is input into a polyphase orthogonal mirror synthesis filter to obtain a synthesized signal; after the synthetic signal is dimensionally inversely regularized, the dimensionally inversely regularized synthesized signal is input into a decoder in the speech synthesis model to output a speech waveform; according to the speech waveform, a target speech corresponding to the text of the speech to be synthesized is synthesized. The embodiment of the present application introduces a polyphase orthogonal mirror filter (PQMF, Polyphase Quadrature Mirror Filter) in the coupling layer of the normalized flow, and establishes a fixed pattern connection between the dimensions of the intermediate vector before the coupling layer is processed. This approach allows all dimensions of the intermediate vector to participate in the processing operations within the coupling layer, thereby helping to make the dimensions of the vector more uniform and consistent. In this way, the overall stability and robustness of the speech synthesis model can be significantly improved, thereby improving the stability of the speech quality synthesized using the normalized flow technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a schematic diagram of a coupling layer of an embodiment of a speech synthesis method of the present application;

[0061] Figure 2 is a flowchart of a speech synthesis method embodiment of the present application;

[0062] Figure 3 This is a schematic diagram of a speech synthesis model structure of a speech synthesis method embodiment of the present application;

[0063] Figure 4 This is a schematic diagram of the flow module structure of a speech synthesis method embodiment of the present application;

[0064] Figure 5 is a structural block diagram of an embodiment of a speech synthesis device of the present application;

[0065] Figure 6 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0067] Reference Figure 1, is a schematic diagram of the coupling layer of an embodiment of a speech synthesis method of the present application.

[0068] In the field of speech generation, using the VITS system to synthesize speech is a classic example of applying normalized flow technology. The core structure of the flow module in the VITS system is the normalized flow. The normalized flow is usually composed of multiple coupling layers in series, which can realize the transformation of the signal from x to z, and the inverse transformation from z to x. Figure 1 , showing the specific operation process of the transformation from x to z for a single coupling layer in the normalized flow.

[0069] Specifically, if the dimensions of signal x and signal z are both D, then by presetting a constant d (d is less than D), the D-dimensional signal can be divided into two parts, namely, the upper half of x 1 ,……,x d and the lower part of x d+1 ,……,x D In the coupled layer transformation, the upper part remains unchanged and can be directly obtained by x 1 ,……,x d Copy to get z 1 ,……,z d The lower part performs an affine transformation, and the coefficients and bias of the affine transformation are calculated by the upper part. Figure 1 As shown, β d+1 , ..., β D is the coefficient of the affine transformation, which is calculated by the neural network submodule F; d+1 ,……,γ D is the bias of the affine transformation, calculated by the neural network submodule H. d+1 ,……,x D After affine transformation, we can get z d+1 ,……,z D The inverse transformation from z to x is to keep the upper part unchanged and directly transform from z 1 ,……,z d Copy to get x 1 ,……,x d ;z d+1 ,……,z D After the inverse affine transformation, we can get x d+1 ,……,x D .

[0070] Therefore, each time after the coupling layer transformation, only a part of the dimensions will change, and the remaining dimensions remain unchanged. To solve this problem, the current VITS system introduces a dimension flipping operation after each coupling layer transformation, swapping the changed part with the unchanged part, and then entering the next coupling layer. In this way, by cascading the coupling layer transformations of even layers, the two parts of the dimensions can roughly undergo the same number of transformations.

[0071] However, this method of alternating transformation of the two parts of the dimensions is not uniform enough. For example, it is like taking one step with the left foot and then one step with the right foot, which is different from the experience of taking one step with both feet at the same time. This alternating transformation method may lead to insufficient stability of the speech synthesis model constructed by the VITS system, thereby affecting the quality stability of the generated speech.

[0072] To this end, the speech synthesis method provided by the embodiments of the present application introduces the PQMF algorithm to establish a fixed pattern connection between the dimensions of the intermediate vector in advance before the coupling layer processing. This approach enables all dimensions of the intermediate vector to participate in the "two-way variable / constant" processing operation within the coupling layer.

[0073] Refer to Figure 2 , which is a flowchart of the steps of an embodiment of the speech synthesis method of the present application, including the following steps:

[0074] Step 201: Receive the text of the speech to be synthesized input by the user.

[0075] In step 201, the text of the speech to be synthesized input by the user is received. In an optional embodiment, for the text of the speech to be synthesized input by the user, a third-party phonetic annotation tool (such as pypinyin) is required to translate the text into the phoneme sequence corresponding to the text of the speech to be synthesized. For example, if the text input by the user is "good person", the phoneme sequence corresponding to the text of the speech to be synthesized is "h3 ao3r2 en2".

[0076] Step 202: Input the phoneme sequence corresponding to the text of the speech to be synthesized into the preprocessing part of the speech synthesis model to obtain an intermediate vector.

[0077] In step 202, the phoneme sequence corresponding to the text of the speech to be synthesized is input into the speech synthesis model. The input phoneme sequence first needs to pass through the preprocessing part of the speech synthesis model, and then the intermediate vector is obtained through preprocessing.

[0078] In a feasible embodiment, the preprocessing part includes at least a text encoder, a projection layer, and a random duration predictor. First, the phoneme sequence corresponding to the text of the speech to be synthesized is input into the text encoder to obtain a matrix of a preset size. If the input phoneme sequence is "h3 ao3 r2 en2", the resulting matrix size is 4×D, where D is a preset internal dimension. For example, D can be 192. Then, this matrix is ​​input into the projection layer to obtain the mean μ θ and variance σ θ , which can be used to model the corresponding Gaussian distribution. In this example, the mean μ θ and variance σ θ The sizes of are also 4×D, which essentially creates four Gaussian distribution models. Then, take any random number rand and calculate z_p=μ θ +rand×σ θ , z_p is the random sampling result of the Gaussian distribution model, and its dimension is also 4×D. Finally, the matrix is ​​input into the random duration predictor to obtain the replication parameter d; according to the replication parameter d, the random sampling result z_p is replicated at all times to obtain the intermediate vector.

[0079] Step 203: After dimensionally regularizing the intermediate vector, the dimensionally regularized intermediate vector is input into a multi-phase orthogonal mirror analysis filter to obtain an analysis signal.

[0080] In step 203, the intermediate vector is first dimensionally normalized. Specifically, in an optional embodiment, if the intermediate vector z is a D-dimensional [z 1 ,z 2 ,…,z D ], then z is divided into two equal parts of D / 2 dimensions, that is, [z 1 ,z 2 ,…,z D / 2 ] and [z D / 2+1 ,z D / 2+2 ,…,z D ], and then cross the elements one by one to get [z 1 ,z D / 2+1 ,z 2 ,z D / 2+2 ,…,z D / 2 ,z D ], thereby completing the dimensionality regularization of the intermediate vectors.

[0081] Then, the dimensionally regularized intermediate vector is input into the polyphase orthogonal mirror analysis filter to obtain the analysis signal. 1 ,z D / 2 +1,z 2 ,z D / 2+2 ,…,zD / 2 ,z D ] Input the multi-phase orthogonal mirror analysis filter to obtain two analysis signals with a length of D / 2.

[0082] Step 204: input the analysis signal into the coupling layer in the speech synthesis model to perform inverse transformation on the analysis signal to obtain an inverse transformation signal.

[0083] In step 204, the analysis signal is input into the coupling layer in the speech synthesis model to inversely transform the analysis signal to obtain an inverse transformation signal. For example, the two analysis signals of length D / 2 obtained above are inversely input into the coupling layer to output two inverse transformation signals of length D / 2.

[0084] Step 205: Input the inverse transformed signal into a multi-phase orthogonal mirror synthesis filter to obtain a synthesized signal.

[0085] In step 205, the inverse transform signal is input into a multi-phase orthogonal mirror synthesis filter to obtain a synthesized signal. In an optional embodiment, the process of obtaining the synthesized signal can be expressed as follows: after upsampling the inverse transform signal by a preset multiple, the upsampled inverse transform signal is obtained, and the inverse transform signal is input into a multi-phase orthogonal mirror synthesis filter to obtain a preliminary synthesized signal, and all the preliminary synthesized signals are added to obtain a synthesized signal. For example, two inverse transform signals of length D / 2 are upsampled by 2 times respectively, and then input into a multi-phase orthogonal mirror synthesis filter to obtain two preliminary synthesized signals of length D, and the two are added to obtain a synthesized signal of length D.

[0086] Step 206: After de-dimensionalizing the synthesized signal, input the de-dimensionalized synthesized signal into a decoder in the speech synthesis model to output a speech waveform.

[0087] In step 206, the synthesized signal needs to be dimensionally denormalized to restore the element order of the original intermediate vector, and then the denormalized synthesized signal is input into the decoder in the speech synthesis model to output the speech waveform. For example, the synthesized signal x is [x 1 ,x D / 2+1 ,x 2 ,x D / 2+2 ,…,x D / 2 ,x D ], after dimensional inverse normalization, it is restored to [x 1 ,x 2 ,…,x D ], and then [x 1 ,x 2 ,…,x D ]Enter the decoder.

[0088] Step 207: synthesize a target speech corresponding to the text of the speech to be synthesized according to the speech waveform.

[0089] In step 207, a target speech corresponding to the text of the speech to be synthesized is synthesized according to the speech waveform output by the decoder, and fed back to the user.

[0090] In an embodiment of the present application, a text of speech to be synthesized input by a user is received; the phoneme sequence corresponding to the text of the speech to be synthesized is input into the preprocessing part of the speech synthesis model to obtain an intermediate vector; after the intermediate vector is dimensionally regularized, the dimensionally regularized intermediate vector is input into a multi-phase orthogonal mirror analysis filter to obtain an analysis signal; the analysis signal is input into a coupling layer in the speech synthesis model to inversely transform the analysis signal to obtain an inverse transformation signal; the inverse transformation signal is input into a multi-phase orthogonal mirror synthesis filter to obtain a synthesis signal; after the synthesis signal is dimensionally inversely regularized, the dimensionally inversely regularized synthesis signal is input into a decoder in the speech synthesis model to output a speech waveform; according to the speech waveform, a target speech corresponding to the text of the speech to be synthesized is synthesized. In an embodiment of the present application, a fixed pattern connection is established between the dimensions of the intermediate vector before the coupling layer is processed by introducing a multi-phase orthogonal mirror filter in the coupling layer of the normalized flow. This approach enables all dimensions of the intermediate vector to participate in the processing operations within the coupling layer, thereby helping to make the dimensions of the vector more uniform and consistent. In this way, the overall stability and robustness of the speech synthesis model can be significantly improved, thereby improving the stability of the speech quality synthesized using the normalized flow technology.

[0091] In an optional embodiment of the present application, the dimension regularization of the intermediate vector includes:

[0092] Dividing the intermediate vector into two sub-vectors according to a median value of the length of the intermediate vector;

[0093] Cross-combining the elements of the two sub-vectors one by one to obtain the intermediate vector after the dimension is regularized;

[0094] The step of performing dimensionality dewarping on the synthesized signal comprises:

[0095] According to the element arrangement order of the intermediate vector, the element arrangement order of the synthetic signal is restored to obtain the synthetic signal after the dimension is inversely regularized.

[0096] In this embodiment, when the intermediate vector is dimensionally normalized, the intermediate vector may be divided into two sub-vectors according to the median value of the length of the intermediate vector, and then the elements of the two sub-vectors are cross-combined one by one to obtain the dimensionally normalized intermediate vector. For example, if the intermediate vector z is D-dimensional [z 1 ,z2 ,…,z D ], then z is divided into two equal parts of D / 2 dimensions, that is, [z 1 ,z 2 ,…,z D / 2 ] and [z D / 2+1 ,z D / 2+2 ,…,z D ], and then cross the elements one by one to get [z 1 ,z D / 2+1 ,z 2 ,z D / 2+2 ,…,z D / 2 ,z D ], thereby completing the dimensionality regularization of the intermediate vectors.

[0097] After the composite signal is obtained by inputting the multi-phase orthogonal mirror synthesis filter, the composite signal needs to be dimensionally denormalized. Specifically, when performing dimension denormalization, the order of the elements of the composite signal is restored according to the order of the elements of the intermediate vector, that is, according to the original order of the elements, to obtain the composite signal after dimension denormalization. In this example, the composite signal x is [x 1 ,x D / 2+1 ,x 2 ,x D / 2+2 ,…,x D / 2 ,x D ], after dimensional inverse normalization, it is restored to [x 1 ,x 2 ,…,x D ], and then [x 1 ,x 2 ,…,x D ]Enter the decoder.

[0098] The embodiment of the present application can effectively improve the parallelism of data processing by dimensionally regularizing the intermediate vectors and de-dimensionalizing the synthesized signals, thereby accelerating the processing speed; de-dimensionalizing the synthesized signals can ensure that the original structure of the signals is restored.

[0099] In an optional embodiment of the present application, inputting the inverse transformation signal into a multi-phase orthogonal mirror synthesis filter to obtain a synthesized signal includes:

[0100] After upsampling the inverse transformation signal by a preset multiple, an upsampled inverse transformation signal is obtained;

[0101] Inputting the up-sampled inverse transform signal into the multi-phase orthogonal mirror synthesis filter to obtain a preliminary synthesized signal;

[0102] The preliminary composite signals are added to obtain the composite signal.

[0103] In this embodiment, the specific steps of performing PQMF synthesis on the inverse transform signal can be expressed as follows. First, the inverse transform signal is upsampled by a preset multiple to obtain an upsampled inverse transform signal, and then input into a multi-phase orthogonal mirror synthesis filter to obtain a preliminary synthesized signal, and finally all the preliminary synthesized signals are added to obtain a synthesized signal. In a feasible embodiment, two inverse transform signals with a length of D / 2 are upsampled by 2 times respectively, and then input into a multi-phase orthogonal mirror synthesis filter to obtain two preliminary synthesized signals with a length of D, and the two are added to obtain a synthesized signal with a length of D.

[0104] The embodiment of the present application obtains a synthesized signal by inputting the inverse transformed signal into a multi-phase orthogonal mirror synthesis filter, which helps to make the internal dimension of the signal more uniform and consistent, and can significantly improve the overall stability and robustness of the speech synthesis model, thereby improving the stability of the speech quality synthesized using the normalized flow technology.

[0105] In an optional embodiment of the present application, the preprocessing part in the speech synthesis model includes at least a text encoder, a projection layer, and a random duration predictor, and the inputting of the phoneme sequence corresponding to the text of the speech to be synthesized into the preprocessing part in the speech synthesis model to obtain the intermediate vector includes:

[0106] Inputting the phoneme sequence corresponding to the text of the speech to be synthesized into the text encoder to obtain a matrix of a preset size;

[0107] Input the matrix into the projection layer to obtain a mean and a variance, wherein the mean and the variance are used to shape a corresponding Gaussian distribution model;

[0108] Performing random sampling on the Gaussian distribution model to obtain a random sampling result;

[0109] According to a replication parameter, a time replication operation is performed on the random sampling result to obtain the intermediate vector, and the replication parameter is obtained by inputting the matrix into the random duration predictor.

[0110] In this embodiment, the preprocessing part in the speech synthesis model may include at least a text encoder, a projection layer, and a random duration predictor. During preprocessing, the phoneme sequence corresponding to the text of the speech to be synthesized passes through the text encoder, the projection layer, and the random duration predictor in sequence.

[0111] First, the phoneme sequence corresponding to the text to be synthesized is input into the text encoder to obtain a matrix of a preset size. In one example, if the input phoneme sequence is "h3 ao3 r2 en2", the size of the obtained matrix is ​​4×D, where D is a preset internal dimension, for example, D can be 192.

[0112] Next, the matrix is ​​input into the projection layer to obtain the mean and variance, which are used to shape the corresponding Gaussian distribution model. In one example, the mean μ θ and variance σ θ The sizes of are also 4×D, so their essential meaning is to shape 4 Gaussian distribution models. Further, the Gaussian distribution model is randomly sampled to obtain random sampling results. The specific process of random sampling can be expressed as, take any random number rand, calculate z_p = μ θ +rand×σ θ , z_p is the random sampling result of the Gaussian distribution model, and its dimension is also 4×D.

[0113] Finally, the random sampling result is copied at a certain moment according to the copy parameter to obtain an intermediate vector. The copy parameter is obtained by inputting the matrix into the random duration predictor. In one example, the matrix is ​​input into the random duration predictor to obtain the copy parameter d, and then the random sampling result z_p is copied at a certain moment according to the copy parameter d, that is, each moment in z_p is copied several times according to the number of times each moment is repeated, thereby obtaining an intermediate vector.

[0114] The embodiment of the present application preprocesses the phoneme sequence corresponding to the text of the synthesized speech, which can improve the quality of the synthesized speech, make the synthesized speech clearer and more accurate, and prepare for subsequent normalized stream processing.

[0115] In an optional embodiment of the present application, the speech synthesis model further includes at least a posterior encoder and a stream module. Before receiving the text of the speech to be synthesized input by the user, the method further includes:

[0116] Collect training recordings and corresponding training texts;

[0117] Inputting the phoneme sequence corresponding to the training text into the text encoder to obtain a training matrix of a preset size;

[0118] Inputting the training matrix into the projection layer to obtain a first training mean and a first training variance of a first preset size, wherein the first training mean and the first training variance are used to shape a corresponding first training Gaussian distribution model;

[0119] Inputting the linear spectrum corresponding to the training recording into the posterior encoder to obtain a second training mean and a second training variance of a second preset size, wherein the second training mean and the second training variance are used to shape a corresponding second training Gaussian distribution model;

[0120] Performing random sampling on the second training Gaussian distribution model to obtain a training random sampling result;

[0121] Input the training random sampling results into the decoder and the flow module respectively to obtain a generation loss and a training vector;

[0122] Perform temporal alignment on the training vector and the first training Gaussian distribution model to obtain an alignment result and the relative entropy between the training vector and the first training Gaussian distribution model;

[0123] Compare the alignment result with the training copy parameter to obtain a duration loss, where the training copy parameter is obtained by inputting the training matrix into the random duration predictor;

[0124] Sum up the generation loss, the duration loss, and the relative entropy to obtain a loss function value;

[0125] Update the speech synthesis model using the loss function value.

[0126] In this embodiment, before putting the speech synthesis model into use, it is necessary to train the speech synthesis model. The specific training steps are as follows.

[0127] First, collect training recordings and corresponding training texts. In an optional embodiment, the length of each training recording does not exceed 20 seconds. This application does not impose any restrictions on the collected training recordings and corresponding training texts, and those skilled in the art can collect them according to actual needs. Use a third-party phonetic annotation tool (such as pypinyin) to translate the training text into a corresponding phoneme sequence. For Chinese, the phoneme sequence consists of initials and finals. For example, if the training text is "good person", the corresponding phoneme sequence of the training text is "h3 ao3 r2 en2". After the training data is prepared, repeat the following training operations on the training recordings and corresponding training texts.

[0128] Input the phoneme sequence corresponding to the training text into the text encoder to obtain a training matrix of a preset size. In an example, if the input phoneme sequence is "h3 ao3 r2 en2", the size of the obtained training matrix is 4×D, where D is a preset internal dimension. For example, D can be 192.

[0129] Input the training matrix into the projection layer to obtain a first training mean and a first training variance of a first preset size, which are used to shape the corresponding first training Gaussian distribution model. In an example, the obtained first training mean μ θ and the first training variance σ θ also have a first preset size of 4×D respectively, and in essence, they shape 4 first training Gaussian distribution models.

[0130] At the same time, the training recording is processed and the linear spectrum is extracted using a third-party signal processing tool (such as torchaudio that comes with pyTorch). In one example, if the recording length corresponding to "good guy" is 1 second, and the third-party signal processing tool is used to extract the linear spectrum with the following settings: "frame shift 10 milliseconds, FFT window length 1024", the size of the obtained linear spectrum is 100x513. The embodiment of the present application does not impose any restrictions on the process of extracting the linear spectrum of the training recording, and those skilled in the art can select an appropriate method to extract the linear spectrum of the training recording according to actual needs.

[0131] Afterwards, the linear spectrum corresponding to the training recording is input into the a posteriori encoder to obtain a second training mean and a second training variance of a second preset size, and the second training mean and the second training variance are used to shape the corresponding second training Gaussian distribution model. In one example, a linear spectrum of size 100x513 is input into the a posteriori encoder to obtain a second training mean μ q and the second training variance σ q , and the second preset sizes are 100xD respectively. In essence, they shape 100 second training Gaussian distribution models. The second training Gaussian distribution model here is used to describe pronunciation, while the first training Gaussian distribution model is used to describe text.

[0132] Furthermore, the second training Gaussian distribution model is randomly sampled to obtain a training random sampling result. The specific process of random sampling can be expressed as: take any random number rand and calculate z = μ q +rand×σ q , z is the random sampling result of the Gaussian distribution model, and its dimension is also 100×D.

[0133] Next, the training random sampling results need to be output in two ways. The first way is input to the decoder to obtain the training speech waveform, and the error is calculated with the original real training recording to obtain the generation loss; the other way is output to the stream module, and the training vector f is obtained after processing. θ (z). In an optional embodiment, the stream module of the speech synthesis model actually includes a multi-phase orthogonal mirror analysis filter, a coupling layer, and a multi-phase orthogonal mirror synthesis filter, that is, the normalized stream introduced with the PQMF algorithm provided in the embodiment of the present application. In one example, the output training vector f θ The size of (z) is 100×D.

[0134] The training vector is time-series aligned with the first training Gaussian distribution model to obtain an alignment result and a relative entropy between the training vector and the first training Gaussian distribution model. θ and the first training variance σ θ and training vector fθ (z) Use the MAS algorithm (a dynamic search algorithm) for time alignment. In one example, since the length of the former is 4 and the length of the latter is 100, the MAS algorithm is used to correspond each moment of the latter to the target moment of the former. For example, the alignment result of the length of 100 is [1,1,1,1,1,1,2,2,2,3,3,3,3…4,4,4,4,4,4]. For this alignment result, the number of repetitions of the text corresponding to the speech at each moment is counted. For example, the first moment of the text is repeated 6 times and the second moment is repeated 3 times. By repeating different moments of the text, the time alignment of the training vector and the first training Gaussian distribution model is achieved.

[0135] In an optional embodiment, the relative entropy (or KL distance) between the first training Gaussian distribution model (i.e., text Gaussian distribution) and the second training Gaussian distribution model (i.e., pronunciation Gaussian distribution) is calculated according to the alignment result obtained by performing time-series alignment between the training vector and the first training Gaussian distribution model. The specific formula of relative entropy can be expressed as formula (1):

[0136]

[0137] Among them, KL distance is the relative entropy, μ θ is the first training mean, f θ (z) is the training vector, σ θ is the first training variance, σ q is the second training variance.

[0138] At the same time, the alignment result is compared with the training copy parameter to obtain the duration loss, and the training copy parameter is obtained by inputting the training matrix into the random duration predictor. In this embodiment, the training matrix is ​​input into the random duration predictor to obtain the training copy parameter d, d describes the number of repetitions of each moment of the text, and the difference between the training copy parameter d and the alignment result is calculated to obtain the duration loss. In one example, the size of the training copy parameter d is 4x1.

[0139] After obtaining the three loss function parameters, the generation loss, duration loss and relative entropy are summarized to obtain the loss function value required for speech synthesis model training, and the speech synthesis model is updated using the loss function value. In an optional embodiment, the neural network parameters of the speech synthesis model are updated according to a standard neural network gradient backpropagation algorithm.

[0140] Reference Figure 3 , is a schematic diagram of the speech synthesis model structure of an embodiment of the speech synthesis method of the present application.

[0141] Figure 3The structure of the speech synthesis model and the flow steps of its training process of the embodiment of the present application are shown. The speech synthesis model includes a posterior encoder, a decoder, a text encoder, a projection layer, a stream module, a timing alignment module and a random duration predictor.

[0142] like Figure 3 As shown, the phoneme sequence corresponding to the training text is first input into the text encoder, and the output training matrix h text The training matrix h text Further input the projection layer to obtain the first training mean μ θ and the first training variance σ θ , their essential meaning is to shape the 4 first training Gaussian distribution models.

[0143] The linear spectrum corresponding to the training recording is input into the posterior encoder to obtain the second training mean and the second training variance of the second preset size. The second training mean and the second training variance are used to shape the corresponding second training Gaussian distribution model, and the second training Gaussian distribution model is randomly sampled to obtain the training random sampling result z. The training random sampling result z needs to be output in two ways. The first way is input into the decoder to obtain the training speech waveform, and the error is calculated with the original real training recording to obtain the generation loss; the other way is output to the stream module, and the training vector f is obtained after processing. θ (z).

[0144] For the training vector f θ (z) and the first training Gaussian distribution model (including the first training mean μ θ and the first training variance σ θ ) to perform time alignment, obtain the alignment result d and the relative entropy between the training vector and the first training Gaussian distribution model. Input the training matrix into the random duration predictor to obtain the training copy parameter, and compare the alignment result d with the training copy parameter to obtain the duration loss. Finally, the speech synthesis model is updated using the loss function value obtained by summarizing the generation loss, duration loss, and relative entropy, thereby obtaining an updated speech synthesis model.

[0145] The embodiments of the present application train and update the speech synthesis model so that the speech synthesis model can learn rich speech features and language expressions, which helps to improve the generalization ability of the speech synthesis model and significantly improve the performance of the speech synthesis model, so that it can generate more natural, fluent and high-quality speech output.

[0146] In an optional embodiment of the present application, the stream module includes the multi-phase orthogonal mirror analysis filter, the coupling layer, and the multi-phase orthogonal mirror synthesis filter, and the training random sampling result is input into the decoder and the stream module respectively to obtain the generation loss and the training vector, including:

[0147] After dimensionally regularizing the training random sampling results, the dimensionally regularized training random sampling results are input into the multi-phase orthogonal mirror analysis filter to obtain a training analysis signal;

[0148] Inputting the training analysis signal into the coupling layer to perform forward transformation on the training analysis signal to obtain a forward transformation signal;

[0149] Inputting the forward transformed signal into the multi-phase orthogonal mirror synthesis filter to obtain a training synthesis signal;

[0150] The training synthetic signal is dimensionally inversely regularized to obtain the training vector.

[0151] In this embodiment, the stream module of the speech synthesis model actually includes a multi-phase orthogonal mirror analysis filter, a coupling layer and a multi-phase orthogonal mirror synthesis filter, that is, the normalized stream introduced with the PQMF algorithm provided in the embodiment of the present application.

[0152] In this embodiment, when training the speech synthesis model, after the training random sampling result is input into the stream module, the training random sampling result is first dimensionally regularized. When dimensionally regularizing the training random sampling result, the training random sampling result can be divided into two sub-vectors according to the median of the length of the training random sampling result, and then the elements of the two sub-vectors are cross-combined one by one to obtain the dimensionally regularized training random sampling result. For example, if the training random sampling result x is a D-dimensional [x 1 ,x 2 ,…,x D ], then divide x into two equal parts of D / 2 dimensions, that is, [x 1 ,x 2 ,…,x D / 2 ] and [x D / 2+1 ,x D / 2+2 ,…,x D ], and then cross the elements one by one to get [x 1 ,x D / 2+1 ,x 2 ,x D / 2+2 ,…,x D / 2 ,x D ], thereby completing the dimension regularization of the training random sampling results.

[0153] Then, the dimensionally regularized training random sampling result is input into the multi-phase orthogonal mirror analysis filter to obtain the training analysis signal. For example, the dimensionally regularized training random sampling result [x 1 ,x D / 2+1 ,x 2 ,x D / 2+2 ,…,x D / 2 ,xD ] Input the multi-phase orthogonal mirror analysis filter to obtain two training analysis signals with a length of D / 2.

[0154] The training analysis signal is input into the coupling layer in the speech synthesis model to perform forward transformation on the training analysis signal to obtain a forward transformation signal. For example, the two training analysis signals of length D / 2 obtained above are forward input into the coupling layer to output two forward transformation signals of length D / 2.

[0155] The forward transform signal is input into a multi-phase orthogonal mirror synthesis filter to obtain a training synthesis signal. In an optional embodiment, the process of obtaining the training synthesis signal can be expressed as follows: after upsampling the forward transform signal by a preset multiple, the upsampled forward transform signal is obtained, and the upsampled forward transform signal is input into a multi-phase orthogonal mirror synthesis filter to obtain a preliminary training synthesis signal, and all the preliminary training synthesis signals are added to obtain the training synthesis signal. For example, two forward transform signals of length D / 2 are upsampled by 2 times respectively, and then input into a multi-phase orthogonal mirror synthesis filter to obtain two preliminary training synthesis signals of length D, and the two are added to obtain a training synthesis signal of length D.

[0156] After the polyphase orthogonal mirror synthesis filter is input to obtain the training synthetic signal, the training synthetic signal needs to be dimensionally denormalized. Specifically, when performing dimension denormalization, the element arrangement order of the training synthetic signal is restored according to the element arrangement order of the training random sampling result, that is, according to the original element arrangement order, to obtain the training vector. In this example, the training synthetic signal x is [z 1 ,z D / 2+1 ,z 2 ,z D / 2+2 ,…,z D / 2 ,z D ], after dimensional inverse normalization, it is restored to [z 1 ,z 2 ,…,z D ], and then align the training vector with the first training Gaussian distribution model in time.

[0157] Reference Figure 4 , is a schematic diagram of the flow module structure of an embodiment of a speech synthesis method of the present application.

[0158] The coupled layer structure can realize the forward transformation from x to z and the inverse transformation from z to x, and the forward transformation can be used in the training stage of the speech synthesis model, and the inverse transformation can be used in the speech generation stage of the speech synthesis model.

[0159] As for the stream module provided in the embodiment of the present application, it adds pre-processing and post-processing outside the original coupling layer. The specific forward transformation process is: when training the speech synthesis model, the training random sampling result x is input into the multi-phase orthogonal mirror analysis filter (i.e., PQMF analysis) after dimension regularization, and then input into the multi-phase orthogonal mirror synthesis filter (i.e., PQMF synthesis) after forward transformation of the coupling layer, and finally, it is inversely dimensionally regularized to obtain the training vector z.

[0160] The specific inverse transformation process of the stream module provided in the embodiment of the present application is as follows: when using a speech synthesis model to synthesize speech, the intermediate vector z is input into a multi-phase orthogonal mirror analysis filter (i.e., PQMF analysis) after dimension normalization, and then input into a multi-phase orthogonal mirror synthesis filter (i.e., PQMF synthesis) after inverse transformation through the coupling layer, and finally, it is dimensionally inversely normalized to obtain a dimensionally inversely normalized synthetic signal x.

[0161] The stream module of the speech synthesis model provided in the embodiment of the present application introduces the PQMF algorithm, so that all dimensions of the training random sampling results can participate in the processing operations within the coupling layer, thereby helping to make the dimensions of the training vectors more uniform and consistent, significantly improving the overall stability and robustness of the speech synthesis model, and further improving the stability of the speech quality synthesized using the normalized stream technology.

[0162] In an optional embodiment of the present application, the step of inputting the training random sampling result into the decoder and the stream module respectively to obtain a generation loss and a training vector includes:

[0163] Inputting the training random sampling result into the decoder and outputting a training speech waveform;

[0164] The error between the training speech waveform and the training recording is calculated to obtain the generation loss.

[0165] In the embodiment of the present application, the training random sampling result is input into the decoder, and a training speech waveform can be output, which is the speech generated by the speech synthesis model. The error between the training speech waveform and the real training recording is further calculated to obtain the generation loss, which is used to subsequently update the speech synthesis model.

[0166] The embodiment of the present application inputs the training random sampling results into the decoder to generate a training speech waveform and a corresponding generation loss. The generation loss, as one of the loss function values, can guide the updating of the speech synthesis model parameters, help the speech synthesis model to gradually approach the characteristics of the real speech during the training process, and improve the naturalness and comprehensibility of the synthesized speech.

[0167] In an embodiment of the present application, a text of speech to be synthesized input by a user is received; the phoneme sequence corresponding to the text of the speech to be synthesized is input into the preprocessing part of the speech synthesis model to obtain an intermediate vector; after the intermediate vector is dimensionally regularized, the dimensionally regularized intermediate vector is input into a multi-phase orthogonal mirror analysis filter to obtain an analysis signal; the analysis signal is input into a coupling layer in the speech synthesis model to inversely transform the analysis signal to obtain an inverse transformation signal; the inverse transformation signal is input into a multi-phase orthogonal mirror synthesis filter to obtain a synthesis signal; after the synthesis signal is dimensionally inversely regularized, the dimensionally inversely regularized synthesis signal is input into a decoder in the speech synthesis model to output a speech waveform; according to the speech waveform, a target speech corresponding to the text of the speech to be synthesized is synthesized. In an embodiment of the present application, a fixed pattern connection is established between the dimensions of the intermediate vector before the coupling layer is processed by introducing a multi-phase orthogonal mirror filter in the coupling layer of the normalized flow. This approach enables all dimensions of the intermediate vector to participate in the processing operations within the coupling layer, thereby helping to make the dimensions of the vector more uniform and consistent. In this way, the overall stability and robustness of the speech synthesis model can be significantly improved, thereby improving the stability of the speech quality synthesized using the normalized flow technology.

[0168] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0169] On the basis of the above embodiments, this embodiment further provides a speech synthesis device, which is applied to electronic devices such as terminal devices and servers.

[0170] Reference Figure 5 , shows a structural block diagram of an embodiment of a speech synthesis device of the present application, which may specifically include the following modules:

[0171] The text input module 501 is used to receive the text to be synthesized into speech input by the user;

[0172] A preprocessing module 502, used for inputting the phoneme sequence corresponding to the text of the speech to be synthesized into the preprocessing part of the speech synthesis model to obtain an intermediate vector;

[0173] An analysis signal acquisition module 503 is used to input the dimensionally regularized intermediate vector into a multi-phase orthogonal mirror analysis filter to obtain an analysis signal after dimension regularization of the intermediate vector;

[0174] An inverse transformation module 504, configured to input the analysis signal into a coupling layer in the speech synthesis model to perform an inverse transformation on the analysis signal to obtain an inverse transformation signal;

[0175] A synthetic signal acquisition module 505 is used to input the inverse transformation signal into a multi-phase orthogonal mirror synthesis filter to obtain a synthetic signal;

[0176] A dimensionality deregulation module 506 is used for inputting the synthesized signal after dimensionality deregulation into a decoder in the speech synthesis model to output a speech waveform;

[0177] The speech synthesis module 507 is used to synthesize a target speech corresponding to the text of the speech to be synthesized according to the speech waveform.

[0178] Optionally, the analysis signal acquisition module 503 further includes:

[0179] A vector division submodule, used for dividing the intermediate vector into two sub-vectors according to the median value of the length of the intermediate vector;

[0180] A dimension regularization submodule, used for cross-combining the elements of the two sub-vectors one by one to obtain the intermediate vector after dimension regularization;

[0181] The dimension deregulation module 506 further includes:

[0182] The vector restoration submodule is used to restore the element arrangement order of the synthetic signal according to the element arrangement order of the intermediate vector to obtain the synthetic signal after the dimension is inversely regularized.

[0183] Optionally, the synthetic signal acquisition module 505 is further used to:

[0184] After upsampling the inverse transformation signal by a preset multiple, an upsampled inverse transformation signal is obtained;

[0185] Inputting the up-sampled inverse transform signal into the multi-phase orthogonal mirror synthesis filter to obtain a preliminary synthesized signal;

[0186] The preliminary composite signals are added to obtain the composite signal.

[0187] Optionally, the preprocessing part in the speech synthesis model includes at least a text encoder, a projection layer and a random duration predictor, and the preprocessing module 502 includes:

[0188] A text encoder submodule, used for inputting the phoneme sequence corresponding to the text of the speech to be synthesized into the text encoder to obtain a matrix of a preset size;

[0189] A projection layer submodule, used for inputting the matrix into the projection layer to obtain a mean and a variance, wherein the mean and the variance are used to shape a corresponding Gaussian distribution model;

[0190] A Gaussian distribution sampling submodule, used to perform random sampling on the Gaussian distribution model to obtain a random sampling result;

[0191] The time replication submodule is used to perform a time replication operation on the random sampling result according to a replication parameter to obtain the intermediate vector, and the replication parameter is obtained by inputting the matrix into the random duration predictor.

[0192] Optionally, the speech synthesis model further includes at least a posterior encoder and a stream module, and the device further includes:

[0193] A sample collection module, used to collect training recordings and corresponding training texts;

[0194] A text encoder input module, used for inputting the phoneme sequence corresponding to the training text into the text encoder to obtain a training matrix of a preset size;

[0195] A projection layer input module, used for inputting the training matrix into the projection layer to obtain a first training mean and a first training variance of a first preset size, wherein the first training mean and the first training variance are used for shaping a corresponding first training Gaussian distribution model;

[0196] a posterior encoder input module, used for inputting the linear spectrum corresponding to the training recording into the posterior encoder to obtain a second training mean and a second training variance of a second preset size, wherein the second training mean and the second training variance are used to shape a corresponding second training Gaussian distribution model;

[0197] A training random sampling module, used for performing random sampling on the second training Gaussian distribution model to obtain a training random sampling result;

[0198] A normalized stream processing module, used for inputting the training random sampling results into the decoder and the stream module respectively, to obtain a generation loss and a training vector;

[0199] A timing alignment module, used to perform timing alignment on the training vector and the first training Gaussian distribution model to obtain an alignment result and a relative entropy between the training vector and the first training Gaussian distribution model;

[0200] A random duration predictor input module, used for comparing the alignment result with a training copy parameter to obtain a duration loss, wherein the training copy parameter is obtained by inputting the training matrix into the random duration predictor;

[0201] A loss function module, used to summarize the generation loss, the duration loss and the relative entropy to obtain a loss function value;

[0202] A model updating module is used to update the speech synthesis model using the loss function value.

[0203] Optionally, the stream module includes the multi-phase orthogonal mirror analysis filter, the coupling layer and the multi-phase orthogonal mirror synthesis filter, and the normalized stream processing module includes:

[0204] A forward analysis submodule, configured to input the dimensionally regularized training random sampling results into the multi-phase orthogonal mirror analysis filter after dimensionally regularizing the training random sampling results to obtain a training analysis signal;

[0205] A forward transformation submodule, used for inputting the training analysis signal into the coupling layer to perform a forward transformation on the training analysis signal to obtain a forward transformation signal;

[0206] A forward synthesis submodule, used for inputting the forward transformation signal into the multi-phase orthogonal mirror synthesis filter to obtain a training synthesis signal;

[0207] The forward dimensionality dewarping submodule is used to perform dimensionality dewarping on the training synthetic signal to obtain the training vector.

[0208] Optionally, the normalized stream processing module is further used to:

[0209] Inputting the training random sampling result into the decoder and outputting a training speech waveform;

[0210] The error between the training speech waveform and the training recording is calculated to obtain the generation loss.

[0211] In an embodiment of the present application, a text of speech to be synthesized input by a user is received; the phoneme sequence corresponding to the text of the speech to be synthesized is input into the preprocessing part of the speech synthesis model to obtain an intermediate vector; after the intermediate vector is dimensionally regularized, the dimensionally regularized intermediate vector is input into a multi-phase orthogonal mirror analysis filter to obtain an analysis signal; the analysis signal is input into a coupling layer in the speech synthesis model to inversely transform the analysis signal to obtain an inverse transformation signal; the inverse transformation signal is input into a multi-phase orthogonal mirror synthesis filter to obtain a synthesis signal; after the synthesis signal is dimensionally inversely regularized, the dimensionally inversely regularized synthesis signal is input into a decoder in the speech synthesis model to output a speech waveform; according to the speech waveform, a target speech corresponding to the text of the speech to be synthesized is synthesized. In an embodiment of the present application, a fixed pattern connection is established between the dimensions of the intermediate vector before the coupling layer is processed by introducing a multi-phase orthogonal mirror filter in the coupling layer of the normalized flow. This approach enables all dimensions of the intermediate vector to participate in the processing operations within the coupling layer, thereby helping to make the dimensions of the vector more uniform and consistent. In this way, the overall stability and robustness of the speech synthesis model can be significantly improved, thereby improving the stability of the speech quality synthesized using the normalized flow technology.

[0212] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0213] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0214] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Figure 6 An exemplary apparatus 600 that may be used to implement various embodiments described in this application is schematically illustrated.

[0215] For one embodiment, Figure 6An exemplary apparatus 600 is shown having one or more processors 602, a control module (chip set) 604 coupled to at least one of the processor(s) 602, a memory 606 coupled to the control module 604, a non-volatile memory (NVM) / storage device 608 coupled to the control module 604, one or more input / output devices 610 coupled to the control module 604, and a network interface 612 coupled to the control module 604.

[0216] The processor 602 may include one or more single-core or multi-core processors, and the processor 602 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 600 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0217] In some embodiments, the apparatus 600 may include one or more computer-readable media (e.g., memory 606 or NVM / storage device 608) having instructions 614 and one or more processors 602 combined with the one or more computer-readable media and configured to execute the instructions 614 to implement a module to perform the actions described in the present disclosure.

[0218] For one embodiment, control module 604 may include any suitable interface controller to provide any suitable interface to at least one of processor(s) 602 and / or any suitable device or component in communication with control module 604 .

[0219] The control module 604 may include a memory controller module to provide an interface to the memory 606. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0220] The memory 606 may be used, for example, to load and store data and / or instructions 614 for the device 600. For one embodiment, the memory 606 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 606 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0221] For one embodiment, control module 604 may include one or more input / output controllers to provide an interface to NVM / storage device 608 and input / output device(s) 610 .

[0222] For example, NVM / storage 608 may be used to store data and / or instructions 614. NVM / storage 608 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0223] NVM / storage device 608 may include storage resources that are physically part of the device on which apparatus 600 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 608 may be accessed via input / output device(s) 610 over a network.

[0224] (One or more) input / output devices 610 may provide an interface for the apparatus 600 to communicate with any other appropriate device, and the input / output device 610 may include a communication component, an audio component, a sensor component, etc. The network interface 612 may provide an interface for the apparatus 600 to communicate through one or more networks, and the apparatus 600 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0225] For one embodiment, at least one of the processor(s) 602 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 604. For one embodiment, at least one of the processor(s) 602 may be packaged together with the logic of one or more controllers of the control module 604 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 602 may be integrated on the same die with the logic of one or more controllers of the control module 604. For one embodiment, at least one of the processor(s) 602 may be integrated on the same die with the logic of one or more controllers of the control module 604 to form a system-on-chip (SoC).

[0226] In various embodiments, the device 600 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 600 may have more or fewer components and / or different architectures. For example, in some embodiments, the device 600 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0227] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0228] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0229] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0230] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable speech synthesis terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable speech synthesis terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0231] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable speech synthesis terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0232] These computer program instructions may also be loaded onto a computer or other programmable speech synthesis terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0233] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0234] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0235] The above is a detailed introduction to a speech synthesis method and device, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Receive text of speech to be synthesized input by a user; Inputting the phoneme sequence corresponding to the text of the speech to be synthesized into the preprocessing part of the speech synthesis model to obtain an intermediate vector; After dimensionally regularizing the intermediate vector, the dimensionally regularized intermediate vector is input into a multi-phase orthogonal mirror analysis filter to obtain an analysis signal; Inputting the analysis signal into a coupling layer in the speech synthesis model to inversely transform the analysis signal to obtain an inverse transformation signal; Inputting the inverse transformed signal into a polyphase quadrature mirror synthesis filter to obtain a synthesized signal; After performing dimension denormalization on the synthesized signal, the dimension denormalized synthesized signal is input into a decoder in the speech synthesis model to output a speech waveform; A target speech corresponding to the text of the speech to be synthesized is synthesized according to the speech waveform.

2. The method according to claim 1, characterized in that The dimension regularization of the intermediate vector includes: Dividing the intermediate vector into two sub-vectors according to a median value of the length of the intermediate vector; Cross-combining the elements of the two sub-vectors one by one to obtain the intermediate vector after the dimension is regularized; The step of performing dimensionality dewarping on the synthesized signal comprises: According to the element arrangement order of the intermediate vector, the element arrangement order of the synthetic signal is restored to obtain the synthetic signal after the dimension is inversely regularized.

3. The method according to claim 2, characterized in that The step of inputting the inverse transformed signal into a multi-phase orthogonal mirror synthesis filter to obtain a synthesized signal comprises: After upsampling the inverse transformation signal by a preset multiple, an upsampled inverse transformation signal is obtained; Inputting the up-sampled inverse transform signal into the multi-phase orthogonal mirror synthesis filter to obtain a preliminary synthesized signal; The preliminary composite signals are added to obtain the composite signal.

4. The method according to claim 1, characterized in that: The preprocessing part in the speech synthesis model at least includes a text encoder, a projection layer and a random duration predictor, and the phoneme sequence corresponding to the text of the speech to be synthesized is input into the preprocessing part in the speech synthesis model to obtain an intermediate vector, including: Inputting the phoneme sequence corresponding to the text of the speech to be synthesized into the text encoder to obtain a matrix of a preset size; Input the matrix into the projection layer to obtain a mean and a variance, wherein the mean and the variance are used to shape a corresponding Gaussian distribution model; Performing random sampling on the Gaussian distribution model to obtain a random sampling result; According to a replication parameter, a time replication operation is performed on the random sampling result to obtain the intermediate vector, and the replication parameter is obtained by inputting the matrix into the random duration predictor.

5. The method according to claim 4, characterized in that The speech synthesis model at least includes a posterior encoder and a stream module. Before receiving the text of the speech to be synthesized input by the user, the method further includes: Collect training recordings and corresponding training texts; Inputting the phoneme sequence corresponding to the training text into the text encoder to obtain a training matrix of a preset size; Inputting the training matrix into the projection layer to obtain a first training mean and a first training variance of a first preset size, wherein the first training mean and the first training variance are used to shape a corresponding first training Gaussian distribution model; Inputting the linear spectrum corresponding to the training recording into the posterior encoder to obtain a second training mean and a second training variance of a second preset size, wherein the second training mean and the second training variance are used to shape a corresponding second training Gaussian distribution model; Performing random sampling on the second training Gaussian distribution model to obtain a training random sampling result; Input the training random sampling results into the decoder and the stream module respectively to obtain the generation loss and the training vector; Performing time-series alignment on the training vector and the first training Gaussian distribution model to obtain an alignment result and a relative entropy between the training vector and the first training Gaussian distribution model; Comparing the alignment result with a training replication parameter to obtain a duration loss, wherein the training replication parameter is obtained by inputting the training matrix into the random duration predictor; Summarizing the generation loss, the duration loss, and the relative entropy to obtain a loss function value; The speech synthesis model is updated using the loss function value.

6. The method according to claim 5, characterized in that The stream module includes the polyphase orthogonal mirror analysis filter, the coupling layer and the polyphase orthogonal mirror synthesis filter, and the training random sampling result is input into the decoder and the stream module respectively to obtain the generation loss and the training vector, including: After dimensionally regularizing the training random sampling results, the dimensionally regularized training random sampling results are input into the multi-phase orthogonal mirror analysis filter to obtain a training analysis signal; Inputting the training analysis signal into the coupling layer to perform forward transformation on the training analysis signal to obtain a forward transformation signal; Inputting the forward transformed signal into the multi-phase orthogonal mirror synthesis filter to obtain a training synthesis signal; The training synthetic signal is dimensionally inversely regularized to obtain the training vector.

7. The method according to claim 5, characterized in that The step of inputting the training random sampling results into the decoder and the stream module respectively to obtain a generated loss and a training vector comprises: Inputting the training random sampling result into the decoder and outputting a training speech waveform; The error between the training speech waveform and the training recording is calculated to obtain the generation loss.

8. A speech synthesis device, characterized in that: The device comprises: A text input module, used to receive text to be synthesized speech input by a user; A preprocessing module, used for inputting the phoneme sequence corresponding to the text of the speech to be synthesized into the preprocessing part of the speech synthesis model to obtain an intermediate vector; An analysis signal acquisition module is used to input the dimensionally regularized intermediate vector into a multi-phase orthogonal mirror analysis filter to obtain an analysis signal after dimensionally regularizing the intermediate vector; An inverse transformation module, used for inputting the analysis signal into a coupling layer in the speech synthesis model to perform an inverse transformation on the analysis signal to obtain an inverse transformation signal; A synthetic signal acquisition module, used for inputting the inverse transformation signal into a multi-phase orthogonal mirror synthesis filter to obtain a synthetic signal; A dimensionality deregulation module, configured to, after performing dimensionality deregulation on the synthesized signal, input the synthesized signal after dimensionality deregulation into a decoder in the speech synthesis model, and output a speech waveform; The speech synthesis module is used to synthesize a target speech corresponding to the text of the speech to be synthesized according to the speech waveform.

9. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to execute the speech synthesis method as described in one or more of claims 1-7.

10. One or more machine-readable media having executable codes stored thereon, which, when executed, cause a processor to perform the speech synthesis method according to one or more of claims 1-7.

Images