A speech synthesis method, apparatus, device, and storage medium

Abstract

This application provides a speech synthesis method, apparatus, device, and storage medium. The method involves receiving text input by a user as speech to be synthesized; inputting the phoneme sequence corresponding to the text into a preprocessing section of a speech synthesis model to obtain an intermediate vector; inputting the dimension-normalized intermediate vector into a polyphase orthogonal mirror analysis filter to obtain an analysis signal; inputting the analysis signal into a coupling layer of the speech synthesis model to perform an inverse transformation on the analysis signal to obtain an inverse transform signal; inputting the inverse transform signal into a polyphase orthogonal mirror synthesis filter to obtain a synthesized signal; and inputting the dimension-inverse-normalized synthesized signal into a decoder in the speech synthesis model to synthesize the target speech based on the speech waveform. This application, by introducing a polyphase orthogonal mirror filter into the coupling layer of the normalized flow, can significantly improve the overall stability and robustness of the speech synthesis model, thereby enhancing the stability of the speech quality synthesized using normalized flow technology.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, and in particular to a speech synthesis method and apparatus, an electronic device, and a storage medium. Background Technology

[0002] In the field of speech generation, using the VITS system (Variational Inference with Transformer-based Stochastic Speech Generation, a speech synthesis technique based on variational inference and the Transformer architecture) to synthesize speech is a classic example of applying normalized flow technology. The core structure of the flow module in the VITS system is a normalized flow. A normalized flow typically consists of multiple coupled layers connected in series. These coupled layers can perform the transformation of a signal from x to z, and the inverse transformation from z to x. Specifically, if both signals x and z have a dimension of D, then by pre-setting a constant d (d < D), the D-dimensional signal can be divided into two parts. In the coupled layer transformation, the upper part remains unchanged, while the lower part undergoes an affine transformation. The coefficients and biases of the affine transformation are calculated from the upper part.

[0003] Therefore, with each coupling layer transformation, only a portion of the dimensions change, while the rest remain unchanged. To address this issue, the current VITS system introduces a dimension flipping operation after each coupling layer transformation, swapping the changed and unchanged parts before proceeding to the next coupling layer. In this way, even-numbered coupling layer transformations in series ensure that both dimensional components undergo approximately the same number of transformations.

[0004] However, this method of alternating between two dimensions is not uniform enough. To illustrate, it's like hopping one step with your left foot, then one step with your right foot—the experience is different from hopping with both feet simultaneously. This alternating approach may lead to insufficient stability in the speech synthesis model built by the VITS system, thus affecting the stability of the generated speech quality. Summary of the Invention

[0005] This application provides a speech synthesis method to address the problem of poor speech quality stability when using normalized flow technology in the prior art.

[0006] Accordingly, embodiments of this application also provide a speech synthesis device, an electronic device, and a storage medium to ensure the implementation and application of the above methods.

[0007] To address the aforementioned problems, this application discloses a speech synthesis method, the method comprising:

[0008] Receives text input from the user to be synthesized speech;

[0009] The phoneme sequence corresponding to the text of the speech to be synthesized is input into the preprocessing part of the speech synthesis model to obtain an intermediate vector;

[0010] After dimension normalization of the intermediate vector, the dimension normalized intermediate vector is input into a multiphase orthogonal mirror analysis filter to obtain the analysis signal;

[0011] The analyzed signal is input into the coupling layer of the speech synthesis model to perform an inverse transformation on the analyzed signal to obtain an inverse transformed signal;

[0012] The inverse-transformed signal is input into a multiphase orthogonal mirror synthesis filter to obtain a synthesized signal;

[0013] After performing inverse dimension normalization on the synthesized signal, the inverse dimension normalized synthesized signal is input into the decoder in the speech synthesis model to output a speech waveform;

[0014] Based on the speech waveform, target speech corresponding to the text of the speech to be synthesized is synthesized.

[0015] Optionally, the dimension normalization of the intermediate vector includes:

[0016] Based on the median length of the intermediate vector, the intermediate vector is divided into two sub-vectors;

[0017] The elements of the two sub-vectors are combined one by one to obtain the intermediate vector after the dimension is normalized;

[0018] The step of performing inverse dimensional normalization on the synthesized signal includes:

[0019] The element order of the synthesized signal is restored according to the element order of the intermediate vector to obtain the synthesized signal after inverse dimension normalization.

[0020] Optionally, the step of inputting the inverse transform signal into a polyphase quadrature mirror synthesis filter to obtain a synthesized signal includes:

[0021] After upsampling the inverse transform signal by a preset multiple, the upsampled inverse transform signal is obtained;

[0022] The upsampled inverse transform signal is input into the multiphase quadrature mirror synthesis filter to obtain a preliminary synthesized signal;

[0023] The preliminary synthesized signals are added together to obtain the synthesized signal.

[0024] Optionally, the preprocessing part of the speech synthesis model includes at least a text encoder, a projection layer, and a random duration predictor. The step of inputting the phoneme sequence corresponding to the text of the speech to be synthesized into the preprocessing part of the speech synthesis model to obtain an intermediate vector includes:

[0025] The phoneme sequence corresponding to the text to be synthesized is input into the text encoder to obtain a matrix of a preset size;

[0026] The matrix is ​​input into the projection layer to obtain the mean and variance, which are used to shape the corresponding Gaussian distribution model.

[0027] Random sampling is performed on the Gaussian distribution model to obtain random sampling results;

[0028] Based on the replication parameters, a time-based replication operation is performed on the random sampling results to obtain the intermediate vector. The replication parameters are obtained by inputting the matrix into the random duration predictor.

[0029] Optionally, the speech synthesis model further includes at least a posterior encoder and a streaming module, and before receiving the text of the speech to be synthesized from user input, the method further includes:

[0030] Collect training recordings and corresponding training texts;

[0031] The phoneme sequence corresponding to the training text is input into the text encoder to obtain a training matrix of a preset size;

[0032] The training matrix is ​​input into the projection layer to obtain a first training mean and a first training variance of a first preset size. The first training mean and the first training variance are used to shape the corresponding first training Gaussian distribution model.

[0033] The linear spectrum corresponding to the training recording is input into the posterior encoder to obtain a second training mean and a second training variance of a second preset size. The second training mean and the second training variance are used to shape the corresponding second training Gaussian distribution model.

[0034] Random sampling is performed on the second trained Gaussian distribution model to obtain the training random sampling results;

[0035] The training random sampling results are input into the decoder and the streaming module respectively to obtain the generation loss and training vector;

[0036] The training vector is time-aligned with the first training Gaussian distribution model to obtain the alignment result and the relative entropy between the training vector and the first training Gaussian distribution model;

[0037] The alignment result is compared with the training replication parameters to obtain the duration loss, wherein the training replication parameters are obtained by inputting the training matrix into the random duration predictor;

[0038] By summing the generation loss, the duration loss, and the relative entropy, the loss function value is obtained.

[0039] The speech synthesis model is updated using the loss function value.

[0040] Optionally, the streaming module includes the multiphase orthogonal mirror analysis filter, the coupling layer, and the multiphase orthogonal mirror synthesis filter. The step of inputting the training random sampling results to the decoder and the streaming module respectively to obtain the generation loss and training vector includes:

[0041] After dimension normalizing the training random sampling results, the dimension normalized training random sampling results are input into the multiphase orthogonal mirror analysis filter to obtain the training analysis signal;

[0042] The training analysis signal is input into the coupling layer to perform a forward transformation on the training analysis signal to obtain a forward transformed signal;

[0043] The positive transform signal is input into the multiphase orthogonal mirror synthesis filter to obtain the training synthesized signal;

[0044] The training synthetic signal is subjected to dimensionality inverse normalization to obtain the training vector.

[0045] Optionally, the step of inputting the training random sampling results into the decoder and the streaming module respectively to obtain the generation loss and training vector includes:

[0046] The training random sampling results are input into the decoder, and the training speech waveform is output.

[0047] The error between the training speech waveform and the training recording is calculated to obtain the generation loss.

[0048] This application also discloses a speech synthesis device, the device comprising:

[0049] The text input module is used to receive the text input by the user to be synthesized speech;

[0050] The preprocessing module is used to input the phoneme sequence corresponding to the text of the speech to be synthesized into the preprocessing part of the speech synthesis model to obtain an intermediate vector;

[0051] The analysis signal acquisition module is used to input the dimension-normalized intermediate vector into a multiphase orthogonal mirror analysis filter after dimension normalization of the intermediate vector to obtain the analysis signal.

[0052] The inverse transform module is used to input the analyzed signal into the coupling layer in the speech synthesis model to perform an inverse transform on the analyzed signal to obtain an inverse transform signal;

[0053] The synthesized signal acquisition module is used to input the inverse transformed signal into a multiphase orthogonal mirror synthesis filter to obtain a synthesized signal;

[0054] The dimension inverse normalization module is used to input the dimension inverse normalized synthesized signal into the decoder in the speech synthesis model after performing dimension inverse normalization on the synthesized signal, and output the speech waveform.

[0055] The speech synthesis module is used to synthesize target speech corresponding to the text of the speech to be synthesized based on the speech waveform.

[0056] This application also discloses an electronic device, including: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform one or more speech synthesis methods as described in the embodiments of this application.

[0057] This application also discloses one or more machine-readable media storing executable code thereon, which, when executed, causes a processor to perform one or more of the speech synthesis methods described in this application.

[0058] Compared with the prior art, the embodiments of this application have the following advantages:

[0059] In this embodiment, the system receives text input by the user as the speech to be synthesized; the phoneme sequence corresponding to the text is input into the preprocessing part of the speech synthesis model to obtain an intermediate vector; after dimension normalization of the intermediate vector, it is input into a polyphase quadrature mirror filter to obtain an analysis signal; the analysis signal is input into the coupling layer of the speech synthesis model to perform an inverse transformation to obtain an inverse transform signal; the inverse transform signal is input into a polyphase quadrature mirror synthesis filter to obtain a synthesized signal; after inverse dimension normalization of the synthesized signal, it is input into the decoder of the speech synthesis model to output a speech waveform; based on the speech waveform, the target speech corresponding to the text of the speech to be synthesized is synthesized. This embodiment introduces a polyphase quadrature mirror filter (PQMF) into the coupling layer of the normalized flow, establishing a fixed pattern relationship between the dimensions of the intermediate vector before the coupling layer processing. This approach allows all dimensions of the intermediate vector to participate in the processing operations within the coupling layer, thereby helping to make the dimensions of the vector more uniform and consistent. In this way, the overall stability and robustness of the speech synthesis model can be significantly improved, thereby enhancing the stability of the speech quality synthesized using normalized flow technology. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the coupling layer of an embodiment of the speech synthesis method of this application;

[0061] Figure 2 This is a flowchart illustrating the steps of an embodiment of a speech synthesis method according to this application;

[0062] Figure 3 This is a schematic diagram of the speech synthesis model structure of an embodiment of the speech synthesis method of this application;

[0063] Figure 4 This is a schematic diagram of the stream module structure of an embodiment of the speech synthesis method of this application;

[0064] Figure 5 This is a structural block diagram of an embodiment of a speech synthesis device according to this application;

[0065] Figure 6 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0066] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] Reference Figure 1This is a schematic diagram of the coupling layer of an embodiment of a speech synthesis method of this application.

[0068] In the field of speech generation, using the VITS system to synthesize speech is a classic example of applying normalized stream technology. The core structure of the stream module in the VITS system is a normalized stream. A normalized stream typically consists of multiple coupled layers connected in series, which can perform the transformation of the signal from x to z, and the inverse transformation from z to x. For example... Figure 1 This illustrates the specific operation of the transformation from x to z in a single coupled layer within a normalized flow.

[0069] Specifically, if both signals x and z have a dimension of D, then by pre-setting a constant d (d < D), the D-dimensional signal can be divided into two parts: the upper part consisting of x1, ..., xz. d and the lower half of x d+1 ... x D In the coupling layer transformation, the upper part remains unchanged and can be directly derived from x1, ..., x d Copying yields z1, ..., z d The lower part performs an affine transformation, with the coefficients and biases calculated from the upper part. For example... Figure 1 As shown, β d+1 ..., β D The coefficients of the affine transformation are calculated by the neural network submodule F; γ d+1 ..., γ D The bias for the affine transformation is calculated using the neural network submodule H. d+1 ... x D z can be obtained after affine transformation. d+1 ... z D The inverse transformation from z to x is that the upper part remains unchanged, and the transformation is directly performed by z1, ..., z... d Copying yields x1, ..., x d ;z d+1 ... z D x can be obtained through inverse affine transformation. d+1 ... x D .

[0070] Therefore, with each coupling layer transformation, only a portion of the dimensions change, while the rest remain unchanged. To address this issue, the current VITS system introduces a dimension flipping operation after each coupling layer transformation, swapping the changed and unchanged parts before proceeding to the next coupling layer. In this way, even-numbered coupling layer transformations in series ensure that both dimensional components undergo approximately the same number of transformations.

[0071] However, this method of alternating transformation of two - part dimensions is not uniform enough. For example, it is like taking a step with the left foot and then a step with the right foot, which is different from the experience of taking a step with both feet simultaneously. This way of alternating transformation may lead to insufficient stability of the speech synthesis model constructed by the VITS system, thereby affecting the quality stability of the generated speech.

[0072] Therefore, the speech synthesis method provided in the embodiments of this application introduces the PQMF algorithm and pre - establishes a fixed pattern connection between the dimensions of the intermediate vector before the coupling layer processing. This approach enables all dimensions of the intermediate vector to participate in the "two - path variable / constant" processing operation within the coupling layer.

[0073] Refer to Figure 2 , which is a flowchart of the steps of an embodiment of the speech synthesis method of this application, including the following steps:

[0074] Step 201: Receive the text of the speech to be synthesized input by the user.

[0075] In step 201, the text of the speech to be synthesized input by the user is received. In an optional embodiment, for the text of the speech to be synthesized input by the user, a third - party phonetic annotation tool (such as pypinyin) is required to translate the text into the phoneme sequence corresponding to the text of the speech to be synthesized. For example, if the text input by the user is "good person", the phoneme sequence corresponding to the text of the speech to be synthesized is "h3 ao3r2 en2".

[0076] Step 202: Input the phoneme sequence corresponding to the text of the speech to be synthesized into the pre - processing part of the speech synthesis model to obtain an intermediate vector.

[0077] In step 202, the phoneme sequence corresponding to the text of the speech to be synthesized is input into the speech synthesis model. The input phoneme sequence first needs to pass through the pre - processing part of the speech synthesis model, and then the intermediate vector is obtained through pre - processing.

[0078] In a feasible embodiment, the pre - processing part at least includes a text encoder, a projection layer, and a random duration predictor. First, the phoneme sequence corresponding to the text of the speech to be synthesized is input into the text encoder to obtain a matrix of a preset size. If the input phoneme sequence is "h3 ao3 r2 en2", the size of the obtained matrix is 4×D, where D is a preset internal dimension. For example, D can be 192. Then, this matrix is input into the projection layer to obtain the mean μ θ and variance σ θ , which can be used to shape the corresponding Gaussian distribution model. In this example, the mean μ θ and variance σ θThe dimensions are also 4×D, which essentially means that four Gaussian distribution models are created. Then, a random number rand is randomly selected, and z_p = μ is calculated. θ +rand×σ θ z_p, as a random sampling result of a Gaussian distribution model, also has a dimension of 4×D. Finally, the matrix is ​​input into a random duration predictor to obtain the replication parameter d; based on the replication parameter d, a time-time replication operation is performed on the random sampling result z_p to obtain the intermediate vector.

[0079] Step 203: After dimension normalizing the intermediate vector, input the dimension normalized intermediate vector into the multiphase orthogonal mirror analysis filter to obtain the analysis signal.

[0080] In step 203, the intermediate vector is first normalized in terms of dimension. Specifically, in an optional embodiment, if the intermediate vector z is D-dimensional [z1, z2, ..., z...], then... D If z is divided into two equal parts of dimension D / 2, then [z1, z2, ..., z2], then z is divided into two equal parts of dimension D / 2. D / 2 ] and [z D / 2+1 ,z D / 2+2 ,…,z D Then, interleave the elements one by one to obtain [z1, z2]. D / 2+1 ,z2,z D / 2+2 ,…,z D / 2 ,z D This allows for dimensional regularization of the intermediate vector.

[0081] Then, the dimension-normalized intermediate vector is input into a multiphase orthogonal mirror analysis filter to obtain the analysis signal. For example, the dimension-normalized intermediate vector [z1, z2] is used to input the multiphase orthogonal mirror analysis filter to obtain the analysis signal. D / 2 +1,z2,z D / 2+2 ,…,z D / 2 ,z D By inputting a multiphase orthogonal mirror analysis filter, two analysis signals of length D / 2 can be obtained.

[0082] Step 204: Input the analyzed signal into the coupling layer of the speech synthesis model to perform an inverse transformation on the analyzed signal to obtain an inverse transformed signal.

[0083] In step 204, the analyzed signal is input into the coupling layer of the speech synthesis model to perform an inverse transformation on the analyzed signal to obtain an inverse transformed signal. For example, the two analyzed signals of length D / 2 obtained above are inversely input into the coupling layer, and two inverse transformed signals of length D / 2 are output.

[0084] Step 205: Input the inverse transformed signal into a polyphase quadrature mirror synthesis filter to obtain a synthesized signal.

[0085] In step 205, the inverse-transformed signal is input into a polyphase quadrature mirror synthesis filter to obtain a synthesized signal. In an optional embodiment, the process of obtaining the synthesized signal can be represented as follows: After upsampling the inverse-transformed signal by a preset factor, the upsampled inverse-transformed signal is obtained, and then input into the polyphase quadrature mirror synthesis filter to obtain a preliminary synthesized signal. All preliminary synthesized signals are then added together to obtain the synthesized signal. For example, two inverse-transformed signals of length D / 2 are upsampled by a factor of 2, and then input into the polyphase quadrature mirror synthesis filter to obtain two preliminary synthesized signals of length D. Adding these two preliminary synthesized signals together yields a synthesized signal of length D.

[0086] Step 206: After performing inverse dimension normalization on the synthesized signal, input the inverse dimension normalized synthesized signal into the decoder in the speech synthesis model to output the speech waveform.

[0087] In step 206, the synthesized signal also needs to undergo inverse dimension normalization to restore the element order of the original intermediate vectors. The dimensionally normalized synthesized signal is then input into the decoder in the speech synthesis model to output a speech waveform. For example, the synthesized signal x is [x1, x2, ..., x3]. D / 2+1 ,x2,x D / 2+2 ,…,x D / 2 ,x D After inverse dimensional normalization, it is restored to [x1, x2, ..., x]. D ], then [x1,x2,…,x D Input decoder.

[0088] Step 207: Based on the speech waveform, synthesize the target speech corresponding to the text of the speech to be synthesized.

[0089] In step 207, the target speech corresponding to the text to be synthesized is synthesized based on the speech waveform output by the decoder, and then fed back to the user.

[0090] In this embodiment, the system receives text input by the user as the speech to be synthesized; the phoneme sequence corresponding to the text is input into the preprocessing part of the speech synthesis model to obtain an intermediate vector; after dimension normalization of the intermediate vector, it is input into a polyphase orthogonal mirror analysis filter to obtain an analysis signal; the analysis signal is input into the coupling layer of the speech synthesis model to perform an inverse transformation to obtain an inverse transform signal; the inverse transform signal is input into a polyphase orthogonal mirror synthesis filter to obtain a synthesized signal; after inverse dimension normalization of the synthesized signal, it is input into the decoder of the speech synthesis model to output a speech waveform; based on the speech waveform, the target speech corresponding to the text of the speech to be synthesized is synthesized. This embodiment introduces a polyphase orthogonal mirror filter into the coupling layer of the normalized flow, establishing a fixed pattern relationship between the dimensions of the intermediate vector before processing in the coupling layer. This approach allows all dimensions of the intermediate vector to participate in the processing operations within the coupling layer, thereby helping to make the dimensions of the vector more uniform and consistent. In this way, the overall stability and robustness of the speech synthesis model can be significantly improved, thereby enhancing the stability of the speech quality synthesized using normalized flow technology.

[0091] In one optional embodiment of this application, the dimensionality normalization of the intermediate vector includes:

[0092] Based on the median length of the intermediate vector, the intermediate vector is divided into two sub-vectors;

[0093] The elements of the two sub-vectors are combined one by one to obtain the intermediate vector after the dimension is normalized;

[0094] The step of performing inverse dimensional normalization on the synthesized signal includes:

[0095] The element order of the synthesized signal is restored according to the element order of the intermediate vector to obtain the synthesized signal after inverse dimension normalization.

[0096] In this embodiment, when performing dimension normalization on the intermediate vector, the intermediate vector can be divided into two sub-vectors based on the median length of the intermediate vector. Then, the elements of the two sub-vectors are sequentially cross-combined to obtain the dimension-normalized intermediate vector. For example, if the intermediate vector z is D-dimensional [z1, z2, ..., z...]... D If z is divided into two equal parts of dimension D / 2, then [z1, z2, ..., z2], then z is divided into two equal parts of dimension D / 2. D / 2 ] and [z D / 2+1 ,z D / 2+2 ,…,z D Then, interleave the elements one by one to obtain [z1, z2]. D / 2+1 ,z2,zD / 2+2 ,…,z D / 2 ,z D This allows for dimensional regularization of the intermediate vector.

[0097] After inputting the multiphase orthogonal mirror synthesis filter and obtaining the synthesized signal, it is necessary to perform inverse dimensional normalization on the synthesized signal. Specifically, during inverse dimensional normalization, the element order of the synthesized signal is restored according to the element arrangement order of the intermediate vector, that is, according to the original element arrangement order, to obtain the dimensionally normalized synthesized signal. In this example, the synthesized signal x is [x1, x2, x3, x4, x5, x6, x7, x8, x9, x1, x1, x1, x2, x1, x2, x3 ... D / 2+1 ,x2,x D / 2+2 ,…,x D / 2 ,x D After inverse dimensional normalization, it is restored to [x1, x2, ..., x]. D ], then [x1,x2,…,x D Input decoder.

[0098] The embodiments of this application can effectively improve the parallelism of data processing by performing dimension normalization on the intermediate vector and inverse dimension normalization on the synthesized signal, thereby speeding up the processing speed; performing inverse dimension normalization after synthesizing the signal can ensure that the original structure of the signal is restored.

[0099] In an optional embodiment of this application, the step of inputting the inverse transform signal into a polyphase quadrature mirror synthesis filter to obtain a synthesized signal includes:

[0100] After upsampling the inverse transform signal by a preset multiple, the upsampled inverse transform signal is obtained;

[0101] The upsampled inverse transform signal is input into the multiphase quadrature mirror synthesis filter to obtain a preliminary synthesized signal;

[0102] The preliminary synthesized signals are added together to obtain the synthesized signal.

[0103] In this embodiment, the specific steps for PQMF synthesis of the inverse transform signal can be represented as follows: First, the inverse transform signal is upsampled by a preset factor to obtain the upsampled inverse transform signal. Then, it is input into a polyphase quadrature mirror synthesis filter to obtain a preliminary synthesized signal. Finally, all preliminary synthesized signals are added together to obtain the synthesized signal. In a feasible embodiment, two inverse transform signals of length D / 2 are upsampled by a factor of 2 and then input into a polyphase quadrature mirror synthesis filter to obtain two preliminary synthesized signals of length D. Adding the two preliminary synthesized signals together yields a synthesized signal of length D.

[0104] The embodiments of this application obtain a synthesized signal by inputting the inverse transform signal into a polyphase orthogonal mirror synthesis filter. This helps to make the internal dimensions of the signal more uniform and consistent, which can significantly improve the overall stability and robustness of the speech synthesis model, thereby improving the stability of the speech quality synthesized using normalized flow technology.

[0105] In an optional embodiment of this application, the preprocessing part of the speech synthesis model includes at least a text encoder, a projection layer, and a random duration predictor. The step of inputting the phoneme sequence corresponding to the text of the speech to be synthesized into the preprocessing part of the speech synthesis model to obtain an intermediate vector includes:

[0106] The phoneme sequence corresponding to the text to be synthesized is input into the text encoder to obtain a matrix of a preset size;

[0107] The matrix is ​​input into the projection layer to obtain the mean and variance, which are used to shape the corresponding Gaussian distribution model.

[0108] Random sampling is performed on the Gaussian distribution model to obtain random sampling results;

[0109] Based on the replication parameters, a time-based replication operation is performed on the random sampling results to obtain the intermediate vector. The replication parameters are obtained by inputting the matrix into the random duration predictor.

[0110] In this embodiment, the preprocessing part of the speech synthesis model may include at least a text encoder, a projection layer, and a random duration predictor. During preprocessing, the phoneme sequence corresponding to the text of the speech to be synthesized passes through the text encoder, the projection layer, and the random duration predictor in sequence.

[0111] First, the phoneme sequence corresponding to the text to be synthesized is input into the text encoder to obtain a matrix of a preset size. In one example, if the input phoneme sequence is "h3 ao3 r2 en2", the resulting matrix size is 4×D, where D is a preset internal dimension. For example, D can be 192.

[0112] Next, the matrix is ​​input into the projection layer to obtain the mean and variance, which are used to shape the corresponding Gaussian distribution model. In one example, the obtained mean μ θ and variance σ θ The dimensions are also 4×D, so they essentially shape four Gaussian distribution models. Further, random sampling is performed on the Gaussian distribution models to obtain the random sampling results. The specific process of random sampling can be represented as: arbitrarily selecting a random number rand, and calculating z_p = μ. θ +rand×σ θz_p, as a random sampling result of the Gaussian distribution model, also has a dimension of 4×D.

[0113] Finally, based on the replication parameter, a time-based replication operation is performed on the random sampling result to obtain the intermediate vector. The replication parameter is obtained by inputting the matrix into the random duration predictor. In one example, the matrix is ​​input into the random duration predictor to obtain the replication parameter d. Then, based on the replication parameter d, a time-based replication operation is performed on the random sampling result z_p, that is, each time step in z_p is replicated a certain number of times according to the replication parameter d, thus obtaining the intermediate vector.

[0114] The embodiments of this application preprocess the phoneme sequence corresponding to the text to be synthesized speech, which can improve the quality of synthesized speech, making the synthesized speech clearer and more accurate, and preparing for subsequent normalized stream processing.

[0115] In an optional embodiment of this application, the speech synthesis model further includes at least a posterior encoder and a streaming module, and before receiving the text of the speech to be synthesized input by the user, the method further includes:

[0116] Collect training recordings and corresponding training texts;

[0117] The phoneme sequence corresponding to the training text is input into the text encoder to obtain a training matrix of a preset size;

[0118] The training matrix is ​​input into the projection layer to obtain a first training mean and a first training variance of a first preset size. The first training mean and the first training variance are used to shape the corresponding first training Gaussian distribution model.

[0119] The linear spectrum corresponding to the training recording is input into the posterior encoder to obtain a second training mean and a second training variance of a second preset size. The second training mean and the second training variance are used to shape the corresponding second training Gaussian distribution model.

[0120] Random sampling is performed on the second trained Gaussian distribution model to obtain the training random sampling results;

[0121] The training random sampling results are input into the decoder and the streaming module respectively to obtain the generation loss and training vector;

[0122] The training vector is time-aligned with the first training Gaussian distribution model to obtain the alignment result and the relative entropy between the training vector and the first training Gaussian distribution model;

[0123] The alignment result is compared with the training replication parameters to obtain the duration loss, wherein the training replication parameters are obtained by inputting the training matrix into the random duration predictor;

[0124] Sum up the generated loss, the duration loss, and the relative entropy to obtain a loss function value;

[0125] Use the loss function value to update the speech synthesis model.

[0126] In this embodiment, before putting the speech synthesis model into use, it is necessary to train the speech synthesis model. The specific training steps are as follows.

[0127] First, collect training recordings and corresponding training texts. In an optional embodiment, the length of each training recording does not exceed 20 seconds. This application does not impose any restrictions on the collected training recordings and corresponding training texts, and those skilled in the art can collect them according to actual needs. Use a third-party phonetic annotation tool (such as pypinyin) to translate the training text into the corresponding phoneme sequence. For Chinese, the phoneme sequence consists of initials and finals. For example, if the training text is "good person", the corresponding phoneme sequence is "h3 ao3 r2 en2". After the training data is prepared, repeat the following training operations for the training recordings and corresponding training texts.

[0128] Input the phoneme sequence corresponding to the training text into the text encoder to obtain a training matrix of a preset size. In one example, if the input phoneme sequence is "h3 ao3 r2 en2", the size of the obtained training matrix is 4×D, where D is a preset internal dimension. For example, D can be 192.

[0129] Input the training matrix into the projection layer to obtain a first training mean and a first training variance of a first preset size, which are used to shape the corresponding first training Gaussian distribution model. In one example, the obtained first training mean μ θ and the first training variance σ θ also have a first preset size of 4×D respectively, and in essence, they shape 4 first training Gaussian distribution models.

[0130] Meanwhile, process the training recordings, and use a third-party signal processing tool (such as torchaudio自带 by pyTorch) to extract the linear spectrum. In one example, if the recording length corresponding to "good person" is 1 second, and the following settings are used when using a third-party signal processing tool to extract the linear spectrum: "frame shift 10 milliseconds, FFT window length 1024", the size of the obtained linear spectrum is 100x513. This application embodiment does not impose any restrictions on the process of extracting the linear spectrum of the training recordings, and those skilled in the art can select a suitable method to extract the linear spectrum of the training recordings according to actual needs.

[0131] Next, the linear spectrum corresponding to the training recording is input into the posterior encoder to obtain a second training mean and a second training variance of a second preset size. The second training mean and the second training variance are used to shape the corresponding second training Gaussian distribution model. In one example, a linear spectrum of size 100x513 is input into the posterior encoder to obtain the second training mean μ. q Second training variance σ q The second preset size is 100xD, which essentially means that 100 second training Gaussian distribution models are created. The second training Gaussian distribution model is used to describe pronunciation, while the first training Gaussian distribution model is used to describe text.

[0132] Furthermore, random sampling is performed on the second training Gaussian distribution model to obtain the training random sampling results. The specific process of random sampling can be expressed as follows: arbitrarily select a random number rand, and calculate z = μ. q +rand×σ q z, as a random sampling result of the Gaussian distribution model, also has a dimension of 100×D.

[0133] Next, the training random sampling results need to be output in two ways. The first path is input to the decoder to obtain the training speech waveform, which is then compared with the original real training recording to calculate the error and obtain the generation loss. The second path is output to the streaming module, where it is processed to obtain the training vector f. θ (z). In an optional embodiment, the stream module of the speech synthesis model actually includes a polyphase orthogonal mirror analysis filter, a coupling layer, and a polyphase orthogonal mirror synthesis filter, i.e., the normalized stream that incorporates the PQMF algorithm provided in this embodiment. In one example, the output training vector f θ The size of (z) is 100×D.

[0134] The training vectors are temporally aligned with the first trained Gaussian distribution model to obtain the alignment result and the relative entropy between the training vectors and the first trained Gaussian distribution model. In an optional embodiment, the first training mean μ is... θ and the first training variance σ θ and training vector f θ(z) Use the MAS algorithm (a dynamic search algorithm) for temporal alignment. In one example, since the former has a length of 4 and the latter has a length of 100, the MAS algorithm is used to map each moment of the latter to the target moment in the former. For example, the alignment result for a length of 100 is [1,1,1,1,1,1,2,2,2,3,3,3,3…4,4,4,4,4,4]. For this alignment result, the number of times the text corresponds to the speech is counted. For example, the first moment of the text is repeated 6 times and the second moment is repeated 3 times. By repeating different moments of the text, the training vector is temporally aligned with the first training Gaussian distribution model.

[0135] In an optional embodiment, the relative entropy (or KL distance) between the first training Gaussian distribution model (i.e., text Gaussian distribution) and the second training Gaussian distribution model (i.e., pronunciation Gaussian distribution) is further calculated based on the alignment result obtained by temporally aligning the training vectors with the first training Gaussian distribution model. The specific formula for the relative entropy can be expressed as formula (1):

[0136]

[0137] Wherein, the KL distance is the relative entropy, μ θ f is the first training mean. θ (z) is the training vector, σ θ Let σ be the first training variance. q This represents the second training variance.

[0138] Simultaneously, the alignment result is compared with the trained replication parameters to obtain the duration loss. The trained replication parameters are obtained by inputting the training matrix into a random duration predictor. In this embodiment, the training matrix is ​​input into the random duration predictor to obtain the trained replication parameters d, where d describes the number of repetitions of the text at each time step. The difference between the trained replication parameters d and the alignment result is calculated to obtain the duration loss. In one example, the size of the trained replication parameters d is 4x1.

[0139] After obtaining the three loss function parameters, the generation loss, duration loss, and relative entropy are summarized to obtain the loss function value required for training the speech synthesis model. The loss function value is then used to update the speech synthesis model. In an optional embodiment, the neural network parameters of the speech synthesis model are updated according to the standard neural network gradient backpropagation algorithm.

[0140] Reference Figure 3 This is a schematic diagram of the speech synthesis model structure of an embodiment of the speech synthesis method of this application.

[0141] Figure 3The document illustrates the structure of a speech synthesis model and the flowchart of its training process according to an embodiment of this application. The speech synthesis model includes a posterior encoder, a decoder, a text encoder, a projection layer, a streaming module, a temporal alignment module, and a random duration predictor.

[0142] like Figure 3 As shown, the phoneme sequence corresponding to the training text is first input into the text encoder, and the output is the training matrix h. text Training matrix h text The input is further fed into the projection layer to obtain the first training mean μ. θ and the first training variance σ θ In essence, they shaped four first training Gaussian distribution models.

[0143] The linear spectrum corresponding to the training recording is input into the posterior encoder to obtain a second training mean and a second training variance of a second preset size. The second training mean and the second training variance are used to shape the corresponding second training Gaussian distribution model. Random sampling is performed on the second training Gaussian distribution model to obtain the training random sampling result z. The training random sampling result z needs to be output in two ways. The first path is input to the decoder to obtain the training speech waveform, and the error is calculated by comparing it with the original real training recording to obtain the generation loss. The second path is output to the streaming module, where it is processed to obtain the training vector f. θ (z).

[0144] For training vector f θ (z) and the first training Gaussian distribution model (including the first training mean μ) θ and the first training variance σ θ Temporal alignment is performed to obtain the alignment result d and the relative entropy between the training vector and the first trained Gaussian distribution model. The training matrix is ​​input into the random duration predictor to obtain the training replication parameters, and the duration loss is obtained by comparing the alignment result d with the training replication parameters. Finally, the speech synthesis model is updated using the loss function value obtained by summing the generation loss, duration loss, and relative entropy, thus obtaining the updated speech synthesis model.

[0145] This application embodiment trains and updates the speech synthesis model, enabling it to learn rich speech features and language expressions. This helps improve the generalization ability of the speech synthesis model, significantly enhances its performance, and allows it to generate more natural, fluent, and high-quality speech output.

[0146] In an optional embodiment of this application, the streaming module includes the multiphase orthogonal mirror analysis filter, the coupling layer, and the multiphase orthogonal mirror synthesis filter. The step of inputting the training random sampling results to the decoder and the streaming module respectively to obtain the generation loss and training vector includes:

[0147] After dimension normalizing the training random sampling results, the dimension normalized training random sampling results are input into the multiphase orthogonal mirror analysis filter to obtain the training analysis signal;

[0148] The training analysis signal is input into the coupling layer to perform a forward transformation on the training analysis signal to obtain a forward transformed signal;

[0149] The positive transform signal is input into the multiphase orthogonal mirror synthesis filter to obtain the training synthesized signal;

[0150] The training synthetic signal is subjected to dimensionality inverse normalization to obtain the training vector.

[0151] In this embodiment, the stream module of the speech synthesis model actually includes a polyphase orthogonal mirror analysis filter, a coupling layer, and a polyphase orthogonal mirror synthesis filter, that is, the normalized stream that introduces the PQMF algorithm provided in this embodiment.

[0152] In this embodiment, when training the speech synthesis model, after the training random sampling results are input to the streaming module, the random sampling results are first dimensionally normalized. Dimensional normalization of the training random sampling results can be performed by dividing the results into two sub-vectors based on the median length of each sub-vector, and then combining the elements of the two sub-vectors one by one to obtain the dimensionally normalized training random sampling results. For example, if the training random sampling result x is D-dimensional [x1, x2, ..., x...]... D If x is divided into two equal parts of dimension D / 2, then [x1, x2, ..., x], then x is divided into two equal parts of dimension D / 2. D / 2 ] and [x D / 2+1 ,x D / 2+2 ,…,x D Then, interleave the elements one by one to obtain [x1, x...] D / 2+1 ,x2,x D / 2+2 ,…,x D / 2 ,x D This allows for dimensional regularization of the training random sampling results.

[0153] Then, the dimension-normalized training random sampling results are input into a multiphase orthogonal mirror analysis filter to obtain the training analysis signal. For example, the dimension-normalized training random sampling results [x1, x...] are input into a multiphase orthogonal mirror analysis filter to obtain the training analysis signal. D / 2+1 ,x2,x D / 2+2 ,…,x D / 2 ,x D By inputting a multiphase orthogonal mirror analysis filter, two training analysis signals of length D / 2 can be obtained.

[0154] The training analysis signal is input into the coupling layer of the speech synthesis model to perform a forward transformation on the training analysis signal to obtain a forward transformed signal. For example, if the two training analysis signals of length D / 2 obtained above are input forward into the coupling layer, two forward transformed signals of length D / 2 will be output.

[0155] The forward transform signal is input into a polyphase orthogonal mirror synthesis filter to obtain a training synthesized signal. In an optional embodiment, the process of obtaining the training synthesized signal can be represented as follows: After upsampling the forward transform signal by a preset factor, the upsampled forward transform signal is obtained, and then input into the polyphase orthogonal mirror synthesis filter to obtain a preliminary training synthesized signal. All preliminary training synthesized signals are then added together to obtain the final training synthesized signal. For example, two forward transform signals of length D / 2 are upsampled by a factor of 2, and then input into the polyphase orthogonal mirror synthesis filter to obtain two preliminary training synthesized signals of length D. Adding these two preliminary training synthesized signals together yields a single training synthesized signal of length D.

[0156] After obtaining the training synthesized signal from the input multiphase orthogonal mirror synthesis filter, inverse dimensionality normalization is required. Specifically, during inverse dimensionality normalization, the element order of the training synthesized signal is restored according to the element arrangement order of the training random sampling results, i.e., according to the original element arrangement order, to obtain the training vector. In this example, the training synthesized signal x is [z1, z2]. D / 2+1 ,z2,z D / 2+2 ,…,z D / 2 ,z D After inverse dimensionality normalization, it is restored to [z1, z2, ..., z]. D Then, the training vectors are time-aligned with the first training Gaussian distribution model.

[0157] Reference Figure 4 This is a schematic diagram of the flow module structure of an embodiment of the speech synthesis method of this application.

[0158] The coupling layer structure can realize the forward transformation from x to z and the inverse transformation from z to x. The forward transformation can be used in the training stage of the speech synthesis model, and the inverse transformation can be used in the speech generation stage of the speech synthesis model.

[0159] For the streaming module provided in this application embodiment, preprocessing and postprocessing are added outside the original coupling layer. The specific forward transformation process is as follows: when training the speech synthesis model, the training random sampling result x is input into the polyphase orthogonal mirror analysis filter (i.e., PQMF analysis) after dimensional normalization. After the forward transformation of the coupling layer, it is input into the polyphase orthogonal mirror synthesis filter (i.e., PQMF synthesis). Finally, after dimensional inverse normalization, the training vector z is obtained.

[0160] The specific inverse transformation process of the streaming module provided in this application embodiment is as follows: When synthesizing speech using a speech synthesis model, the intermediate vector z is input into a polyphase orthogonal mirror analysis filter (i.e., PQMF analysis) after dimensional normalization. After inverse transformation by the coupling layer, it is input into a polyphase orthogonal mirror synthesis filter (i.e., PQMF synthesis). Finally, after inverse dimensional normalization, the synthesized signal x after inverse dimensional normalization is obtained.

[0161] The speech synthesis model provided in this application introduces the PQMF algorithm into its streaming module, enabling all dimensions of the training random sampling results to participate in the processing operations within the coupling layer. This helps to make the dimensions of the training vectors more uniform and consistent, significantly improving the overall stability and robustness of the speech synthesis model, and thus enhancing the stability of the speech quality synthesized using normalized streaming technology.

[0162] In an optional embodiment of this application, the step of inputting the training random sampling results into the decoder and the streaming module respectively to obtain the generation loss and training vector includes:

[0163] The training random sampling results are input into the decoder, and the training speech waveform is output.

[0164] The error between the training speech waveform and the training recording is calculated to obtain the generation loss.

[0165] In this embodiment, the training random sampling results are input to the decoder, which outputs a training speech waveform, which is the speech generated by the speech synthesis model. The error between the training speech waveform and the actual training recording is further calculated to obtain the generation loss, which is used to subsequently update the speech synthesis model.

[0166] In this embodiment, the training random sampling results are input into the decoder to generate training speech waveforms and corresponding generation loss. The generation loss, as one of the loss function values, can guide the updating of speech synthesis model parameters, which helps the speech synthesis model gradually approach the features of real speech during the training process and improve the naturalness and intelligibility of synthesized speech.

[0167] In this embodiment, the system receives text input by the user as the speech to be synthesized; the phoneme sequence corresponding to the text is input into the preprocessing part of the speech synthesis model to obtain an intermediate vector; after dimension normalization of the intermediate vector, it is input into a polyphase orthogonal mirror analysis filter to obtain an analysis signal; the analysis signal is input into the coupling layer of the speech synthesis model to perform an inverse transformation to obtain an inverse transform signal; the inverse transform signal is input into a polyphase orthogonal mirror synthesis filter to obtain a synthesized signal; after inverse dimension normalization of the synthesized signal, it is input into the decoder of the speech synthesis model to output a speech waveform; based on the speech waveform, the target speech corresponding to the text of the speech to be synthesized is synthesized. This embodiment introduces a polyphase orthogonal mirror filter into the coupling layer of the normalized flow, establishing a fixed pattern relationship between the dimensions of the intermediate vector before processing in the coupling layer. This approach allows all dimensions of the intermediate vector to participate in the processing operations within the coupling layer, thereby helping to make the dimensions of the vector more uniform and consistent. In this way, the overall stability and robustness of the speech synthesis model can be significantly improved, thereby enhancing the stability of the speech quality synthesized using normalized flow technology.

[0168] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0169] Based on the above embodiments, this embodiment also provides a speech synthesis device that can be applied to electronic devices such as terminal devices and servers.

[0170] Reference Figure 5 The diagram shows a structural block diagram of an embodiment of a speech synthesis device according to this application, which may specifically include the following modules:

[0171] The text input module 501 is used to receive text input by the user for the speech to be synthesized;

[0172] The preprocessing module 502 is used to input the phoneme sequence corresponding to the text of the speech to be synthesized into the preprocessing part of the speech synthesis model to obtain an intermediate vector;

[0173] The analysis signal acquisition module 503 is used to input the dimension-normalized intermediate vector into a multiphase orthogonal mirror analysis filter after dimension normalization of the intermediate vector to obtain the analysis signal.

[0174] The inverse transform module 504 is used to input the analysis signal into the coupling layer in the speech synthesis model to perform an inverse transform on the analysis signal to obtain an inverse transform signal;

[0175] The synthesized signal acquisition module 505 is used to input the inverse transform signal into a multiphase orthogonal mirror synthesis filter to obtain a synthesized signal;

[0176] The dimension inverse normalization module 506 is used to input the dimension inverse normalized synthesized signal into the decoder in the speech synthesis model after performing dimension inverse normalization on the synthesized signal, and output a speech waveform.

[0177] The speech synthesis module 507 is used to synthesize target speech corresponding to the text of the speech to be synthesized based on the speech waveform.

[0178] Optionally, the analysis signal acquisition module 503 further includes:

[0179] The vector partitioning submodule is used to divide the intermediate vector into two subvectors based on the median length of the intermediate vector;

[0180] The dimension normalization submodule is used to cross-combine the elements of the two sub-vectors one by one to obtain the dimension normalized intermediate vector;

[0181] The dimensional inverse regularization module 506 also includes:

[0182] The vector restoration submodule is used to restore the element arrangement order of the synthesized signal according to the element arrangement order of the intermediate vector, so as to obtain the synthesized signal after the dimension inverse normalization.

[0183] Optionally, the synthesized signal acquisition module 505 is further configured to:

[0184] After upsampling the inverse transform signal by a preset multiple, the upsampled inverse transform signal is obtained;

[0185] The upsampled inverse transform signal is input into the multiphase quadrature mirror synthesis filter to obtain a preliminary synthesized signal;

[0186] The preliminary synthesized signals are added together to obtain the synthesized signal.

[0187] Optionally, the preprocessing part of the speech synthesis model includes at least a text encoder, a projection layer, and a random duration predictor. The preprocessing module 502 includes:

[0188] The text encoder submodule is used to input the phoneme sequence corresponding to the text of the speech to be synthesized into the text encoder to obtain a matrix of a preset size;

[0189] The projection layer submodule is used to input the matrix into the projection layer to obtain the mean and variance, which are used to shape the corresponding Gaussian distribution model.

[0190] The Gaussian distribution sampling submodule is used to perform random sampling on the Gaussian distribution model to obtain random sampling results.

[0191] The time-copying submodule is used to perform time-copying operations on the random sampling results according to the copying parameters to obtain the intermediate vector. The copying parameters are obtained by inputting the matrix into the random duration predictor.

[0192] Optionally, the speech synthesis model further includes at least a posterior encoder and a streaming module, and the device further includes:

[0193] The sample collection module is used to collect training recordings and corresponding training texts;

[0194] The text encoder input module is used to input the phoneme sequence corresponding to the training text into the text encoder to obtain a training matrix of a preset size;

[0195] The projection layer input module is used to input the training matrix into the projection layer to obtain a first training mean and a first training variance of a first preset size. The first training mean and the first training variance are used to shape the corresponding first training Gaussian distribution model.

[0196] The posterior encoder input module is used to input the linear spectrum corresponding to the training recording into the posterior encoder to obtain a second training mean and a second training variance of a second preset size. The second training mean and the second training variance are used to shape the corresponding second training Gaussian distribution model.

[0197] The training random sampling module is used to perform random sampling on the second training Gaussian distribution model to obtain the training random sampling results;

[0198] The normalized stream processing module is used to input the training random sampling results into the decoder and the stream module respectively to obtain the generation loss and training vector;

[0199] The temporal alignment module is used to perform temporal alignment between the training vector and the first training Gaussian distribution model to obtain the alignment result and the relative entropy between the training vector and the first training Gaussian distribution model.

[0200] The random duration predictor input module is used to compare the alignment result with the training replication parameters to obtain the duration loss, wherein the training replication parameters are obtained by inputting the training matrix into the random duration predictor.

[0201] The loss function module is used to summarize the generation loss, the duration loss, and the relative entropy to obtain the loss function value;

[0202] The model update module is used to update the speech synthesis model using the loss function value.

[0203] Optionally, the stream module includes the multiphase orthogonal mirror analysis filter, the coupling layer, and the multiphase orthogonal mirror synthesis filter; the normalized stream processing module includes:

[0204] The forward analysis submodule is used to input the dimension-normalized training random sampling results into the multiphase orthogonal mirror analysis filter after dimension normalization of the training random sampling results to obtain the training analysis signal;

[0205] The forward transform submodule is used to input the training analysis signal into the coupling layer to perform a forward transform on the training analysis signal to obtain a forward transform signal;

[0206] The forward synthesis submodule is used to input the forward transform signal into the multiphase orthogonal mirror synthesis filter to obtain the training synthesized signal;

[0207] The forward dimensionality inverse normalization submodule is used to perform dimensionality inverse normalization on the training synthesized signal to obtain the training vector.

[0208] Optionally, the normalized stream processing module is further configured to:

[0209] The training random sampling results are input into the decoder, and the training speech waveform is output.

[0210] The error between the training speech waveform and the training recording is calculated to obtain the generation loss.

[0211] In this embodiment, the system receives text input by the user as the speech to be synthesized; the phoneme sequence corresponding to the text is input into the preprocessing part of the speech synthesis model to obtain an intermediate vector; after dimension normalization of the intermediate vector, it is input into a polyphase orthogonal mirror analysis filter to obtain an analysis signal; the analysis signal is input into the coupling layer of the speech synthesis model to perform an inverse transformation to obtain an inverse transform signal; the inverse transform signal is input into a polyphase orthogonal mirror synthesis filter to obtain a synthesized signal; after inverse dimension normalization of the synthesized signal, it is input into the decoder of the speech synthesis model to output a speech waveform; based on the speech waveform, the target speech corresponding to the text of the speech to be synthesized is synthesized. This embodiment introduces a polyphase orthogonal mirror filter into the coupling layer of the normalized flow, establishing a fixed pattern relationship between the dimensions of the intermediate vector before processing in the coupling layer. This approach allows all dimensions of the intermediate vector to participate in the processing operations within the coupling layer, thereby helping to make the dimensions of the vector more uniform and consistent. In this way, the overall stability and robustness of the speech synthesis model can be significantly improved, thereby enhancing the stability of the speech quality synthesized using normalized flow technology.

[0212] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0213] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0214] The embodiments of this disclosure can be implemented as an apparatus configured as desired using any suitable hardware, firmware, software, or any combination thereof, including electronic devices such as terminal devices, servers (clusters), etc. Figure 6 An exemplary apparatus 600 is schematically shown that can be used to implement the various embodiments described in this application.

[0215] In one embodiment, Figure 6An exemplary device 600 is shown, which includes one or more processors 602, a control module (chipset) 604 coupled to at least one of the processors 602, a memory 606 coupled to the control module 604, a non-volatile memory (NVM) / storage device 608 coupled to the control module 604, one or more input / output devices 610 coupled to the control module 604, and a network interface 612 coupled to the control module 604.

[0216] Processor 602 may include one or more single-core or multi-core processors, and processor 602 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 600 can serve as a terminal device, server (cluster), or other device as described in the embodiments of this application.

[0217] In some embodiments, the apparatus 600 may include one or more computer-readable media (e.g., memory 606 or NVM / storage device 608) having instructions 614 and one or more processors 602 that are combined with the one or more computer-readable media and configured to execute the instructions 614 to implement the module and thus perform the actions described in this disclosure.

[0218] In one embodiment, the control module 604 may include any suitable interface controller to provide any suitable interface to at least one of the processors 602 and / or any suitable device or component communicating with the control module 604.

[0219] The control module 604 may include a memory controller module to provide an interface to the memory 606. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0220] Memory 606 may be used, for example, to load and store data and / or instructions 614 for device 600. In one embodiment, memory 606 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 606 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0221] In one embodiment, the control module 604 may include one or more input / output controllers to provide an interface to the NVM / storage device 608 and (one or more) input / output devices 610.

[0222] For example, NVM / storage device 608 may be used to store data and / or instructions 614. NVM / storage device 608 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0223] NVM / storage device 608 may include storage resources that are physically part of a device on which device 600 is mounted, or that are accessible to the device but do not necessarily have to be part of the device. For example, NVM / storage device 608 may be accessed via a network through one or more input / output devices 610.

[0224] One or more input / output devices 610 may provide an interface for device 600 to communicate with any other suitable device. Input / output devices 610 may include communication components, audio components, sensor components, etc. A network interface 612 may provide an interface for device 600 to communicate via one or more networks. Device 600 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, etc., or combinations thereof.

[0225] In one embodiment, at least one of the processors 602 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 604. In one embodiment, at least one of the processors 602 may be logically packaged with one or more controllers of the control module 604 to form a system-in-package (SiP). In one embodiment, at least one of the processors 602 may be integrated with the logic of one or more controllers of the control module 604 on the same die. In one embodiment, at least one of the processors 602 may be integrated with the logic of one or more controllers of the control module 604 on the same die to form a system-on-a-chip (SoC).

[0226] In various embodiments, device 600 may be, but is not limited to, a server, desktop computing device, or mobile computing device (e.g., laptop, handheld computing device, tablet, netbook, etc.). In various embodiments, device 600 may have more or fewer components and / or different architectures. For example, in some embodiments, device 600 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0227] The detection device may use a main control chip as a processor or control module, and sensor data, position information, etc. may be stored in a memory or NVM / storage device. The sensor group may be used as an input / output device, and the communication interface may include a network interface.

[0228] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0229] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0230] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable speech synthesis terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable speech synthesis terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0231] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable speech synthesis terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0232] These computer program instructions can also be loaded onto a computer or other programmable speech synthesis terminal device, causing a series of operational steps to be performed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0233] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0234] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0235] The above provides a detailed description of a speech synthesis method and apparatus, an electronic device, and a storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A speech synthesis method characterized by, The method comprises: receiving user input text to be synthesized speech; inputting the phoneme sequence corresponding to the text to be synthesized speech into the preprocessing part of the speech synthesis model to obtain an intermediate vector; after dimension regularization of the intermediate vector, inputting the dimension-regularized intermediate vector into a polyphase quadrature mirror analysis filter to obtain an analysis signal; inputting the analysis signal into the coupling layer of the speech synthesis model to perform inverse transformation on the analysis signal to obtain an inverse transformation signal; inputting the inverse transformation signal into a polyphase quadrature mirror synthesis filter to obtain a synthesis signal; after dimension inverse regularization of the synthesis signal, inputting the dimension-inversely-regularized synthesis signal into the decoder of the speech synthesis model to output a speech waveform; synthesizing target speech corresponding to the text to be synthesized speech according to the speech waveform; wherein the dimension regularization of the intermediate vector comprises: dividing the intermediate vector into two sub-vectors according to the median of the length of the intermediate vector; cross-combining the elements of the two sub-vectors to obtain the dimension-regularized intermediate vector; the dimension inverse regularization of the synthesis signal comprises: restoring the element arrangement order of the synthesis signal according to the element arrangement order of the intermediate vector to obtain the dimension-inversely-regularized synthesis signal.

2. The method of claim 1, wherein, The inputting of the inverse transformation signal into the polyphase quadrature mirror synthesis filter to obtain a synthesis signal comprises: after up-sampling the inverse transformation signal by a preset multiple to obtain an up-sampled inverse transformation signal; inputting the up-sampled inverse transformation signal into the polyphase quadrature mirror synthesis filter to obtain a preliminary synthesis signal; adding the preliminary synthesis signal to obtain the synthesis signal.

3. The method of claim 1, wherein, The preprocessing part of the speech synthesis model at least comprises a text encoder, a projection layer and a random duration predictor, and the inputting of the phoneme sequence corresponding to the text to be synthesized speech into the preprocessing part of the speech synthesis model to obtain an intermediate vector comprises: inputting the phoneme sequence corresponding to the text to be synthesized speech into the text encoder to obtain a matrix of a preset size; inputting the matrix into the projection layer to obtain a mean and a variance, the mean and the variance being used to shape a corresponding Gaussian distribution model; randomly sampling the Gaussian distribution model to obtain a random sampling result; performing a time copying operation on the random sampling result according to a copying parameter to obtain the intermediate vector, the copying parameter being obtained by inputting the matrix into the random duration predictor.

4. The method of claim 3, wherein, The speech synthesis model at least further comprises a posteriori encoder and a flow module, and before the receiving of the user input text to be synthesized speech, the method further comprises: collecting training audios and corresponding training texts; inputting the phoneme sequence corresponding to the training text into the text encoder to obtain a training matrix of a preset size; inputting the training matrix into the projection layer to obtain a first training mean and a first training variance of a first preset size, the first training mean and the first training variance being used to shape a corresponding first training Gaussian distribution model; inputting the linear spectrum corresponding to the training recording into the posterior encoder to obtain a second training mean value and a second training variance of a second preset size, the second training mean value and the second training variance being used to shape a corresponding second training Gaussian distribution model; randomly sampling the second training Gaussian distribution model to obtain a training random sampling result; inputting the training random sampling result into the decoder and the stream module respectively to obtain a generation loss and a training vector; performing time alignment on the training vector and the first training Gaussian distribution model to obtain an alignment result and a relative entropy between the training vector and the first training Gaussian distribution model; comparing the alignment result with a training replication parameter to obtain a duration loss, the training replication parameter being obtained by inputting the training matrix into the random duration predictor; summing up the generation loss, the duration loss and the relative entropy to obtain a loss function value; updating the speech synthesis model using the loss function value.

5. The method of claim 4, wherein, The stream module includes the polyphase quadrature mirror analysis filter, the coupling layer and the polyphase quadrature mirror synthesis filter, and the inputting of the training random sampling result into the decoder and the stream module respectively to obtain a generation loss and a training vector includes: after dimension regularization is performed on the training random sampling result, inputting the dimension-regularized training random sampling result into the polyphase quadrature mirror analysis filter to obtain a training analysis signal; inputting the training analysis signal into the coupling layer to perform forward transformation on the training analysis signal to obtain a forward transformation signal; inputting the forward transformation signal into the polyphase quadrature mirror synthesis filter to obtain a training synthesis signal; performing dimension inverse regularization on the training synthesis signal to obtain the training vector.

6. The method of claim 4, wherein, The inputting of the training random sampling result into the decoder and the stream module respectively to obtain a generation loss and a training vector includes: inputting the training random sampling result into the decoder to output a training speech waveform; calculating an error between the training speech waveform and the training recording to obtain the generation loss.

7. A speech synthesis apparatus characterized by comprising: The device includes: a text input module configured to receive a text of a speech to be synthesized input by a user; a preprocessing module configured to input a phoneme sequence corresponding to the text of the speech to be synthesized into a preprocessing part in a speech synthesis model to obtain an intermediate vector; an analysis signal acquisition module configured to input the dimension-regularized intermediate vector into a polyphase quadrature mirror analysis filter to obtain an analysis signal after dimension regularization is performed on the intermediate vector; an inverse transformation module configured to input the analysis signal into a coupling layer in the speech synthesis model to perform inverse transformation on the analysis signal to obtain an inverse transformation signal; a synthesis signal acquisition module configured to input the inverse transformation signal into a polyphase quadrature mirror synthesis filter to obtain a synthesis signal; a dimension inverse regularization module configured to input the dimension-inversely-regularized synthesis signal into a decoder in the speech synthesis model to output a speech waveform after dimension inverse regularization is performed on the synthesis signal. The speech synthesis module is configured to synthesize target speech corresponding to the text of the speech to be synthesized according to the speech waveform. The analysis signal obtaining module further comprises: The vector division sub-module is configured to divide the intermediate vector into two sub-vectors according to the median of the length of the intermediate vector. The dimension regularization sub-module is configured to cross-combine elements of the two sub-vectors one by one to obtain the intermediate vector after dimension regularization. The dimension inverse regularization module further comprises: The vector restoration sub-module is configured to restore the element arrangement order of the synthesized signal according to the element arrangement order of the intermediate vector to obtain the synthesized signal after dimension inverse regularization.

8. An electronic device, comprising: The speech synthesis method comprises: a processor; and a memory having stored thereon executable code that, when executed, causes the processor to perform the speech synthesis method according to one or more of claims 1-6.

9. One or more machine readable media having stored thereon executable code that, when executed, causes a processor to perform the speech synthesis method according to one or more of claims 1-6.

Images