Methods, devices, and electronic equipment for generating speech synthesis models

Abstract

The embodiment of the present application provides a method, device and electronic device for generating a speech synthesis model, the method comprising: being able to obtain multiple text samples; training the text samples to obtain a first speech synthesis model, wherein the first speech synthesis model includes multiple convolution processing modules, each of which includes multiple parallel convolution layers; when i takes each integer from 1 to n, determining a convolution layer equivalent to the multiple parallel convolution layers included in the i-th convolution processing module as the i-th target convolution layer; replacing the multiple parallel convolution layers included in the i-th convolution processing module of the first speech synthesis model with the i-th target convolution layer; when replacing the multiple parallel convolution layers included in the n-th convolution processing module with the n-th target convolution layer, a second speech synthesis model is obtained. Therefore, the target convolution layer is equivalently transformed to obtain the second speech synthesis model, thereby saving the memory of the model in later applications.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and electronic device for generating a speech synthesis model. Background Art

[0002] At present, the synthesis scheme of the model usually focuses on the combination of multi-path parallel and attention structure in terms of structure. The advantage of this combination design is that it can achieve a faster convergence speed during training, and thanks to the Attention structure's information modeling of the entire training sample, its synthesis effect is delicate and has a strong sense of anthropomorphism.

[0003] For example, in the acoustic model synthesis solution of speech synthesis technology, the Fast Speech model uses a large amount of Attention and multi-channel parallel structures, which will help optimize the gradient update of the model during the model training stage, avoid the problem of the underlying parameter gradient disappearance, and thus accelerate the convergence of the model. However, when this multi-channel parallel structure model is applied later, memory will be allocated separately for each input channel, thereby increasing the memory consumption of the system.

[0004] Therefore, it can be seen that in the existing technology, the model combining Attention and multi-way parallel structure can converge quickly in the training stage, but it will occupy a large amount of memory in the later application. Summary of the invention

[0005] The embodiments of the present application provide a method, device and electronic device for generating a speech synthesis model to solve the problem that a model combining Attention and a multi-path parallel structure will occupy a large amount of memory in later applications.

[0006] In a first aspect, an embodiment of the present application provides a method for generating a speech synthesis model, the method comprising:

[0007] Get multiple text samples;

[0008] Training the text sample to obtain a first speech synthesis model, wherein the first speech synthesis model includes a plurality of convolution processing modules, and each convolution processing module includes a plurality of parallel convolution layers;

[0009] When i takes each integer from 1 to n, determine a convolution layer equivalent to the multiple parallel convolution layers included in the i-th convolution processing module as the i-th target convolution layer, where n is the number of the convolution processing modules;

[0010] Replacing the multiple parallel convolutional layers included in the i-th convolutional processing module of the first speech synthesis model with the i-th target convolutional layer;

[0011] When the multiple parallel convolutional layers included in the nth convolutional processing module are replaced by the nth target convolutional layer, a second speech synthesis model is obtained.

[0012] In a second aspect, an embodiment of the present application provides a device for generating a speech synthesis model, the device comprising:

[0013] A sample acquisition module, used to acquire multiple text samples;

[0014] A model acquisition module, used to train the text sample to obtain a first speech synthesis model, wherein the first speech synthesis model includes a plurality of convolution processing modules, each of which includes a plurality of parallel convolution layers;

[0015] an equivalent transformation module, used for determining a convolution layer equivalent to the multiple parallel convolution layers included in the i-th convolution processing module as the i-th target convolution layer when i takes each integer from 1 to n, wherein n is the number of the convolution processing modules;

[0016] A replacement module, used to replace the multiple parallel convolutional layers included in the i-th convolutional processing module of the first speech synthesis model with the i-th target convolutional layer;

[0017] The model generation module is used to obtain a second speech synthesis model when the multiple parallel convolutional layers included in the nth convolutional processing module are replaced by the nth target convolutional layer.

[0018] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the above-mentioned method for generating a speech synthesis model.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned method for generating a speech synthesis model are implemented.

[0020] In an embodiment of the present application, multiple text samples can be obtained; the text samples are trained to obtain a first speech synthesis model, wherein the first speech synthesis model includes multiple convolution processing modules, each convolution processing module includes multiple parallel convolution layers; when i takes each integer from 1 to n, a convolution layer equivalent to the multiple parallel convolution layers included in the i-th convolution processing module is determined as the i-th target convolution layer, wherein n is the number of convolution processing modules; the multiple parallel convolution layers included in the i-th convolution processing module of the first speech synthesis model are replaced with the i-th target convolution layer; when the multiple parallel convolution layers included in the n-th convolution processing module are replaced with the n-th target convolution layer, a second speech synthesis model is obtained.

[0021] Among them, the first speech synthesis model adopts a multi-channel model structure in the training stage, and in the generation stage, through identity transformation, without changing the input and output, the multiple parallel convolutional layers included in the multiple convolution processing modules in the first speech synthesis model are equivalently transformed into a target convolutional layer, thereby obtaining the second speech synthesis model, that is, converting the first speech synthesis model of the multi-channel model structure into the second speech synthesis model of the single-channel model structure. Therefore, in the embodiment of the present application, the excellent performance of the multi-channel model structure is retained in the training stage to help the model converge quickly, and the advantage of the single-channel model structure is lightweight in the deployment stage, thereby solving the problem that the model composed of Attention and multi-channel parallel structures will occupy a large amount of memory in later applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 It is a flowchart of the steps of a method for generating a speech synthesis model provided in an embodiment of the present application;

[0024] Figure 2 It is a schematic diagram of the Fast Speech model architecture in the prior art provided in the embodiment of the present application;

[0025] Figure 3 It is a schematic diagram of the structure of an encoder composed of FFT blocks in the prior art provided by an embodiment of the present application;

[0026] Figure 4 It is a structural diagram of Length Regular in the prior art provided by the embodiment of the present application;

[0027] Figure 5 It is a structural diagram of Duration Predictor in the prior art provided by an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of the first speech synthesis model architecture provided in an embodiment of the present application;

[0029] Figure 7 It is a schematic diagram of the process of equivalent transformation of convolution kernel provided in the embodiment of the present application;

[0030] Figure 8 It is a flowchart of the model training phase provided in the embodiment of the present application;

[0031] Fig. 9 It is a flow chart of the model application stage provided in the embodiment of the present application;

[0032] Fig.10 It is a structural block diagram of a device for generating a speech synthesis model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] In order to facilitate understanding of the method for generating a speech synthesis model in the embodiment of the present application, the existing related technologies are now introduced as follows:

[0035] At present, in the acoustic model synthesis scheme of speech synthesis technology, the fast speech synthesis (Fast Speech) system architecture is as follows: Figure 2 As shown in the figure, the Fast Speech system architecture includes the rain removal network (PReNet) and Figure 3 The encoder (Encoder) composed of the fast Fourier transform module (FFT block) shown in Figure 4The length prediction model (LengthRegular) and the decoder (Decoder) are also composed of FFT blocks. The input of the Encoder is the sum of the feature vector and positional encoding vector (Positional Encoding) of the same phoneme in the text phoneme sequence to be processed obtained by the pre-processing structure (Phoneme Embedding); Length Regular is used to predict the frame length used for each phoneme; the Decoder processes the input Encoder and Length Regular, and outputs the acoustic features through the neural network module (Linear Layer).

[0036] The training process of the Fast Speech system architecture model is as follows: Steps F1 to F5:

[0037] Step F1: The text sample is processed by the front-end to obtain a phoneme sequence (Phoneme), and then enters the pre-processing structure of the Encoder for sequence integration.

[0038] Step F2: The preprocessed phoneme features are added to the positional encoding information and input into the encoder for information encoding. The basic structure of the encoder consists of N multi-channel parallel attention modules (Multi-Headattention) stacked together (i.e., N FFT Blocks). Figure 3 As shown in the figure, the phoneme features are shaped and normalized by each Multi-Headatattention module self-encoding (self-Attention) followed by a feed-forward neural network (FFN), a one-dimensional convolution (Conv1D) and a residual connection normalization module (Add&Norm).

[0039] Step F3: Length Regular receives the output of the Encoder, predicts the duration of each input phoneme, and expands the encoded output into a new matrix according to the phoneme duration as the input of the Decoder, that is, Figure 4 As shown, the output of the encoder (Hpho) is processed as follows Figure 5 The duration predictor (Duration Predictor) shown predicts the duration of each input phoneme, where "D = [2, 2, 3, 1]" is the phoneme duration, and is then expanded into a new matrix through the length regulator (i.e. LR) as the input of the Decoder (Hmel), where "a = 1.0" is the matrix expansion coefficient (alpha) of 1.0.

[0040] Step F4: The decoder structure is composed of N stacked Multi-Head attention modules. The matrix encoding information output by Length Regular is encoded in the same position and input into the Multi-Head attention structure for self-encoding. Finally, the acoustic features decoded by the decoder are post-processed (PostNet) to obtain optimized and smoothed acoustic features.

[0041] Step F5: The acoustic features are passed through a vocoder to obtain the target speaker's speech.

[0042] In addition, the Fast Speech model uses a large number of Attention and multi-way parallel structures. The advantage of the multi-way parallel structure is that it can help optimize the gradient update of the model during the training phase by taking advantage of the convenience of residual derivation, avoiding the problem of the vanishing gradient of the underlying parameters, thereby accelerating the convergence of the model. Its disadvantage is that it allocates memory separately for each input, thereby increasing the memory consumption of the system. That is, the Attention structure models the entire input sequence, making the context information richer and more accurate, but the memory it consumes is O(n^2) obtained according to the space complexity algorithm, where n is the sequence length. It can be seen from this that although the Fast Speech model can converge quickly during the training phase, it will occupy a large amount of memory in later applications.

[0043] In an embodiment of the present application, multiple text samples can be obtained; the text samples are trained to obtain a first speech synthesis model, wherein the first speech synthesis model includes multiple convolution processing modules, each convolution processing module includes multiple parallel convolution layers; when i takes each integer from 1 to n, a convolution layer equivalent to the multiple parallel convolution layers included in the i-th convolution processing module is determined as the i-th target convolution layer, wherein n is the number of convolution processing modules; the multiple parallel convolution layers included in the i-th convolution processing module of the first speech synthesis model are replaced with the i-th target convolution layer; when the multiple parallel convolution layers included in the n-th convolution processing module are replaced with the n-th target convolution layer, a second speech synthesis model is obtained.

[0044] That is, in the embodiment of the present application, the first speech synthesis model adopts a multi-channel model structure in the training stage, and in the generation stage, through identity transformation, the multiple parallel convolutional layers included in the multiple convolution processing modules in the first speech synthesis model are equivalently transformed into a target convolutional layer without changing the input and output, thereby obtaining the second speech synthesis model, that is, converting the first speech synthesis model of the multi-channel model structure into the second speech synthesis model of the single-channel model structure. Therefore, in the embodiment of the present application, the second speech synthesis model retains the excellent performance of the multi-channel model structure in the training stage, helping the model to converge quickly, and also has the advantage of the single-channel model structure being lightweight in the deployment stage, thereby solving the problem that the model composed of Attention and multi-channel parallel structures will occupy a large amount of memory in later applications.

[0045] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the method for generating the speech synthesis model provided in the embodiment of the present application is described in detail below in combination with the accompanying drawings and specific implementation methods.

[0046] Reference Figure 1 , shows a step flow chart of a method for generating a speech synthesis model in an embodiment of the present application, and the method may include the following steps 101 to 105.

[0047] Step 101: Obtain multiple text samples.

[0048] For example, text samples can be obtained through keyboard input, image recognition, etc., where, when keyboard input is used, the text sample is the text generated by a keyboard key sequence; when image recognition is used, the text sample is the text recognized from the image.

[0049] Step 102: Train the text sample to obtain a first speech synthesis model, wherein the first speech synthesis model includes a plurality of convolution processing modules, and each convolution processing module includes a plurality of parallel convolution layers.

[0050] In addition, the first speech synthesis model is a model for outputting the speech acoustic features of the text to be processed. Therefore, the input of the first speech model is the text to be processed, and the output is the speech acoustic features corresponding to the text to be processed.

[0051] Step 103: When i takes each integer from 1 to n, determine a convolution layer equivalent to the multiple parallel convolution layers included in the i-th convolution processing module as the i-th target convolution layer, where n is the number of the convolution processing modules.

[0052] It can be seen from this that in the embodiment of the present application, for each convolution processing module of the first speech synthesis model, there is an equivalent target convolution layer.

[0053] In addition, a convolution layer equivalent to the multiple parallel convolution layers included in the i-th convolution processing module is determined as the i-th target convolution layer, so that the output result obtained after the input of the model is processed by the multiple parallel convolution layers included in the i-th convolution processing module is the same as the output result obtained after being processed by the i-th target convolution layer.

[0054] Therefore, the multiple parallel convolutional layers included in the i-th convolutional processing module are equivalent to the i-th target convolutional layer, that is, the same input can obtain the same output after being processed by the multiple parallel convolutional layers included in the i-th convolutional processing module and the i-th target convolutional layer.

[0055] Step 104: Replace the multiple parallel convolutional layers included in the i-th convolutional processing module of the first speech synthesis model with the i-th target convolutional layer.

[0056] Among them, after the model training is completed, the multiple parallel convolutional layers in the i-th convolutional processing module included in the obtained first speech synthesis model can be equivalently transformed into the i-th target convolutional layer.

[0057] Step 105: When the multiple parallel convolutional layers included in the nth convolutional processing module are replaced by the nth target convolutional layer, a second speech synthesis model is obtained.

[0058] In an embodiment of the present application, by replacing multiple parallel convolutional layers included in multiple convolutional processing modules in the first speech synthesis model with a target convolutional layer, a second speech synthesis model can be obtained.

[0059] Among them, the second speech synthesis model is a model for outputting the speech acoustic features of the text to be processed. Therefore, the input of the second speech synthesis model is the text to be processed, and the output is the speech acoustic features corresponding to the text to be processed. Therefore, in the embodiment of the present application, the text to be processed can be input into the second speech synthesis model in the actual application stage, so as to obtain the speech acoustic features corresponding to the text to be processed.

[0060] In addition, after the same input is input to the first speech synthesis model and the second speech synthesis model respectively, in the first speech synthesis model, it is processed by multiple parallel convolutional layers included in multiple convolutional processing modules, and the system allocates memory for each parallel convolutional layer; while in the second speech synthesis model, it only needs to be processed by target convolutional layers equivalent to the parallel convolutional layers included in the convolutional processing module, that is, only memory needs to be allocated for these target convolutional layers. From this comparison, it can be seen that the second speech synthesis model can save system memory compared to the first speech synthesis model.

[0061] As can be seen from the above steps 101 to 105, in the embodiments of the present application, multiple text samples can be obtained; the text samples are trained to obtain a first speech synthesis model, where the first speech synthesis model includes multiple convolutional processing modules, and each convolutional processing module includes multiple parallel convolutional layers; when i takes each integer from 1 to n, one convolutional layer equivalent to the multiple parallel convolutional layers included in the i-th convolutional processing module is determined as the i-th target convolutional layer, where n is the number of convolutional processing modules; the multiple parallel convolutional layers included in the i-th convolutional processing module of the first speech synthesis model are replaced with the i-th target convolutional layer; when the multiple parallel convolutional layers included in the n-th convolutional processing module are replaced with the n-th target convolutional layer, a second speech synthesis model is obtained.

[0062] Among them, the first speech synthesis model adopts a multi-path model structure in the training stage. In the generation stage, through an identity transformation, without changing the input and output, the multiple parallel convolutional layers included in multiple convolutional processing modules in the first speech synthesis model are equivalently transformed into one target convolutional layer, thereby obtaining the second speech synthesis model, that is, converting the first speech synthesis model with a multi-path model structure into the second speech synthesis model with a single-path model structure. Therefore, in the embodiments of the present application, the second speech synthesis model not only retains the excellent performance of the multi-path model structure in the training stage to help the model converge quickly, but also has the advantage of lightweight in the deployment stage of the single-path model structure, thus solving the problem that the model combining Attention and the multi-path parallel structure will occupy a large amount of memory in the later application.

[0063] Optionally, after obtaining the second speech synthesis model, the method further includes:

[0064] Obtain the text to be processed;

[0065] Obtain the phoneme sequence of the text to be processed;

[0066] Obtain the feature vector of each phoneme in the phoneme sequence and the position encoding vector of each phoneme in the phoneme sequence;

[0067] Input the feature vector and the position encoding vector into the second speech synthesis model, and output the speech acoustic features of the text to be processed.

[0068] Among them, a phoneme is the smallest speech unit divided according to the natural attributes of speech. Analyzed according to the pronunciation actions in a syllable, one action constitutes one phoneme. For example, the Chinese syllable "ā" has only one phoneme, "ài" has two phonemes, "dài" has three phonemes, etc. Then the phoneme sequence is the arrangement sequence of each phoneme corresponding to the Chinese syllable.

[0069] For example, if the text to be processed is "get the speaker's voice", the phoneme sequence of the text to be processed is "huoqu shuo hua ren yu yin"; then, obtain the feature vector of each phoneme in the phoneme sequence, such as obtaining the feature vector corresponding to "q" and the position coding vector of "q" in the phoneme sequence, wherein the position of "q" in the phoneme sequence is the 4th phoneme, then obtain the position coding vector corresponding to the 4th phoneme; then, add the feature vector and the position coding vector of the same phoneme in the phoneme sequence, thereby inputting the added vector into the second speech synthesis model, and then outputting the speech acoustic features of the text to be processed.

[0070] In addition, the speech acoustic features output by the second speech synthesis model can be passed through a vocoder to obtain speech text.

[0071] Optionally, the first speech synthesis model includes an encoder, a duration prediction module, and a decoder, wherein the encoder includes at least one convolution processing module and the decoder includes at least one convolution processing module.

[0072] The encoder may be composed of a PReNet and a boundary representation module (Rep block), and the decoder may be composed of a Rep block. Figure 6 As shown, in an embodiment of the present application, the overall architecture of the first speech synthesis model may include three parts: an Encoder composed of a PReNet and a boundary representation module (Rep block), a Length Regular, and a Decoder also composed of a Repblock.

[0073] In addition, the input of the Encoder is the sum of the feature vector and positional encoding vector (Positional Encoding) of the same phoneme in the text phoneme sequence to be processed obtained through Phoneme Embedding; Length Regular is used to predict the frame length used for each phoneme; the Decoder processes the input Encoder and Length Regular, passes through the Linear Layer, and outputs the acoustic features.

[0074] Optionally, each convolution layer includes a convolution kernel, and the convolution kernel included in the i-th target convolution layer is the sum of the convolution kernels of multiple convolution layers included in the i-th convolution processing module and the i-th unit matrix, and the number of rows and columns of the i-th unit matrix are the same as the number of rows and columns of the convolution kernel with the least elements in the i-th convolution processing module.

[0075] For example, one of the convolution processing modules of the first speech synthesis model includes two parallel convolution layers, and the convolution kernels of the two convolution layers are as follows: Figure 7 The first preset convolution kernel 701 and the second preset convolution kernel 702 are shown. Wherein, the first preset convolution kernel is a 3*3 matrix, and the second preset convolution kernel is a 1*1 matrix, then the second preset convolution kernel and the 1*1 unit matrix 703 need to be padded with zeros to form a 3*3 matrix, and then the first preset convolution kernel and the second preset convolution kernel after zero padding are added to the 1*1 unit matrix after zero padding to obtain a new 3*3 matrix, and this new 3*3 matrix is ​​the target convolution kernel 704 equivalent to the first preset convolution kernel and the second preset convolution kernel.

[0076] Or, for example, the first preset convolution kernel is a 3*3 matrix, and the second preset convolution kernel is a 3*3 matrix, then the unit matrix is ​​a 3*3 matrix, and then the first preset convolution kernel, the second preset convolution kernel and the 3*3 unit matrix are added to obtain a new 3*3 matrix. This new 3*3 matrix is ​​the target convolution kernel equivalent to the first preset convolution kernel and the second preset convolution kernel.

[0077] That is, after the same vector is processed by the first preset convolution kernel and the second preset convolution kernel respectively, the result of adding the obtained matrix to the unit matrix 703 is the same as the result after being processed only by the target convolution kernel 704.

[0078] Optionally, training the text sample to obtain a first speech synthesis model includes:

[0079] Obtaining a phoneme sequence of the text sample;

[0080] Obtaining a feature vector of each phoneme in the phoneme sequence and a position encoding vector of each phoneme in the phoneme sequence;

[0081] Adding the feature vector and the position encoding vector of the same phoneme to obtain a plurality of first vectors;

[0082] Using the predetermined parameters of the third speech synthesis model, the first vector of the same text sample is processed to obtain speech acoustic features corresponding to each text sample;

[0083] Modifying the parameters of the third speech synthesis model according to the obtained speech acoustic features to obtain the parameters of the fourth speech synthesis model;

[0084] The parameters of the fourth speech synthesis model are used to process the first vector of the same text sample until the speech acoustic features corresponding to the obtained text sample meet preset conditions, and the speech synthesis model that meets the preset conditions is determined as the first speech synthesis model.

[0085] Among them, when the first speech synthesis model includes an encoder, a duration prediction module, and a decoder, the parameters of the third speech synthesis model and the parameters of the fourth speech synthesis model include parameters of the encoder, parameters of the duration prediction module, and parameters of the decoder.

[0086] In addition, each text sample corresponds to a real speech text, so that each speech text synthesized by the speech acoustic features output by the speech synthesis model can be compared with the real speech text corresponding to the corresponding text sample, so as to obtain the similarity between the two, and then the speech acoustic features with a similarity greater than the preset similarity can be marked with a preset mark. Therefore, the above preset condition can be that the number of speech acoustic features marked with a preset mark reaches a preset number. That is, the number of speech acoustic features marked with a preset mark reaches a preset number, indicating that the accuracy of the speech synthesis model that outputs the speech acoustic features has met the requirements, that is, the speech synthesis model has reached the convergence condition.

[0087] Alternatively, the speech acoustic features output by the speech synthesis model each time are compared with the actual speech acoustic features of each text sample to calculate the loss value of the model. Therefore, the above preset condition can be that the calculated model loss value is less than the preset loss value. That is, if the calculated model loss value is less than the preset loss value, it means that the accuracy of the speech synthesis model that outputs the speech acoustic features has met the requirements, that is, the speech synthesis model has reached the convergence condition.

[0088] Optionally, the third speech synthesis model includes an encoder, a duration prediction module and a decoder;

[0089] The process of processing one of the first vectors of a text sample using the predetermined parameters of the third speech synthesis model includes:

[0090] Inputting the second vector into an encoder of the third speech synthesis model, and outputting an encoded vector, wherein the second vector is one of the first vectors of one of the text samples;

[0091] Input the encoding vector into the duration prediction module of the third speech synthesis model, and output a phoneme duration matrix obtained by extending the encoding vector according to the duration of the phoneme corresponding to the second vector;

[0092] The phoneme duration matrix is ​​input into the decoder of the third speech synthesis model, and the speech acoustic features of the phonemes corresponding to the second vector are output.

[0093] It can be seen that the input of the Encoder is the sum of the feature vector and position encoding vector of the same phoneme in the text phoneme sequence to be processed; Length Regular is used to predict the frame length used for each phoneme; Decoder processes the input Encoder and Length Regular and outputs acoustic features.

[0094] For example, if the phoneme sequence of a text sample is "huo qu shuo hua ren yu yin", each phoneme in this phoneme sequence will correspond to a first vector, and all the obtained first vectors are input into the third speech synthesis model, and then the encoder of the third speech synthesis model processes each first vector respectively, and outputs the encoding vector corresponding to each first vector. Then, the duration prediction module of the third speech synthesis model processes the encoding vector corresponding to each first vector respectively, and outputs the phoneme duration matrix of each encoding vector expanded according to the duration of the phoneme corresponding to each first vector. Finally, the duration prediction module of the third speech synthesis model processes each phoneme duration matrix respectively, and outputs the speech acoustic features of the phoneme corresponding to each first vector.

[0095] Optionally, the encoder of the third speech synthesis model includes a first convolution processing module and a second convolution processing module, the first convolution processing module includes a first convolution layer and a second convolution layer, the first convolution layer includes a first preset convolution kernel, the second convolution layer includes a second preset convolution kernel, the second convolution processing module includes a third convolution layer and a fourth convolution layer, the third convolution layer includes a third preset convolution kernel, and the fourth convolution layer includes a fourth preset convolution kernel;

[0096] The step of inputting the second vector to the encoder of the third speech synthesis model and outputting the encoded vector comprises:

[0097] Processing the second vector through the first preset convolution kernel to obtain a first matrix, and processing the second vector through the second preset convolution kernel to obtain a second matrix;

[0098] Add the first matrix, the second matrix and the first identity matrix to obtain a third matrix;

[0099] Normalizing the third matrix to obtain a third vector;

[0100] Processing the third vector through the third preset convolution kernel to obtain a fifth matrix, and processing the third vector through the fourth preset convolution kernel to obtain a sixth matrix;

[0101] Add the fifth matrix, the sixth matrix and the second unit matrix to obtain a seventh matrix;

[0102] Normalizing the seventh matrix to obtain the encoding vector;

[0103] Among them, the number of rows and columns of the first unit matrix are the same as the number of rows and columns of the convolution kernel with the least elements in the first convolution processing module, and the number of rows and columns of the second unit matrix are the same as the number of rows and columns of the convolution kernel with the least elements in the second convolution processing module.

[0104] For example Figure 8 In the training stage shown in the figure, the second vector is processed by the first preset convolution kernel to obtain a 3*3 first matrix, wherein the first preset convolution kernel is a 3*3 convolution kernel; the second vector is processed by the second preset convolution kernel to obtain a 1*1 second matrix, wherein the second preset convolution kernel is a 1*1 convolution kernel; the second matrix and the first unit matrix (i.e., the 1*1 unit matrix) are processed into 3*3 matrices by zero padding operation, and then added to the first matrix to obtain a 3*3 third matrix; the third matrix is ​​processed by the normalization module (Batch Norm) is used to obtain a third vector; the third vector is processed by a third preset convolution kernel to obtain a 3*3 fifth matrix, wherein the third preset convolution kernel is a 3*3 convolution kernel; the third vector is processed by a fourth preset convolution kernel to obtain a 1*1 sixth matrix, wherein the fourth preset convolution kernel is a 1*1 convolution kernel; the sixth matrix and the second unit matrix (i.e., the 1*1 unit matrix) are respectively processed into 3*3 matrices through zero-padding operations, and then added to the fifth matrix to obtain a 3*3 seventh matrix; the seventh matrix is ​​normalized by Batch Norm, and the encoding vector corresponding to the second vector is output.

[0105] Optionally, the decoder of the third speech synthesis model includes a third convolution processing module and a fourth convolution processing module, the third convolution processing module includes a fifth convolution layer and a sixth convolution layer, the fifth convolution layer includes a fifth preset convolution kernel, the sixth convolution layer includes a sixth preset convolution kernel, the fourth convolution processing module includes a seventh convolution layer and an eighth convolution layer, the seventh convolution layer includes a seventh preset convolution kernel, and the eighth convolution layer includes an eighth preset convolution kernel;

[0106] The step of inputting the phoneme duration matrix into a decoder of the third speech synthesis model and outputting speech acoustic features of the phonemes corresponding to the second vector comprises:

[0107] Processing the phoneme duration matrix through the fifth preset convolution kernel to obtain an eighth matrix, and processing the phoneme duration matrix through the sixth preset convolution kernel to obtain a ninth matrix;

[0108] Adding the eighth matrix, the ninth matrix and the third unit matrix to obtain a tenth matrix;

[0109] Normalizing the tenth matrix to obtain a fourth vector;

[0110] Processing the fourth vector through the seventh preset convolution kernel to obtain an eleventh matrix, and processing the fourth vector through the eighth preset convolution kernel to obtain a twelfth matrix;

[0111] Add the eleventh matrix, the twelfth matrix and the fourth unit matrix to obtain a thirteenth matrix;

[0112] Normalizing the thirteenth matrix to obtain speech acoustic features of the phoneme corresponding to the second vector;

[0113] Among them, the number of rows and columns of the third unit matrix are the same as the number of rows and columns of the convolution kernel with the least elements in the third convolution processing module, and the number of rows and columns of the fourth unit matrix are the same as the number of rows and columns of the convolution kernel with the least elements in the fourth convolution processing module.

[0114] For example, in Figure 8 In the training stage shown in the figure, the phoneme duration matrix is ​​processed by the fifth preset convolution kernel to obtain a 3*3 eighth matrix, wherein the fifth preset convolution kernel is a 3*3 convolution kernel; the phoneme duration matrix is ​​processed by the sixth preset convolution kernel to obtain a 1*1 ninth matrix, wherein the sixth preset convolution kernel is a 1*1 convolution kernel; the ninth matrix and the third unit matrix (i.e., the 1*1 unit matrix) are processed into 3*3 matrices by zero padding operation, and then added to the eighth matrix to obtain a 3*3 tenth matrix; the tenth matrix is ​​processed by Batch Norm is used for normalization to obtain a fourth vector; the fourth vector is processed by the seventh preset convolution kernel to obtain a 3*3 eleventh matrix, wherein the seventh preset convolution kernel is a 3*3 convolution kernel; the fourth vector is processed by the eighth preset convolution kernel to obtain a 1*1 twelfth matrix, wherein the eighth preset convolution kernel is a 1*1 convolution kernel; the twelfth matrix and the fourth unit matrix (i.e., the 1*1 unit matrix) are respectively processed into 3*3 matrices through zero-padding operations, and then added to the eleventh matrix to obtain a 3*3 thirteenth matrix; the thirteenth matrix is ​​normalized by Batch Norm, and the speech acoustic features of the phonemes corresponding to the second vector are output.

[0115] In addition, after the training of the first speech synthesis model is completed, the convolution kernel of each convolution processing module in the first speech synthesis model can be converted into an equivalent convolution kernel, and the equivalent convolution kernel obtained by the conversion is the convolution kernel in the second speech synthesis model.

[0116] For example, in the second speech synthesis model, the encoder includes a 3*3 first target convolution kernel and a 3*3 second target convolution kernel, and the decoder includes a 3*3 third target convolution kernel and a 3*3 fourth target convolution kernel. Fig. 9 As shown, in the actual application stage of the second speech synthesis model, in the phoneme sequence of the to-be-processed text, the sum of the feature vector and the position encoding vector of the same phoneme sequence is input into the second speech synthesis model, and is processed by the first target convolution kernel in the encoder to obtain a 3*3 first target matrix, which is normalized by Batch Norm to obtain a first target vector; the first target vector is processed by the second target convolution kernel to obtain a 3*3 second target matrix, which is normalized by Batch Norm to output the encoding vector; the encoding vector is input into the duration prediction module of the second speech synthesis model, and the phoneme duration matrix in which the encoding vector is expanded according to the duration of the phoneme is output; further, the phoneme duration matrix is ​​processed by the third target convolution kernel in the decoder to obtain a 3*3 third target matrix, which is normalized by Batch Norm to obtain a third target vector; the third target vector is processed by the fourth target convolution kernel in the decoder to obtain a 3*3 fourth target matrix, which is normalized by Batch Norm to output the speech acoustic features of the phoneme corresponding to the second vector.

[0117] In summary, the specific implementation of the method for generating a speech synthesis model in the embodiment of the present application can be described as follows:

[0118] Step H1: Acquire multiple text samples, and process one of the text samples through the front-end to obtain a corresponding phoneme sequence.

[0119] Step H2: Add the feature vector and position coding vector of the same phoneme in the phoneme sequence to obtain multiple first vectors. Steps H3 to H5 are performed during the training phase of the first speech synthesis model. Step H6 is performed during the generation phase of the second speech synthesis model.

[0120] Step H3: Using the predetermined parameters of the third speech synthesis model, the first vector of the same text sample is processed to obtain speech acoustic features corresponding to each text sample.

[0121] Step H4: According to the obtained speech acoustic features corresponding to each text sample, modify the parameters of the third speech synthesis model to obtain the parameters of the fourth speech synthesis model.

[0122] Step H5: Use the parameters of the fourth speech synthesis model to process the first vector of the same text sample until the speech acoustic features corresponding to the obtained text sample meet the preset conditions, and determine the speech synthesis model that meets the preset conditions as the first speech synthesis model.

[0123] Step H6: Equivalently transform the multiple parallel convolutional layers included in each convolution processing module in the first speech synthesis model into a target convolutional layer to obtain a second speech synthesis model.

[0124] Among them, after the second speech synthesis model is generated, in actual application, the text to be processed is processed to obtain the phoneme sequence of the text to be processed, and the sum of the feature vector of the same phoneme and the position coding vector in the phoneme sequence is calculated to obtain multiple input vectors, and then the multiple input vectors are input into the second speech synthesis model, so that the speech acoustic features of the text sample to be processed can be output.

[0125] From the above, it can be seen that in the embodiment of the present application, the first speech synthesis model adopts a multi-channel model structure in the training stage, and in the generation stage, through the identity transformation, without changing the input and output, the multiple parallel convolutional layers included in the multiple convolution processing modules in the first speech synthesis model are equivalently transformed into a target convolutional layer, thereby obtaining the second speech synthesis model, that is, converting the first speech synthesis model of the multi-channel model structure into the second speech synthesis model of the single-channel model structure. Therefore, in the embodiment of the present application, the second speech synthesis model retains the excellent performance of the multi-channel model structure in the training stage, helping the model to converge quickly, and has the advantage of the lightweight single-channel model structure in the deployment stage, thereby solving the problem that the model composed of Attention and multi-channel parallel structures will occupy a large amount of memory in later applications.

[0126] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0127] Reference Fig.10 , shows a structural block diagram of a device for generating a speech synthesis model in an embodiment of the present application. The device 1000 for generating a speech synthesis model may include the following modules:

[0128] The sample acquisition module 1001 is used to acquire multiple text samples;

[0129] A model acquisition module 1002, used to train the text sample to obtain a first speech synthesis model, wherein the first speech synthesis model includes a plurality of convolution processing modules, each of which includes a plurality of parallel convolution layers;

[0130] The equivalent transformation module 1003 is used to determine a convolution layer equivalent to the multiple parallel convolution layers included in the i-th convolution processing module as the i-th target convolution layer when i takes each integer from 1 to n, where n is the number of the convolution processing modules;

[0131] A replacement module 1004 is used to replace the multiple parallel convolutional layers included in the i-th convolutional processing module of the first speech synthesis model with the i-th target convolutional layer;

[0132] The model generation module 1005 is used to obtain a second speech synthesis model when the multiple parallel convolutional layers included in the nth convolutional processing module are replaced by the nth target convolutional layer.

[0133] Optionally, the speech synthesis model generation device 1000 further includes:

[0134] A first acquisition module, used to acquire the text to be processed;

[0135] A second acquisition module, used to acquire a phoneme sequence of the text to be processed;

[0136] A third acquisition module is used to obtain a feature vector of each phoneme in the phoneme sequence and a position encoding vector of each phoneme in the phoneme sequence;

[0137] An output module is used to input the feature vector and the position encoding vector into the second speech synthesis model, and output the speech acoustic features of the text to be processed.

[0138] Optionally, the first speech synthesis model includes an encoder, a duration prediction module, and a decoder, wherein the encoder includes at least one convolution processing module and the decoder includes at least one convolution processing module.

[0139] Optionally, each convolution layer includes a convolution kernel, and the convolution kernel included in the i-th target convolution layer is the sum of the convolution kernels of multiple convolution layers included in the i-th convolution processing module and the i-th unit matrix, and the number of rows and columns of the i-th unit matrix are the same as the number of rows and columns of the convolution kernel with the least elements in the i-th convolution processing module.

[0140] Optionally, the model acquisition module 1002 includes:

[0141] A first acquisition submodule, used to acquire a phoneme sequence of the text sample;

[0142] A second acquisition submodule is used to acquire a feature vector of each phoneme in the phoneme sequence and a position encoding vector of each phoneme in the phoneme sequence;

[0143] A first vector acquisition submodule, used for adding a feature vector and a position coding vector of a same phoneme to obtain a plurality of first vectors;

[0144] A third acquisition submodule, used to process the first vector of the same text sample using the predetermined parameters of the third speech synthesis model to obtain speech acoustic features corresponding to each text sample;

[0145] A fourth acquisition submodule, used to modify the parameters of the third speech synthesis model according to the acquired speech acoustic features, so as to obtain the parameters of the fourth speech synthesis model;

[0146] A determination submodule is used to adopt the parameters of the fourth speech synthesis model to process the first vector of the same text sample until the speech acoustic features corresponding to the obtained text sample meet the preset conditions, and the speech synthesis model that meets the preset conditions is determined as the first speech synthesis model.

[0147] Optionally, the third speech synthesis model includes an encoder, a duration prediction module and a decoder;

[0148] The third acquisition submodule includes:

[0149] A second output unit, configured to input the second vector into an encoder of the third speech synthesis model, and output an encoded vector, wherein the second vector is one of the first vectors of one of the text samples;

[0150] A third output unit, configured to input the encoding vector into a duration prediction module of the third speech synthesis model, and output a phoneme duration matrix obtained by extending the encoding vector according to the duration of the phoneme corresponding to the second vector;

[0151] The fourth output unit is used to input the phoneme duration matrix into the decoder of the third speech synthesis model, and output the speech acoustic features of the phonemes corresponding to the second vector.

[0152] Optionally, the encoder of the third speech synthesis model includes a first convolution processing module and a second convolution processing module, the first convolution processing module includes a first convolution layer and a second convolution layer, the first convolution layer includes a first preset convolution kernel, the second convolution layer includes a second preset convolution kernel, the second convolution processing module includes a third convolution layer and a fourth convolution layer, the third convolution layer includes a third preset convolution kernel, and the fourth convolution layer includes a fourth preset convolution kernel;

[0153] The second output unit is specifically used for:

[0154] Processing the second vector through the first preset convolution kernel to obtain a first matrix, and processing the second vector through the second preset convolution kernel to obtain a second matrix;

[0155] Add the first matrix, the second matrix and the first identity matrix to obtain a third matrix;

[0156] Normalizing the third matrix to obtain a third vector;

[0157] Processing the third vector through the third preset convolution kernel to obtain a fifth matrix, and processing the third vector through the fourth preset convolution kernel to obtain a sixth matrix;

[0158] Add the fifth matrix, the sixth matrix and the second unit matrix to obtain a seventh matrix;

[0159] Normalizing the seventh matrix to obtain the encoding vector;

[0160] Among them, the number of rows and columns of the first unit matrix are the same as the number of rows and columns of the convolution kernel with the least elements in the first convolution processing module, and the number of rows and columns of the second unit matrix are the same as the number of rows and columns of the convolution kernel with the least elements in the second convolution processing module.

[0161] Optionally, the decoder of the third speech synthesis model includes a third convolution processing module and a fourth convolution processing module, the third convolution processing module includes a fifth convolution layer and a sixth convolution layer, the fifth convolution layer includes a fifth preset convolution kernel, the sixth convolution layer includes a sixth preset convolution kernel, the fourth convolution processing module includes a seventh convolution layer and an eighth convolution layer, the seventh convolution layer includes a seventh preset convolution kernel, and the eighth convolution layer includes an eighth preset convolution kernel;

[0162] The fourth output unit is specifically used for:

[0163] Processing the phoneme duration matrix through the fifth preset convolution kernel to obtain an eighth matrix, and processing the phoneme duration matrix through the sixth preset convolution kernel to obtain a ninth matrix;

[0164] Adding the eighth matrix, the ninth matrix and the third unit matrix to obtain a tenth matrix;

[0165] Normalizing the tenth matrix to obtain a fourth vector;

[0166] Processing the fourth vector through the seventh preset convolution kernel to obtain an eleventh matrix, and processing the fourth vector through the eighth preset convolution kernel to obtain a twelfth matrix;

[0167] Add the eleventh matrix, the twelfth matrix and the fourth unit matrix to obtain a thirteenth matrix;

[0168] Normalizing the thirteenth matrix to obtain speech acoustic features of the phoneme corresponding to the second vector;

[0169] Among them, the number of rows and columns of the third unit matrix are the same as the number of rows and columns of the convolution kernel with the least elements in the third convolution processing module, and the number of rows and columns of the fourth unit matrix are the same as the number of rows and columns of the convolution kernel with the least elements in the fourth convolution processing module.

[0170] From the above, it can be seen that in an embodiment of the present application, multiple text samples can be obtained; the text samples are trained to obtain a first speech synthesis model, wherein the first speech synthesis model includes multiple convolution processing modules, each convolution processing module includes multiple parallel convolution layers; when i takes each integer from 1 to n, a convolution layer equivalent to the multiple parallel convolution layers included in the i-th convolution processing module is determined as the i-th target convolution layer, wherein n is the number of convolution processing modules; the multiple parallel convolution layers included in the i-th convolution processing module of the first speech synthesis model are replaced with the i-th target convolution layer; when the multiple parallel convolution layers included in the n-th convolution processing module are replaced with the n-th target convolution layer, a second speech synthesis model is obtained.

[0171] Among them, the first speech synthesis model adopts a multi-channel model structure in the training stage, and in the generation stage, through identity transformation, without changing the input and output, the multiple parallel convolutional layers included in the multiple convolution processing modules in the first speech synthesis model are equivalently transformed into a target convolutional layer, thereby obtaining the second speech synthesis model, that is, converting the first speech synthesis model of the multi-channel model structure into the second speech synthesis model of the single-channel model structure. Therefore, in the embodiment of the present application, the second speech synthesis model retains the excellent performance of the multi-channel model structure in the training stage, helping the model to converge quickly, and has the advantage of the single-channel model structure being lightweight in the deployment stage, thereby solving the problem that the model composed of Attention and multi-channel parallel structures will occupy a large amount of memory in later applications.

[0172] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0173] The present application also provides an electronic device, including:

[0174] One or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, enable the electronic device to perform the method described in the embodiments of the present application.

[0175] The embodiments of the present application also provide one or more machine-readable media on which instructions are stored, which, when executed by one or more processors, enable the processors to execute the methods described in the embodiments of the present application.

[0176] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0177] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0179] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0181] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0182] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0183] The above is a detailed introduction to a method and device for generating a speech synthesis model provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for generating a speech synthesis model, characterized in that: The method comprises: Get multiple text samples; Training the text sample to obtain a first speech synthesis model, wherein the first speech synthesis model includes a plurality of convolution processing modules, and each convolution processing module includes a plurality of parallel convolution layers; When i takes each integer from 1 to n, determine a convolution layer equivalent to the multiple parallel convolution layers included in the i-th convolution processing module as the i-th target convolution layer, where n is the number of the convolution processing modules; Replacing the multiple parallel convolutional layers included in the i-th convolutional processing module of the first speech synthesis model with the i-th target convolutional layer; When the multiple parallel convolutional layers included in the nth convolutional processing module are replaced by the nth target convolutional layer, a second speech synthesis model is obtained.

2. The method according to claim 1, characterized in that After obtaining the second speech synthesis model, the method further includes: Get the text to be processed; Obtaining a phoneme sequence of the text to be processed; Obtaining a feature vector of each phoneme in the phoneme sequence and a position encoding vector of each phoneme in the phoneme sequence; The feature vector and the position encoding vector are input into the second speech synthesis model, and the speech acoustic features of the text to be processed are output.

3. The method according to claim 1, characterized in that The first speech synthesis model includes an encoder, a duration prediction module, and a decoder, wherein the encoder includes at least one convolution processing module and the decoder includes at least one convolution processing module.

4. The method according to claim 1, characterized in that: Each convolution layer includes a convolution kernel, and the convolution kernel included in the i-th target convolution layer is the sum of the convolution kernels of multiple convolution layers included in the i-th convolution processing module and the i-th unit matrix, wherein the number of rows and columns of the i-th unit matrix is ​​the same as the number of rows and columns of the convolution kernel with the least elements in the i-th convolution processing module.

5. The method according to claim 1, characterized in that Training the text sample to obtain a first speech synthesis model includes: Obtaining a phoneme sequence of the text sample; Obtaining a feature vector of each phoneme in the phoneme sequence and a position encoding vector of each phoneme in the phoneme sequence; Adding the feature vector and the position encoding vector of the same phoneme to obtain a plurality of first vectors; Using the predetermined parameters of the third speech synthesis model, the first vector of the same text sample is processed to obtain speech acoustic features corresponding to each text sample; Modifying the parameters of the third speech synthesis model according to the obtained speech acoustic features to obtain the parameters of the fourth speech synthesis model; The parameters of the fourth speech synthesis model are used to process the first vector of the same text sample until the speech acoustic features corresponding to the obtained text sample meet preset conditions, and the speech synthesis model that meets the preset conditions is determined as the first speech synthesis model.

6. The method according to claim 5, characterized in that The third speech synthesis model includes an encoder, a duration prediction module and a decoder; The process of processing one of the first vectors of a text sample using the predetermined parameters of the third speech synthesis model includes: Inputting the second vector into an encoder of the third speech synthesis model, and outputting an encoded vector, wherein the second vector is one of the first vectors of one of the text samples; Input the encoding vector into the duration prediction module of the third speech synthesis model, and output a phoneme duration matrix obtained by extending the encoding vector according to the duration of the phoneme corresponding to the second vector; The phoneme duration matrix is ​​input into the decoder of the third speech synthesis model, and the speech acoustic features of the phonemes corresponding to the second vector are output.

7. The method according to claim 6, characterized in that The encoder of the third speech synthesis model includes a first convolution processing module and a second convolution processing module, the first convolution processing module includes a first convolution layer and a second convolution layer, the first convolution layer includes a first preset convolution kernel, the second convolution layer includes a second preset convolution kernel, the second convolution processing module includes a third convolution layer and a fourth convolution layer, the third convolution layer includes a third preset convolution kernel, and the fourth convolution layer includes a fourth preset convolution kernel; The step of inputting the second vector to the encoder of the third speech synthesis model and outputting the encoded vector comprises: Processing the second vector through the first preset convolution kernel to obtain a first matrix, and processing the second vector through the second preset convolution kernel to obtain a second matrix; Add the first matrix, the second matrix and the first identity matrix to obtain a third matrix; Normalizing the third matrix to obtain a third vector; Processing the third vector through the third preset convolution kernel to obtain a fifth matrix, and processing the third vector through the fourth preset convolution kernel to obtain a sixth matrix; Add the fifth matrix, the sixth matrix and the second unit matrix to obtain a seventh matrix; Normalizing the seventh matrix to obtain the encoding vector; Among them, the number of rows and columns of the first unit matrix are the same as the number of rows and columns of the convolution kernel with the least elements in the first convolution processing module, and the number of rows and columns of the second unit matrix are the same as the number of rows and columns of the convolution kernel with the least elements in the second convolution processing module.

8. The method according to claim 6, characterized in that The decoder of the third speech synthesis model includes a third convolution processing module and a fourth convolution processing module, the third convolution processing module includes a fifth convolution layer and a sixth convolution layer, the fifth convolution layer includes a fifth preset convolution kernel, the sixth convolution layer includes a sixth preset convolution kernel, the fourth convolution processing module includes a seventh convolution layer and an eighth convolution layer, the seventh convolution layer includes a seventh preset convolution kernel, and the eighth convolution layer includes an eighth preset convolution kernel; The step of inputting the phoneme duration matrix into a decoder of the third speech synthesis model and outputting speech acoustic features of the phonemes corresponding to the second vector comprises: Processing the phoneme duration matrix through the fifth preset convolution kernel to obtain an eighth matrix, and processing the phoneme duration matrix through the sixth preset convolution kernel to obtain a ninth matrix; Adding the eighth matrix, the ninth matrix and the third identity matrix to obtain a tenth matrix; Normalizing the tenth matrix to obtain a fourth vector; Processing the fourth vector through the seventh preset convolution kernel to obtain an eleventh matrix, and processing the fourth vector through the eighth preset convolution kernel to obtain a twelfth matrix; Add the eleventh matrix, the twelfth matrix and the fourth unit matrix to obtain a thirteenth matrix; Normalizing the thirteenth matrix to obtain speech acoustic features of the phoneme corresponding to the second vector; Among them, the number of rows and columns of the third unit matrix are the same as the number of rows and columns of the convolution kernel with the least elements in the third convolution processing module, and the number of rows and columns of the fourth unit matrix are the same as the number of rows and columns of the convolution kernel with the least elements in the fourth convolution processing module.

9. A device for generating a speech synthesis model, characterized in that: The device comprises: A sample acquisition module, used to acquire multiple text samples; A model acquisition module, used to train the text sample to obtain a first speech synthesis model, wherein the first speech synthesis model includes a plurality of convolution processing modules, each of which includes a plurality of parallel convolution layers; an equivalent transformation module, used for determining a convolution layer equivalent to the multiple parallel convolution layers included in the i-th convolution processing module as the i-th target convolution layer when i takes each integer from 1 to n, wherein n is the number of the convolution processing modules; A replacement module, used to replace the multiple parallel convolutional layers included in the i-th convolutional processing module of the first speech synthesis model with the i-th target convolutional layer; The model generation module is used to obtain a second speech synthesis model when the multiple parallel convolutional layers included in the nth convolutional processing module are replaced by the nth target convolutional layer.

10. An electronic device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method for generating a speech synthesis model as claimed in any one of claims 1 to 8.

Images