Speech synthesis and training method and device of speech synthesis model

Abstract

The present disclosure provides a method, device, electronic device and readable storage medium for speech synthesis and training of speech synthesis models, and relates to the fields of artificial intelligence technology such as deep learning and natural language processing. The speech synthesis method includes: obtaining a text to be processed, and obtaining a phoneme sequence of the text to be processed; obtaining multiple frames of first features according to the phoneme sequence; obtaining a second feature corresponding to the first feature of each frame according to the first feature of each frame and the adjacent feature frames of the first feature of each frame; obtaining audio data corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame. The training method of the speech synthesis model includes: obtaining a data set; constructing a neural network model including an encoder, a decoder and a vocoder; training the neural network model based on multiple training texts and training audio data of multiple training texts until the neural network model converges to obtain a speech synthesis model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology such as deep learning and natural language processing. Provided are a method, device, electronic device and readable storage medium for training speech synthesis and speech synthesis models. Background Art

[0002] In speech synthesis tasks, usually after inputting the text to be processed, the audio data corresponding to the text to be processed is output. However, in the existing speech synthesis method, the return of the first audio data needs to wait until the entire calculation is completed before it can start, which cannot meet the low-latency application scenarios and the response time of speech synthesis is long. Summary of the invention

[0003] According to a first aspect of the present disclosure, a speech synthesis method is provided, comprising: acquiring a text to be processed and obtaining a phoneme sequence of the text to be processed; obtaining multiple frames of first features according to the phoneme sequence; obtaining a second feature corresponding to the first feature of each frame according to the first feature of each frame and the adjacent feature frames of the first feature of each frame; obtaining audio data corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame.

[0004] According to a second aspect of the present disclosure, a method for training a speech synthesis model is provided, comprising: obtaining a data set, wherein the data set comprises a plurality of training texts and training audio data of the plurality of training texts; constructing a neural network model comprising an encoder, a decoder and a vocoder, wherein the encoder is used to obtain a plurality of frames of predicted first features according to a phoneme sequence of the training texts, the decoder is used to obtain a predicted second feature corresponding to the predicted first feature of each frame according to the predicted first feature of each frame and the adjacent feature frames of the predicted first feature of each frame, and the vocoder is used to obtain a predicted audio data corresponding to the predicted second feature of each frame according to the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame; and training the neural network model based on the plurality of training texts and the training audio data of the plurality of training texts until the neural network model converges to obtain the speech synthesis model.

[0005] According to a third aspect of the present disclosure, a speech synthesis device is provided, comprising: a first acquisition unit, used to acquire a text to be processed and obtain a phoneme sequence of the text to be processed; a first processing unit, used to obtain multiple frames of first features according to the phoneme sequence; a second processing unit, used to obtain a second feature corresponding to the first feature of each frame according to the first feature of each frame and the adjacent feature frames of the first feature of each frame; and a synthesis unit, used to obtain audio data corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame.

[0006] According to a fourth aspect of the present disclosure, a training device for a speech synthesis model is provided, comprising: a second acquisition unit, used to acquire a data set, wherein the data set includes multiple training texts and training audio data of the multiple training texts; a construction unit, used to construct a neural network model including an encoder, a decoder and a vocoder, wherein the encoder is used to obtain a multi-frame predicted first feature according to a phoneme sequence of the training text, the decoder is used to obtain a predicted second feature corresponding to the predicted first feature of each frame according to the predicted first feature of each frame and the adjacent feature frames of the predicted first feature of each frame, and the vocoder is used to obtain a predicted audio data corresponding to the predicted second feature of each frame according to the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame; and a training unit, used to train the neural network model based on the multiple training texts and the training audio data of the multiple training texts until the neural network model converges to obtain the speech synthesis model.

[0007] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0008] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method as described above.

[0009] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method as described above when executed by a processor.

[0010] It can be seen from the above technical solutions that the present invention obtains the second features and audio data respectively by combining adjacent feature frames, and can fully consider the boundaries of each feature, thereby improving the coherence between the audio data corresponding to different second features. In addition, this embodiment can achieve the purpose of streaming synthesized speech by performing speech synthesis on each second feature, thereby reducing the response time of speech synthesis and improving the speed of speech synthesis.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0014] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0016] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0018] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0019] Figure 7 It is a block diagram of an electronic device used to implement the speech synthesis or speech synthesis model training method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and mechanisms is omitted in the following description.

[0021] Figure 1 Schematic diagram of the first embodiment of the present disclosure. Figure 1 As shown, the speech synthesis method of this embodiment specifically includes the following steps:

[0022] S101, obtaining a text to be processed and obtaining a phoneme sequence of the text to be processed;

[0023] S102, obtaining a plurality of first features of the frames according to the phoneme sequence;

[0024] S103, obtaining a second feature corresponding to the first feature of each frame according to the first feature of each frame and the feature frames adjacent to the first feature of each frame;

[0025] S104. Obtain audio data corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame.

[0026] The speech synthesis method of this embodiment obtains the second feature and the audio data respectively by combining adjacent feature frames, and can fully consider the boundary of each feature, so as to improve the coherence between the obtained audio data corresponding to different second features. In addition, this embodiment can achieve the purpose of streaming synthesized speech by performing speech synthesis on each second feature, thereby reducing the response time of speech synthesis and improving the speed of speech synthesis.

[0027] In this embodiment, when executing S101 to obtain the text to be processed, the text input by the input end or the text selected by the input end may be used as the text to be processed.

[0028] When executing S101 to obtain the phoneme sequence of the text to be processed, this embodiment can first obtain the phonemes corresponding to each character in the text to be processed, and then construct the phoneme sequence of the text to be processed based on the phonemes corresponding to all the characters; wherein, before executing S101 to obtain the phoneme sequence of the text to be processed, this embodiment can also remove punctuation marks in the text to be processed.

[0029] In this embodiment, after executing S101 to obtain the phoneme sequence of the text to be processed, S102 is executed to obtain the first features of multiple frames according to the phoneme sequence.

[0030] Specifically, when executing S102 in this embodiment to obtain the first features of multiple frames according to the phoneme sequence, an optional implementation method that can be adopted is: obtaining encoder features corresponding to the phoneme sequence; segmenting the obtained encoder features, and using the segmentation results as the first features of multiple frames.

[0031] That is to say, this embodiment divides the encoder features corresponding to the phoneme sequence, and uses each block of encoder features obtained by the division as the first feature of a frame, so that the subsequent process of obtaining the second feature and audio data is based on the first feature of each frame, thereby achieving the purpose of streaming synthesized speech.

[0032] In this embodiment, when executing S102 to divide the encoder feature, the encoder feature can be divided into multiple first features according to a preset size, and the preset size can be manually set.

[0033] In addition, when executing S102 to obtain multiple frames of first features according to the phoneme sequence, this embodiment can also input the phoneme sequence into a speech synthesis model, and the encoder in the speech synthesis model processes the phoneme sequence to output multiple frames of first features.

[0034] After executing S102 to obtain multiple frames of first features, this embodiment executes S103 to obtain a second feature corresponding to the first feature of each frame based on the first feature of each frame and the adjacent feature frames of the first feature of each frame; wherein the second feature obtained in this embodiment is an acoustic feature, such as a Melspectrogram, and audio data can be generated using the acoustic feature.

[0035] Specifically, when executing S103, the present embodiment obtains the second feature corresponding to the first feature of each frame according to the first feature of each frame and the adjacent feature frames of the first feature of each frame. An optional implementation method may be adopted: determining the adjacent feature frames of the first feature of each frame from multiple frames of first features; intercepting filling features from the adjacent feature frames of the first feature of each frame, and filling the intercepted filling features into the first feature of each frame; and obtaining the second feature corresponding to the first feature of each frame according to the first feature of each frame after filling.

[0036] That is to say, this embodiment uses the filling features intercepted from the adjacent feature frames of the first feature of each frame to fill the first feature of each frame, so as to obtain the second feature corresponding to the first feature of each frame. Since the first feature of each frame after the filling process contains some features in the adjacent feature frames, the boundary of the first feature of each frame can be accurately calculated, thereby improving the accuracy of the second feature corresponding to the first feature of each frame.

[0037] In this embodiment, when executing S102 to intercept filling features from the adjacent feature frames of the first feature of each frame and fill the intercepted filling features into the first feature of each frame, the feature located at the first preset position of the adjacent feature frame can be intercepted as the filling feature based on the positional relationship between the first feature of each frame and the adjacent feature frame, and the intercepted filling features can be filled into the second preset position of the first feature of each frame.

[0038] In this embodiment, there is a corresponding relationship between the first preset position and the second preset position. If the first preset position is the beginning, the second preset position is the end; if the first preset position is the end, the second preset position is the beginning.

[0039] For example, if the multi-frame first features obtained by executing S102 in this embodiment include EF1, EF2 and EF3, this embodiment executes S103 to determine that the adjacent feature frame of EF1 is EF2, determines that the adjacent feature frames of EF2 are EF1 and EF3, and determines that the adjacent feature frame of EF3 is EF2; for EF1, EF2 is located after EF1, then the features at the beginning of EF2 are intercepted as filling features and filled to the end of EF1; for EF2, EF3 is located after EF2, then the features at the beginning of EF3 are intercepted as filling features and filled to the end of EF2; EF1 is located before EF2, then the features at the end of EF1 are intercepted as filling features and filled to the beginning of EF2; for EF3, EF2 is located before EF3, then the features at the end of EF2 are intercepted as filling features and filled to the beginning of EF3.

[0040] When executing S103 in this embodiment to obtain the second feature corresponding to the first feature of each frame according to the first feature of each frame after padding, an optional implementation method that can be adopted is: obtaining the decoder feature of the first feature of each frame after padding; removing the decoder feature corresponding to the padding feature in the decoder feature, and obtaining the second feature corresponding to the first feature of each frame.

[0041] That is to say, when obtaining the second feature corresponding to the first feature of each frame, this embodiment can also remove the decoder features corresponding to the filling features in the decoder features of the first feature, so as to further improve the accuracy of the obtained second feature and avoid duplication in the different audio data finally generated.

[0042] In addition, when executing S103, the present embodiment obtains the second feature corresponding to the first feature of each frame based on the first feature of each frame and the adjacent feature frames of the first feature of each frame. The first feature of each frame and the adjacent feature frames of the first feature of each frame can also be input into the speech synthesis model, and the decoder in the speech synthesis model processes the first feature of each frame and the adjacent feature frames of the first feature of each frame to output the second feature corresponding to the first feature of each frame.

[0043] After executing S103 to obtain the second feature corresponding to the first feature of each frame, this embodiment executes S104 to obtain audio data corresponding to the second feature of each frame based on the second feature of each frame and the adjacent feature frames of the second feature of each frame.

[0044] Specifically, when this embodiment executes S104 to obtain audio data corresponding to the second feature of each frame based on the second feature of each frame and the adjacent feature frames of the second feature of each frame, an optional implementation method that can be adopted is: determine the adjacent feature frames of the second feature of each frame from multiple frames of second features; intercept fill features from the adjacent feature frames of the second feature of each frame, and fill the intercepted fill features into the second feature of each frame; obtain the audio data corresponding to the second feature of each frame based on the second feature of each frame after filling.

[0045] Similarly, when executing S103, this embodiment intercepts filling features from the adjacent feature frames of the second feature of each frame and fills the intercepted filling features into the second feature of each frame. According to the positional relationship between the second feature of each frame and the adjacent feature frame, the feature located at the first preset position of the adjacent feature frame can be intercepted as the filling feature, and the intercepted filling features can be filled into the second preset position of the second feature of each frame; there is a corresponding relationship between the first preset position and the second preset position.

[0046] That is to say, this embodiment uses the filling features intercepted from the adjacent feature frames of the second feature of each frame to fill the second feature of each frame, so as to obtain the audio data corresponding to the second feature of each frame. Since the second feature of each frame after the filling process contains some features in the adjacent feature frames, the boundary of the second feature of each frame can be accurately calculated, thereby improving the coherence between the audio data corresponding to different second features.

[0047] When executing S104 in this embodiment to obtain audio data corresponding to the second feature of each frame according to the second feature of each frame after filling, an optional implementation method that can be adopted is: obtaining the audio data of the second feature of each frame after filling; removing the audio data corresponding to the filling feature in the audio data, and obtaining audio data corresponding to the second feature of each frame.

[0048] That is to say, when obtaining the audio data corresponding to the second feature of each frame, this embodiment can also remove the audio data corresponding to the filling feature in the audio data to avoid duplication between the generated audio data corresponding to different second features.

[0049] In addition, when executing S104, the present embodiment obtains the audio data corresponding to the second feature of each frame based on the second feature of each frame and the adjacent feature frames of the second feature of each frame. The second feature of each frame and the adjacent feature frames of the second feature of each frame can also be input into a speech synthesis model. The voice coder in the speech synthesis model processes the second feature of each frame and the adjacent feature frames of the second feature of each frame to obtain the audio data corresponding to the second feature of each frame.

[0050] After executing S103 to obtain the audio data corresponding to the second feature of each frame, this embodiment can return the obtained audio data to the input end in sequence. Since the purpose of streaming speech synthesis is achieved, this embodiment can reduce the response time of speech synthesis and improve the speed of speech synthesis.

[0051] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure. Figure 2 As shown, the training method of the speech synthesis model of this embodiment specifically includes the following steps:

[0052] S201, obtaining a data set, wherein the data set includes a plurality of training texts and training audio data of the plurality of training texts;

[0053] S202, constructing a neural network model including an encoder, a decoder and a vocoder, wherein the encoder is used to obtain a multi-frame predicted first feature according to a phoneme sequence of a training text, the decoder is used to obtain a predicted second feature corresponding to the predicted first feature of each frame according to the predicted first feature of each frame and the adjacent feature frames of the predicted first feature of each frame, and the vocoder is used to obtain predicted audio data corresponding to the predicted second feature of each frame according to the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame;

[0054] S203. Train the neural network model based on the multiple training texts and the training audio data of the multiple training texts until the neural network model converges to obtain the speech synthesis model.

[0055] The training method of the speech synthesis model of this embodiment, the decoder and the vocoder in the constructed neural network model, obtain the predicted second feature and the predicted audio data by combining adjacent feature frames, which can improve the accuracy of the predicted second feature and the coherence between different predicted audio data, and generate corresponding audio data according to different feature frames, so that the speech synthesis model can achieve the purpose of streaming speech synthesis, thereby enhancing the synthesis effect of the speech synthesis model.

[0056] After executing S201 to obtain training audio data including multiple training texts and the multiple training texts, this embodiment can also divide the data set into a training set, an evaluation set and a test set; wherein the training set is used to train the neural network model, the evaluation set is used to evaluate the neural network model during the training process, and the test set is used to test the effect of the trained neural network model.

[0057] After executing S201 to obtain the data set, this embodiment can also extract training acoustic features from the training audio data, and then align the phoneme sequence of the training text with the extracted training acoustic features; the purpose of extracting training acoustic features from the training audio data is to calculate the loss function of the encoder and decoder in the neural network model, and then update the parameters in the encoder and decoder according to the loss function.

[0058] In this embodiment, after executing S201 to obtain a data set, S202 is executed to construct a neural network model including an encoder, a decoder, and a vocoder.

[0059] In the neural network model constructed by executing S202 in this embodiment, the encoder and the decoder constitute an acoustic model, which is used to convert the input text into acoustic features, such as Mel spectrum features; the structure of the acoustic model in this embodiment is based on the FastSpeech2 structure.

[0060] In the neural network model constructed by executing S202 in this embodiment, the encoder is used to obtain the first feature of multi-frame prediction based on the phoneme sequence of the training text.

[0061] In the neural network model constructed by executing S202 in this embodiment, the decoder is used to obtain a predicted second feature corresponding to the predicted first feature of each frame based on the predicted first feature of each frame and the adjacent feature frames of the predicted first feature of each frame, and the predicted second feature is, for example, an acoustic feature of a Mel spectrum.

[0062] Specifically, in the neural network model constructed by executing S202 in this embodiment, when the decoder obtains the predicted second feature corresponding to the predicted first feature of each frame based on the predicted first feature of each frame and the adjacent feature frames of the predicted first feature of each frame, an optional implementation method that can be adopted is: determining the adjacent feature frames of the predicted first feature of each frame from the predicted first features of multiple frames; intercepting filling features from the adjacent feature frames of the predicted first feature of each frame, and filling the intercepted filling features into the predicted first feature of each frame; and obtaining the predicted second feature corresponding to the predicted first feature of each frame based on the predicted first feature of each frame after filling.

[0063] In the neural network model constructed by executing S202 of the embodiment, when the decoder predicts the first feature based on each frame after padding and obtains the predicted second feature corresponding to the predicted first feature of each frame, an optional implementation method that can be adopted is: obtaining the decoder features of the predicted first feature of each frame after padding; removing the decoder features corresponding to the padding features in the decoder features, and obtaining the predicted second features corresponding to the predicted first feature of each frame.

[0064] In the existing FastSpeech2 structure, the decoder is usually composed of n layers of FeedForward Transformer (FFT) blocks, but the FFT block needs to rely on the entire sequence when processing, and it cannot achieve streaming synthesis of speech. Therefore, this embodiment improves the decoder in the neural network model so that the decoder can achieve streaming speech synthesis.

[0065] Specifically, in the neural network model constructed by executing S202 in this embodiment, the decoder is composed of a plurality of layers of one-dimensional convolution residual blocks (Conv1D Residual Block), and each layer of one-dimensional convolution residual blocks sequentially includes a one-dimensional convolution layer (Conv1D), an activation function layer (Relu), a one-dimensional batch normalization layer (BatchNorm1D) and a one-dimensional convolution layer (Conv1D).

[0066] Since the structure of the one-dimensional convolution residual block does not rely on the entire sequence for decoding, but relies on the local sequence, this embodiment uses a decoder composed of a one-dimensional convolution residual block, which can decode the first feature of the multi-frame prediction after segmentation, thereby realizing streaming synthesis of speech.

[0067] In addition, in the neural network model constructed by executing S202 in this embodiment, the decoder is composed of a multi-layer feedforward Transformer block, and each layer of the feedforward Transformer block sequentially includes a chunk-based attention layer, an addition & normalization layer (Add&Norm), a causal convolution layer (Causal Convolution) and an addition & normalization layer (Add&Norm).

[0068] That is to say, this embodiment can also improve the original FFT block, replace the multi-head attention layer (Multi-Head Attention) in the FFT block with a chunk-based attention layer, and replace the one-dimensional convolution layer (Conv1D) with (Causal Convolution), so that the decoder based on the improved FFT block can realize streaming synthesis of speech.

[0069] In this embodiment, there are multiple options for the size and position of the chunks used in the chunk-based attention layer, which can be fixed-size chunks or dynamic-size chunks; each chunk can focus only on the values ​​within its own chunk, or only on the values ​​within the chunks from the beginning to itself, or only on the values ​​within itself and the chunks to its left.

[0070] In the neural network model constructed by executing S202 in this embodiment, the vocoder is used to obtain predicted audio data corresponding to the predicted second feature of each frame based on the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame.

[0071] Specifically, in the neural network model constructed by executing S202 of this embodiment, when the voice coder obtains the predicted audio data corresponding to the predicted second feature of each frame based on the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame, an optional implementation method that can be adopted is: determining the adjacent feature frames of the predicted second feature of each frame from multiple frames of predicted second features; intercepting filling features from the adjacent feature frames of the predicted second feature of each frame, and filling the intercepted filling features into the predicted second feature of each frame; and obtaining the predicted audio data corresponding to the predicted second feature of each frame based on the predicted second feature of each frame after filling.

[0072] In the neural network model constructed by executing S202 in this embodiment, when the voice coder predicts the second feature of each frame after padding and obtains the predicted audio data corresponding to the predicted second feature of each frame, an optional implementation method that can be used is: obtaining the predicted audio data of the second feature predicted for each frame after padding; removing the predicted audio data corresponding to the padding feature in the predicted audio data, and obtaining the predicted audio data corresponding to the predicted second feature of each frame.

[0073] After executing S202 to construct a neural network model including an encoder, a decoder and a vocoder, this embodiment executes S203 to train the neural network model based on multiple training texts and training audio data of the multiple training texts until the neural network model converges to obtain a speech synthesis model.

[0074] Specifically, when executing S203 to train a neural network model based on multiple training texts and training audio data of multiple training texts, the present embodiment may adopt an optional implementation method as follows: respectively inputting the phoneme sequences of multiple training texts into the neural network model, and obtaining predicted audio data output by the neural network model for each training text; calculating a first loss function value based on the training audio data of the multiple training texts and the predicted audio data; and adjusting the parameters of the encoder, decoder and vocoder in the neural network model based on the calculated first loss function value until the first loss function value converges to obtain a speech synthesis model.

[0075] In addition, in the data set obtained when executing S201 in the present embodiment, if the training acoustic features corresponding to the phoneme sequences of the training texts are also obtained, then when executing S203 in the present embodiment to train the neural network model based on multiple training texts and the training audio data of multiple training texts, an optional implementation method that can be adopted is: inputting the phoneme sequences of the multiple training texts into the neural network model respectively, and obtaining the predicted second features and predicted audio data output by the neural network model for each training text; calculating the first loss function value according to the training audio data and the predicted audio data of the multiple training texts, and calculating the second loss function value according to the predicted second features and the training acoustic features of the multiple training texts; adjusting the parameters of the vocoder in the neural network model according to the calculated first loss function value, and adjusting the parameters of the encoder and decoder in the neural network model according to the calculated second loss function value, until the first loss function value and the second loss function value converge to obtain a speech synthesis model.

[0076] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure. Figure 3 FIG. 4 shows a flow chart of the present embodiment when obtaining the first feature of multiple frames and obtaining the second feature of multiple frames. Figure 3 EF in represents the first feature, and DF represents the second feature.

[0077] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure. Figure 4 FIG. 4 shows a flowchart of the present embodiment when obtaining audio data corresponding to multiple frames of the second feature. Figure 4 The M in represents the second feature, and W represents audio data.

[0078] Figure 5 is a schematic diagram according to the fifth embodiment of the present disclosure. Figure 5 As shown, the speech synthesis device 500 of this embodiment includes:

[0079] A first acquisition unit 501 is used to acquire a text to be processed and obtain a phoneme sequence of the text to be processed;

[0080] A first processing unit 502 is used to obtain a plurality of first features of the frames according to the phoneme sequence;

[0081] The second processing unit 503 is used to obtain a second feature corresponding to the first feature of each frame according to the first feature of each frame and the adjacent feature frames of the first feature of each frame;

[0082] The synthesis unit 504 is used to obtain audio data corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame.

[0083] When the first acquiring unit 501 acquires the text to be processed, the text input by the input end or the text selected by the input end may be used as the text to be processed.

[0084] When the first acquisition unit 501 obtains the phoneme sequence of the text to be processed, it can first obtain the phonemes corresponding to each character in the text to be processed, and then construct the phoneme sequence of the text to be processed based on the phonemes corresponding to all the characters; wherein, before obtaining the phoneme sequence of the text to be processed, the first acquisition unit 501 can also remove punctuation marks in the text to be processed.

[0085] In this embodiment, after the first acquisition unit 501 obtains the phoneme sequence of the text to be processed, the first processing unit 502 obtains multiple frames of first features according to the phoneme sequence.

[0086] Specifically, when the first processing unit 502 obtains the first features of multiple frames according to the phoneme sequence, an optional implementation method that can be adopted is: obtaining encoder features corresponding to the phoneme sequence; segmenting the obtained encoder features, and using the segmentation results as the first features of multiple frames.

[0087] That is to say, this embodiment divides the encoder features corresponding to the phoneme sequence, and uses each block of encoder features obtained by the division as the first feature of a frame, so that the subsequent process of obtaining the second feature and audio data is based on the first feature of each frame, thereby achieving the purpose of streaming synthesized speech.

[0088] When dividing the encoder feature, the first processing unit 502 may divide the encoder feature into a plurality of first features according to a preset size, and the preset size may be manually set.

[0089] In addition, when the first processing unit 502 obtains the first features of multiple frames based on the phoneme sequence, the phoneme sequence can also be input into the speech synthesis model, and the encoder in the speech synthesis model processes the phoneme sequence to obtain the first features of multiple frames.

[0090] In this embodiment, after the first processing unit 502 obtains multiple frames of first features, the second processing unit 503 obtains the second feature corresponding to the first feature of each frame based on the first feature of each frame and the adjacent feature frames of the first feature of each frame; wherein the second feature obtained by the second processing unit 503 is an acoustic feature, and audio data can be generated using the acoustic feature.

[0091] Specifically, when the second processing unit 503 obtains the second feature corresponding to the first feature of each frame based on the first feature of each frame and the adjacent feature frames of the first feature of each frame, an optional implementation method may be adopted: determining the adjacent feature frames of the first feature of each frame from multiple frames of first features; intercepting filling features from the adjacent feature frames of the first feature of each frame, and filling the intercepted filling features into the first feature of each frame; and obtaining the second feature corresponding to the first feature of each frame based on the first feature of each frame after filling.

[0092] That is to say, this embodiment uses the filling features intercepted from the adjacent feature frames of the first feature of each frame to fill the first feature of each frame, so as to obtain the second feature corresponding to the first feature of each frame. Since the first feature of each frame after the filling process contains some features in the adjacent feature frames, the boundary of the first feature of each frame can be accurately calculated, thereby improving the accuracy of the second feature corresponding to the first feature of each frame.

[0093] When the second processing unit 503 intercepts filling features from the adjacent feature frames of the first feature of each frame and fills the intercepted filling features into the first feature of each frame, it can intercept the features located at the first preset position of the adjacent feature frame as the filling features based on the positional relationship between the first feature of each frame and the adjacent feature frame, and fill the intercepted filling features into the second preset position of the first feature of each frame.

[0094] In this embodiment, there is a corresponding relationship between the first preset position and the second preset position. If the first preset position is the beginning, the second preset position is the end; if the first preset position is the end, the second preset position is the beginning.

[0095] When the second processing unit 503 obtains the second feature corresponding to the first feature of each frame based on the first feature of each frame after padding, an optional implementation method that can be adopted is: obtaining the decoder feature of the first feature of each frame after padding; removing the decoder feature corresponding to the padding feature in the decoder feature, and obtaining the second feature corresponding to the first feature of each frame.

[0096] That is to say, when obtaining the second feature corresponding to the first feature of each frame, this embodiment can also remove the decoder features corresponding to the filling features in the decoder features of the first feature, so as to further improve the accuracy of the obtained second feature and avoid duplication in the different audio data finally generated.

[0097] In addition, when the second processing unit 503 obtains the second feature corresponding to the first feature of each frame based on the first feature of each frame and the adjacent feature frames of the first feature of each frame, it can also input the first feature of each frame and the adjacent feature frames of the first feature of each frame into the speech synthesis model, and the decoder in the speech synthesis model processes the first feature of each frame and the adjacent feature frames of the first feature of each frame to obtain the second feature corresponding to the first feature of each frame.

[0098] In this embodiment, after the second processing unit 503 obtains the second feature corresponding to the first feature of each frame, the synthesis unit 504 obtains audio data corresponding to the second feature of each frame based on the second feature of each frame and the adjacent feature frames of the second feature of each frame.

[0099] Specifically, when the synthesis unit 504 obtains the audio data corresponding to the second feature of each frame based on the second feature of each frame and the adjacent feature frames of the second feature of each frame, an optional implementation method that can be adopted is: determining the adjacent feature frames of the second feature of each frame from multiple frames of second features; intercepting the filling feature from the adjacent feature frames of the second feature of each frame, and filling the intercepted filling feature into the second feature of each frame; and obtaining the audio data corresponding to the second feature of each frame based on the second feature of each frame after filling.

[0100] Similarly, when the synthesis unit 504 intercepts filling features from the adjacent feature frames of the second feature of each frame and fills the intercepted filling features into the second feature of each frame, it can intercept the features located at the first preset position of the adjacent feature frame as the filling features based on the positional relationship between the second feature of each frame and the adjacent feature frame, and fill the intercepted filling features into the second preset position of the second feature of each frame; there is a corresponding relationship between the first preset position and the second preset position.

[0101] That is to say, this embodiment uses the filling features intercepted from the adjacent feature frames of the second feature of each frame to fill the second feature of each frame, so as to obtain the audio data corresponding to the second feature of each frame. Since the second feature of each frame after the filling process contains some features in the adjacent feature frames, the boundary of the second feature of each frame can be accurately calculated, thereby improving the coherence between the audio data corresponding to different second features.

[0102] When the synthesis unit 504 obtains the audio data corresponding to the second feature of each frame according to the second feature of each frame after filling, an optional implementation method that can be adopted is: obtaining the audio data of the second feature of each frame after filling; removing the audio data corresponding to the filling feature in the audio data, and obtaining the audio data corresponding to the second feature of each frame.

[0103] That is to say, when obtaining the audio data corresponding to the second feature of each frame, this embodiment can also remove the audio data corresponding to the filling feature in the audio data to avoid duplication between the generated audio data corresponding to different second features.

[0104] In addition, when the synthesis unit 504 obtains the audio data corresponding to the second feature of each frame based on the second feature of each frame and the adjacent feature frames of the second feature of each frame, it can also input the second feature of each frame and the adjacent feature frames of the second feature of each frame into the speech synthesis model, and the voice coder in the speech synthesis model processes the second feature of each frame and the adjacent feature frames of the second feature of each frame to obtain the audio data corresponding to the second feature of each frame.

[0105] After the synthesis unit 504 obtains the audio data corresponding to the second feature of each frame, it can return the obtained audio data to the input end in sequence. Since the purpose of streaming speech synthesis is achieved, this embodiment can reduce the response time of speech synthesis and improve the speed of speech synthesis.

[0106] Figure 6 is a schematic diagram according to the sixth embodiment of the present disclosure. Figure 6 As shown, the speech synthesis model training device 600 of this embodiment includes:

[0107] A second acquisition unit 601 is used to acquire a data set, wherein the data set includes a plurality of training texts and training audio data of the plurality of training texts;

[0108] A construction unit 602 is used to construct a neural network model including an encoder, a decoder and a vocoder, wherein the encoder is used to obtain a multi-frame predicted first feature according to a phoneme sequence of a training text, the decoder is used to obtain a predicted second feature corresponding to the predicted first feature of each frame according to the predicted first feature of each frame and the adjacent feature frames of the predicted first feature of each frame, and the vocoder is used to obtain predicted audio data corresponding to the predicted second feature of each frame according to the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame;

[0109] The training unit 603 is used to train the neural network model based on the multiple training texts and the training audio data of the multiple training texts until the neural network model converges to obtain the speech synthesis model.

[0110] After the second acquisition unit 601 acquires the training audio data including multiple training texts and the multiple training texts, the data set can also be divided into a training set, an evaluation set and a test set; wherein the training set is used to train the neural network model, the evaluation set is used to evaluate the neural network model during the training process, and the test set is used to test the effect of the trained neural network model.

[0111] After acquiring the data set, the second acquisition unit 601 can also extract training acoustic features from the training audio data, and then align the phoneme sequence of the training text with the extracted training acoustic features; the purpose of extracting training acoustic features from the training audio data is to calculate the loss function of the encoder and decoder in the neural network model, and then update the parameters in the encoder and decoder according to the loss function.

[0112] In this embodiment, after the second acquisition unit 601 acquires the data set, the construction unit 602 constructs a neural network model including an encoder, a decoder and a vocoder.

[0113] In the neural network model constructed by the construction unit 602, the encoder and the decoder constitute an acoustic model, which is used to convert the input text into acoustic features.

[0114] In the neural network model constructed by the construction unit 602, the encoder is used to obtain the first feature of multi-frame prediction based on the phoneme sequence of the training text.

[0115] In the neural network model constructed by the construction unit 602, the decoder is used to obtain a predicted second feature corresponding to the predicted first feature of each frame based on the predicted first feature of each frame and the adjacent feature frames of the predicted first feature of each frame, and the predicted second feature is, for example, an acoustic feature of a Mel spectrum.

[0116] Specifically, in the neural network model constructed by the construction unit 602, when the decoder obtains the predicted second feature corresponding to the predicted first feature of each frame based on the predicted first feature of each frame and the adjacent feature frames of the predicted first feature of each frame, an optional implementation method that can be adopted is: determining the adjacent feature frames of the predicted first feature of each frame from the predicted first features of multiple frames; intercepting the filling feature from the adjacent feature frames of the predicted first feature of each frame, and filling the intercepted filling feature into the predicted first feature of each frame; and obtaining the predicted second feature corresponding to the predicted first feature of each frame based on the predicted first feature of each frame after filling.

[0117] In the neural network model constructed by the construction unit 602, when the decoder predicts the first feature based on each frame after padding and obtains the predicted second feature corresponding to the predicted first feature of each frame, an optional implementation method that can be adopted is: obtaining the decoder features of the predicted first feature of each frame after padding; removing the decoder features corresponding to the padding features in the decoder features, and obtaining the predicted second features corresponding to the predicted first feature of each frame.

[0118] In the neural network model constructed by the construction unit 602, the vocoder is used to obtain predicted audio data corresponding to the predicted second feature of each frame based on the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame.

[0119] Specifically, in the neural network model constructed by the construction unit 602, when the voice coder obtains the predicted audio data corresponding to the predicted second feature of each frame based on the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame, an optional implementation method that can be adopted is: determining the adjacent feature frames of the predicted second feature of each frame from multiple frames of predicted second features; intercepting filling features from the adjacent feature frames of the predicted second feature of each frame, and filling the intercepted filling features into the predicted second feature of each frame; and obtaining the predicted audio data corresponding to the predicted second feature of each frame based on the predicted second feature of each frame after filling.

[0120] In the neural network model constructed by the construction unit 602, when the voice coder predicts the second feature of each frame after padding and obtains the predicted audio data corresponding to the predicted second feature of each frame, an optional implementation method that can be adopted is: obtaining the predicted audio data of the second feature predicted for each frame after padding; removing the predicted audio data corresponding to the padding feature in the predicted audio data, and obtaining the predicted audio data corresponding to the predicted second feature of each frame.

[0121] In this embodiment, after the construction unit 602 constructs the neural network model including the encoder, the decoder and the vocoder, the training unit 603 trains the neural network model based on multiple training texts and the training audio data of the multiple training texts until the neural network model converges to obtain a speech synthesis model.

[0122] Specifically, when the training unit 603 trains a neural network model based on multiple training texts and training audio data of multiple training texts, an optional implementation method that can be adopted is: inputting the phoneme sequences of multiple training texts into the neural network model respectively, and obtaining the predicted audio data output by the neural network model for each training text; calculating the first loss function value according to the training audio data of the multiple training texts and the predicted audio data; adjusting the parameters of the encoder, decoder and vocoder in the neural network model according to the calculated first loss function value until the first loss function value converges to obtain a speech synthesis model.

[0123] In addition, if the second acquisition unit 601 simultaneously acquires the training acoustic features corresponding to the phoneme sequence of the training text, the training unit 603 may adopt the following optional implementation method when training the neural network model based on multiple training texts and the training audio data of multiple training texts: input the phoneme sequences of multiple training texts into the neural network model respectively, and obtain the predicted second features and predicted audio data output by the neural network model for each training text; calculate the first loss function value based on the training audio data and the predicted audio data of the multiple training texts, and calculate the second loss function value based on the predicted second features and the training acoustic features of the multiple training texts; adjust the parameters of the vocoder in the neural network model based on the calculated first loss function value, and adjust the parameters of the encoder and decoder in the neural network model based on the calculated second loss function value, until the first loss function value and the second loss function value converge to obtain a speech synthesis model.

[0124] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0125] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0126] like Figure 7 , is a block diagram of an electronic device according to a method for training speech synthesis or speech synthesis models according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0127] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0128] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0129] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as speech synthesis or a training method for a speech synthesis model. For example, in some embodiments, speech synthesis or a training method for a speech synthesis model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708.

[0130] In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the speech synthesis or speech synthesis model training method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform speech synthesis or speech synthesis model training methods in any other appropriate manner (e.g., by means of firmware).

[0131] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable speech synthesis or speech synthesis model training device, so that the program code, when executed by the processor or controller, causes the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0133] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0135] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0136] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server for a distributed system, or a server combined with a blockchain.

[0137] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0138] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A speech synthesis method, comprising: Acquire a text to be processed and obtain a phoneme sequence of the text to be processed; According to the phoneme sequence, obtaining a plurality of frames of first features; According to the first feature of each frame and the adjacent feature frames of the first feature of each frame, a second feature corresponding to the first feature of each frame is obtained; According to the second feature of each frame and the adjacent feature frames of the second feature of each frame, audio data corresponding to the second feature of each frame is obtained, wherein the adjacent feature frames of the second feature of each frame are two feature frames adjacent to the second feature of each frame; The step of obtaining the second feature corresponding to the first feature of each frame according to the first feature of each frame and the adjacent feature frames of the first feature of each frame includes: Determine adjacent feature frames of the first feature of each frame from the multiple frames; Intercepting a filling feature from a feature frame adjacent to the first feature of each frame, and filling the filling feature into the first feature of each frame; According to the first feature of each frame after filling, the second feature corresponding to the first feature of each frame is obtained.

2. The method according to claim 1, wherein: The obtaining of the first features of multiple frames according to the phoneme sequence comprises: Obtaining encoder features corresponding to the phoneme sequence; The encoder feature is segmented, and the segmentation result is used as the first feature of the multiple frames.

3. The method according to claim 1, wherein: The obtaining, according to the filled first feature of each frame, the second feature corresponding to the first feature of each frame comprises: Obtaining a decoder feature of the first feature of each frame after the padding; The decoder features corresponding to the filling features in the decoder features are removed to obtain the second features corresponding to the first features of each frame.

4. The method according to claim 1, wherein: The step of obtaining audio data corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame comprises: Determine adjacent feature frames of each frame of the second feature from the multiple frames of the second feature; The filling feature is intercepted from the adjacent feature frame of the second feature of each frame, and the filling feature is Fill to the second feature of each frame; According to the second feature of each frame after filling, the audio data corresponding to the second feature of each frame is obtained.

5. The method according to claim 4, wherein: The step of obtaining the audio data corresponding to the second feature of each frame according to the filled second feature of each frame includes: Acquire the audio data of the second feature of each frame after the padding; The audio data corresponding to the filling feature in the audio data is removed to obtain the audio data corresponding to the second feature of each frame.

6. The method according to claim 1, wherein: The obtaining of the first features of multiple frames according to the phoneme sequence comprises: Inputting the phoneme sequence into a speech synthesis model; The encoder in the speech synthesis model processes the phoneme sequence to obtain the multi-frame first features output by the encoder.

7. The method according to claim 1, wherein: The obtaining, according to the first feature of each frame and the adjacent feature frames of the first feature of each frame, the second feature corresponding to the first feature of each frame comprises: Inputting the first feature of each frame and adjacent feature frames of the first feature of each frame into a speech synthesis model; The decoder in the speech synthesis model processes the first feature of each frame and the adjacent feature frames of the first feature of each frame to obtain the second feature corresponding to the first feature of each frame output by the decoder.

8. The method according to claim 1, wherein: The step of obtaining audio data corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame comprises: Inputting the second feature of each frame and the adjacent feature frames of the second feature of each frame into a speech synthesis model; The vocoder in the speech synthesis model processes the second feature of each frame and the feature frames adjacent to the second feature of each frame to obtain the audio data corresponding to the second feature of each frame output by the vocoder.

9. A method for training a speech synthesis model, comprising: Acquire a data set, wherein the data set includes a plurality of training texts and training audio data of the plurality of training texts; Constructing a neural network model including an encoder, a decoder and a vocoder, wherein the encoder is used to obtain a multi-frame predicted first feature according to a phoneme sequence of a training text, the decoder is used to obtain a predicted second feature corresponding to the predicted first feature of each frame according to the predicted first feature of each frame and the adjacent feature frames of the predicted first feature of each frame, and the vocoder is used to obtain predicted audio data corresponding to the predicted second feature of each frame according to the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame, wherein the adjacent feature frames of the predicted second feature of each frame are two feature frames adjacent to and before the predicted second feature of each frame; Training the neural network model based on the multiple training texts and the training audio data of the multiple training texts until the neural network model converges to obtain the speech synthesis model; Wherein, when the decoder obtains the predicted second feature corresponding to the predicted first feature of each frame according to the predicted first feature of each frame and the adjacent feature frames of the predicted first feature of each frame, it includes: Determine, from the plurality of frames predicting the first feature, adjacent feature frames for predicting the first feature of each frame; Extracting filling features from adjacent feature frames of the first feature predicted in each frame, and filling the filling features into the first feature predicted in each frame; According to the predicted first feature of each frame after filling, the predicted second feature corresponding to the predicted first feature of each frame is obtained.

10. The method according to claim 9, wherein: The decoder predicts the first feature of each frame after padding to obtain the predicted second feature corresponding to the predicted first feature of each frame, including: Obtaining decoder features of the first feature predicted for each frame after the padding; The decoder features corresponding to the padding features are removed from the decoder features to obtain the predicted second features corresponding to the predicted first features for each frame.

11. The method according to any one of claims 9 to 10, wherein: The vocoder obtains predicted audio data corresponding to the predicted second feature of each frame according to the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame, including: Determine, from the plurality of frames predicting the second feature, adjacent feature frames for predicting the second feature of each frame; Extracting filling features from adjacent feature frames of the second feature predicted in each frame, and filling the filling features into the second feature predicted in each frame; According to the second feature predicted for each frame after filling, the predicted audio data corresponding to the second feature predicted for each frame is obtained.

12. The method according to claim 11, wherein: The vocoder predicts the second feature according to each frame after padding, and obtains the predicted audio data corresponding to the predicted second feature of each frame, including: Obtaining predicted audio data of the second feature for each frame after the padding; The predicted audio data corresponding to the filling feature in the predicted audio data is removed to obtain the predicted audio data corresponding to the second feature predicted for each frame.

13. A speech synthesis device, comprising: A first acquisition unit is used to acquire a text to be processed and obtain a phoneme sequence of the text to be processed; A first processing unit, configured to obtain a plurality of first features of the frames according to the phoneme sequence; A second processing unit is used to obtain a second feature corresponding to the first feature of each frame according to the first feature of each frame and the adjacent feature frames of the first feature of each frame; A synthesis unit, used for obtaining audio data corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame, wherein the adjacent feature frames of the second feature of each frame are two feature frames adjacent to the second feature of each frame; Wherein, when the second processing unit obtains the second feature corresponding to the first feature of each frame according to the first feature of each frame and the adjacent feature frames of the first feature of each frame, the second processing unit specifically performs: Determine adjacent feature frames of the first feature of each frame from the multiple frames; Intercepting a filling feature from a feature frame adjacent to the first feature of each frame, and filling the filling feature into the first feature of each frame; According to the first feature of each frame after filling, the second feature corresponding to the first feature of each frame is obtained.

14. The device according to claim 13, wherein: When the first processing unit obtains the first feature of multiple frames according to the phoneme sequence, the first processing unit specifically performs: Obtaining encoder features corresponding to the phoneme sequence; The encoder feature is segmented, and the segmentation result is used as the first feature of the multiple frames.

15. The device according to claim 13, wherein: When the second processing unit obtains the second feature corresponding to the first feature of each frame according to the first feature of each frame after filling, the second processing unit specifically performs: Obtaining a decoder feature of the first feature of each frame after the padding; The decoder features corresponding to the filling features in the decoder features are removed to obtain the second features corresponding to the first features of each frame.

16. The device according to claim 13, wherein: When the synthesis unit obtains the audio data corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame, the synthesis unit specifically performs: Determine adjacent feature frames of each frame of the second feature from the multiple frames of the second feature; The filling feature is intercepted from the adjacent feature frame of the second feature of each frame, and the filling feature is Fill to the second feature of each frame; According to the second feature of each frame after filling, the audio data corresponding to the second feature of each frame is obtained.

17. The device according to claim 16, wherein: When the synthesis unit obtains the audio data corresponding to the second feature of each frame according to the second feature of each frame after filling, the synthesis unit specifically performs: Acquire the audio data of the second feature of each frame after the padding; The audio data corresponding to the filling feature in the audio data is removed to obtain the audio data corresponding to the second feature of each frame.

18. The device according to claim 13, wherein: When the first processing unit obtains the first feature of multiple frames according to the phoneme sequence, the first processing unit specifically performs: Inputting the phoneme sequence into a speech synthesis model; The encoder in the speech synthesis model processes the phoneme sequence to obtain the multi-frame first features output by the encoder.

19. The device according to claim 13, wherein: When the second processing unit obtains the second feature corresponding to the first feature of each frame according to the first feature of each frame and the adjacent feature frames of the first feature of each frame, the second processing unit specifically performs: Inputting the first feature of each frame and adjacent feature frames of the first feature of each frame into a speech synthesis model; The decoder in the speech synthesis model processes the first feature of each frame and the adjacent feature frames of the first feature of each frame to obtain the second feature corresponding to the first feature of each frame output by the decoder.

20. The device according to claim 13, wherein: When the synthesis unit obtains the audio data corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame, the synthesis unit specifically performs: Inputting the second feature of each frame and the adjacent feature frames of the second feature of each frame into a speech synthesis model; The vocoder in the speech synthesis model processes the second feature of each frame and the feature frames adjacent to the second feature of each frame to obtain the audio data corresponding to the second feature of each frame output by the vocoder.

21. A training device for a speech synthesis model, comprising: A second acquisition unit, configured to acquire a data set, wherein the data set includes a plurality of training texts and training audio data of the plurality of training texts; A construction unit is used to construct a neural network model including an encoder, a decoder and a vocoder, wherein the encoder is used to obtain a multi-frame predicted first feature according to a phoneme sequence of a training text, the decoder is used to obtain a predicted second feature corresponding to the predicted first feature of each frame according to the predicted first feature of each frame and the adjacent feature frames of the predicted first feature of each frame, and the vocoder is used to obtain predicted audio data corresponding to the predicted second feature of each frame according to the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame, wherein the adjacent feature frames of the predicted second feature of each frame are two feature frames adjacent to and before the predicted second feature of each frame; A training unit, configured to train the neural network model based on the multiple training texts and the training audio data of the multiple training texts until the neural network model converges to obtain the speech synthesis model; Wherein, when the decoder obtains the predicted second feature corresponding to the predicted first feature of each frame according to the predicted first feature of each frame and the adjacent feature frames of the predicted first feature of each frame, it includes: Determine, from the plurality of frames predicting the first feature, adjacent feature frames for predicting the first feature of each frame; Extracting filling features from adjacent feature frames of the first feature predicted in each frame, and filling the filling features into the first feature predicted in each frame; According to the predicted first feature of each frame after filling, the predicted second feature corresponding to the predicted first feature of each frame is obtained.

22. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

23. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 12.

24. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images