Speech synthesis and training method and device of speech synthesis model

Abstract

The present disclosure provides a method, device, electronic device and readable storage medium for speech synthesis and training of speech synthesis models, and relates to the fields of artificial intelligence technology such as deep learning and natural language processing. The speech synthesis method includes: obtaining a text to be processed, obtaining a phoneme sequence of the text to be processed; encoding the phoneme sequence to obtain a first feature; decoding the first feature to obtain multiple frames of second features; obtaining a third feature corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame; obtaining audio data corresponding to the third feature of each frame according to the third feature of each frame and the adjacent feature frames of the third feature of each frame. The training method of the speech synthesis model includes: obtaining a data set; constructing a neural network model including an encoder, a decoder, a posterior network layer and a vocoder; training the neural network model based on multiple training texts and training audio data of multiple training texts to obtain a speech synthesis model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology such as deep learning and natural language processing. Provided are a method, device, electronic device and readable storage medium for training speech synthesis and speech synthesis models. Background Art

[0002] In speech synthesis tasks, usually after inputting the text to be processed, the audio data corresponding to the text to be processed is output. However, in the existing speech synthesis method, the return of the first audio data needs to wait until the entire calculation is completed before it can start, which cannot meet the low-latency application scenarios and the response time of speech synthesis is long. Summary of the invention

[0003] According to a first aspect of the present disclosure, a speech synthesis method is provided, comprising: acquiring a text to be processed and obtaining a phoneme sequence of the text to be processed; encoding the phoneme sequence and obtaining a first feature; decoding the first feature and obtaining multiple frames of second features; obtaining a third feature corresponding to the second feature of each frame based on the second feature of each frame and the adjacent feature frames of the second feature of each frame; obtaining audio data corresponding to the third feature of each frame based on the third feature of each frame and the adjacent feature frames of the third feature of each frame.

[0004] According to a second aspect of the present disclosure, a method for training a speech synthesis model is provided, comprising: obtaining a data set, wherein the data set comprises a plurality of training texts and training audio data of the plurality of training texts; constructing a neural network model comprising an encoder, a decoder, a posterior network layer and a vocoder, wherein the encoder is used to obtain a predicted first feature according to a phoneme sequence of the training text, the decoder is used to obtain a multi-frame predicted second feature according to the predicted first feature, the posterior network layer is used to obtain a predicted third feature corresponding to the predicted second feature of each frame according to the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame, the vocoder is used to obtain a predicted audio data corresponding to the predicted third feature of each frame according to the predicted third feature of each frame and the adjacent feature frames of the predicted third feature of each frame; and training the neural network model based on the plurality of training texts and the training audio data of the plurality of training texts until the neural network model converges to obtain the speech synthesis model.

[0005] According to a third aspect of the present disclosure, a speech synthesis device is provided, comprising: a first acquisition unit, used to acquire a text to be processed and obtain a phoneme sequence of the text to be processed; a first processing unit, used to encode the phoneme sequence and obtain a first feature; a second processing unit, used to decode the first feature and obtain multiple frames of second features; a third processing unit, used to obtain a third feature corresponding to the second feature of each frame based on the second feature of each frame and the adjacent feature frames of the second feature of each frame; and a synthesis unit, used to obtain audio data corresponding to the third feature of each frame based on the third feature of each frame and the adjacent feature frames of the third feature of each frame.

[0006] According to a fourth aspect of the present disclosure, a training device for a speech synthesis model is provided, comprising: a second acquisition unit, used to acquire a data set, wherein the data set includes multiple training texts and training audio data of the multiple training texts; a construction unit, used to construct a neural network model including an encoder, a decoder, a posterior network layer and a vocoder, wherein the encoder is used to obtain a predicted first feature according to a phoneme sequence of the training text, the decoder is used to obtain a multi-frame predicted second feature according to the predicted first feature, the posterior network layer is used to obtain a predicted third feature corresponding to the predicted second feature of each frame according to the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame, and the vocoder is used to obtain the predicted audio data corresponding to the predicted third feature of each frame according to the predicted third feature of each frame and the adjacent feature frames of the predicted third feature of each frame; and a training unit, used to train the neural network model based on the multiple training texts and the training audio data of the multiple training texts until the neural network model converges to obtain the speech synthesis model.

[0007] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0008] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method as described above.

[0009] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method as described above when executed by a processor.

[0010] It can be seen from the above technical solutions that the present invention obtains the third feature and audio data respectively by combining adjacent feature frames, and can fully consider the boundary of each feature, thereby improving the continuity between the audio data corresponding to different third features. In addition, this embodiment can achieve the purpose of streaming synthesized speech by performing speech synthesis on each third feature, thereby reducing the response time of speech synthesis and improving the speed of speech synthesis.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0014] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0016] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0018] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0019] Figure 7 It is a block diagram of an electronic device used to implement the speech synthesis or speech synthesis model training method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and mechanisms is omitted in the following description.

[0021] Figure 1 Schematic diagram of the first embodiment of the present disclosure. Figure 1As shown, the speech synthesis method of this embodiment specifically includes the following steps:

[0022] S101, obtaining a text to be processed and obtaining a phoneme sequence of the text to be processed;

[0023] S102, encoding the phoneme sequence to obtain a first feature;

[0024] S103, decoding the first feature to obtain multiple frames of second features;

[0025] S104, obtaining a third feature corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame;

[0026] S105. Obtain audio data corresponding to the third feature of each frame according to the third feature of each frame and the adjacent feature frames of the third feature of each frame.

[0027] The speech synthesis method of this embodiment obtains the third feature and audio data respectively by combining adjacent feature frames, and can fully consider the boundary of each feature, so as to improve the coherence between the obtained audio data corresponding to different third features. In addition, this embodiment can achieve the purpose of streaming synthesized speech by performing speech synthesis on each third feature, thereby reducing the response time of speech synthesis and improving the speed of speech synthesis.

[0028] In this embodiment, when executing S101 to obtain the text to be processed, the text input by the input end or the text selected by the input end may be used as the text to be processed.

[0029] When executing S101 to obtain the phoneme sequence of the text to be processed, this embodiment can first obtain the phonemes corresponding to each character in the text to be processed, and then construct the phoneme sequence of the text to be processed based on the phonemes corresponding to all the characters; wherein, before executing S101 to obtain the phoneme sequence of the text to be processed, this embodiment can also remove punctuation marks in the text to be processed.

[0030] In this embodiment, after executing S101 to obtain the phoneme sequence of the text to be processed, S102 is executed to encode the phoneme sequence to obtain the first feature.

[0031] In this embodiment, when executing S102 to encode the phoneme sequence to obtain the first feature, an optional implementation method that can be adopted is: obtaining an encoder feature of the phoneme sequence, and using the obtained encoder feature as the first feature.

[0032] In addition, when executing S102 to encode the phoneme sequence to obtain the first feature, the present embodiment can also input the phoneme sequence into a pre-trained speech synthesis model, and the encoder in the speech synthesis model encodes the phoneme sequence, thereby using the output encoder feature as the first feature.

[0033] After executing S102 to obtain the first feature, this embodiment executes S103 to decode the first feature to obtain multiple frames of second features; wherein the second feature obtained in this embodiment is an acoustic feature, such as a Melspectrogram.

[0034] When executing S103 in this embodiment to obtain multiple-frame second features according to the first feature, an optional implementation method that can be adopted is: obtaining a decoder feature of the first feature, and obtaining multiple-frame second features according to the obtained decoder feature; it can be understood that the decoder feature obtained by executing S103 in this embodiment can be directly the multiple-frame second feature, or the obtained decoder feature can be segmented and the segmentation result can be used as the multiple-frame second feature.

[0035] In this embodiment, when executing S103 to decode the first feature and obtain multiple frames of second features, the first feature can be input into a pre-trained speech synthesis model, and the decoder in the speech synthesis model decodes the first feature, thereby using the output multi-frame decoder feature as the multi-frame second feature.

[0036] After executing S103 to obtain the second features of multiple frames, this embodiment executes S104 to obtain a third feature corresponding to the second feature of each frame based on the second feature of each frame and the feature frames adjacent to the second feature of each frame.

[0037] Specifically, when executing S104 to obtain the third feature corresponding to the second feature of each frame based on the second feature of each frame and the adjacent feature frames of the second feature of each frame, the present embodiment may adopt an optional implementation method: determining the adjacent feature frames of the second feature of each frame from multiple frames of second features; intercepting the filling feature from the adjacent feature frames of the second feature of each frame, and filling the intercepted filling feature into the second feature of each frame; and obtaining the third feature corresponding to the second feature of each frame based on the second feature of each frame after filling.

[0038] That is to say, this embodiment uses the filling features intercepted from the adjacent feature frames of the second feature of each frame to fill the second feature of each frame, so as to obtain the third feature corresponding to the second feature of each frame. Since the second feature of each frame after the filling process contains some features in the adjacent feature frames, the boundary of the second feature of each frame can be accurately calculated, thereby improving the accuracy of the third feature corresponding to the second feature of each frame.

[0039] In this embodiment, when executing S104 to intercept filling features from the adjacent feature frames of the second feature of each frame and fill the intercepted filling features into the second feature of each frame, the feature located at the first preset position of the adjacent feature frame can be intercepted as the filling feature based on the positional relationship between the second feature of each frame and the adjacent feature frame, and the intercepted filling features can be filled into the second preset position of the second feature of each frame.

[0040] In this embodiment, there is a corresponding relationship between the first preset position and the second preset position. If the first preset position is the beginning, the second preset position is the end; if the first preset position is the end, the second preset position is the beginning.

[0041] For example, if the multi-frame second features obtained by executing S103 in this embodiment include M1, M2 and M3, this embodiment executes S104 to determine that the adjacent feature frame of M1 is M2, determines that the adjacent feature frames of M2 are M1 and M3, and determines that the adjacent feature frame of M3 is M2; for M1, M2 is located after M1, then the features at the beginning of M2 are intercepted as filling features and filled to the end of M1; for M2, M3 is located after M2, then the features at the beginning of M3 are intercepted as filling features and filled to the end of M2; M1 is located before M2, then the features at the end of M1 are intercepted as filling features and filled to the beginning of M2; for M3, M2 is located before M3, then the features at the end of M2 are intercepted as filling features and filled to the beginning of M3.

[0042] When executing S104 in this embodiment to obtain the third feature corresponding to the second feature of each frame according to the second feature of each frame after padding, an optional implementation method that can be adopted is: obtaining the posterior feature of the second feature of each frame after padding, the posterior feature is a convolution feature, for example, using a convolutional network including 5 layers of convolution (such as a posterior network Post Net) to process the second feature after padding, and using the processing result as the posterior feature of the second feature of each frame after padding; removing the posterior feature corresponding to the padding feature in the posterior feature, to obtain the third feature corresponding to the second feature of each frame.

[0043] That is to say, when obtaining the third feature corresponding to the second feature of each frame, this embodiment can also remove the posterior features corresponding to the filling features in the posterior features of the second feature, so as to further improve the accuracy of the obtained third feature and avoid duplication in the different audio data finally generated.

[0044] When executing S104, the present embodiment obtains the third feature corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame. The second feature of each frame and the adjacent feature frames of the second feature of each frame can also be input into a pre-trained speech synthesis model, and the posterior network layer in the speech synthesis model processes the second feature of each frame and the adjacent feature frames of the second feature of each frame to output the third feature corresponding to the second feature of each frame.

[0045] In addition, when executing S104 to obtain the third feature corresponding to the second feature of each frame based on the second feature of each frame and the adjacent feature frames of the second feature of each frame, this embodiment may also include the following contents: obtaining the number of frames of the obtained second feature. This embodiment may put the obtained second feature into the first buffer, thereby obtaining the number of frames of the second feature stored in the first buffer; when it is determined that the obtained number of frames meets the first preset requirement, obtaining the third feature corresponding to the second feature of each frame based on the second feature of each frame and the adjacent feature frames of the second feature of each frame.

[0046] That is to say, in this embodiment, the step of obtaining the third feature based on the second feature is performed only when the number of frames of the acquired second feature meets the first preset requirement. Since the number of second features is sufficient, the accuracy of the obtained third feature can be improved.

[0047] In this embodiment, when executing S104, when it is determined that the number of frames of the obtained second feature exceeds the first frame number threshold, it can be determined that the number of frames of the obtained second feature meets the first preset requirement.

[0048] After executing S104 to obtain the third feature corresponding to the second feature of each frame, this embodiment executes S105 to obtain audio data corresponding to the third feature of each frame based on the third feature of each frame and the adjacent feature frames of the third feature of each frame.

[0049] Specifically, when this embodiment executes S105 to obtain audio data corresponding to the third feature of each frame based on the third feature of each frame and the adjacent feature frames of the third feature of each frame, an optional implementation method that can be adopted is: determine the adjacent feature frames of the third feature of each frame from multiple frames of third features; intercept fill features from the adjacent feature frames of the third feature of each frame, and fill the intercepted fill features into the third feature of each frame; obtain the audio data corresponding to the third feature of each frame based on the third feature of each frame after filling.

[0050] Similarly, when executing S105, this embodiment intercepts filling features from the adjacent feature frames of the third feature of each frame and fills the intercepted filling features into the third feature of each frame. According to the positional relationship between the third feature of each frame and the adjacent feature frame, the feature located at the first preset position of the adjacent feature frame can be intercepted as the filling feature, and the intercepted filling feature can be filled into the second preset position of the third feature of each frame; there is a corresponding relationship between the first preset position and the second preset position.

[0051] That is to say, this embodiment uses the filling features intercepted from the adjacent feature frames of the third feature of each frame to fill the third feature of each frame, so as to obtain audio data corresponding to the third feature of each frame. Since the third feature of each frame after the filling process contains some features in the adjacent feature frames, the boundary of the third feature of each frame can be accurately calculated, thereby improving the coherence between the audio data corresponding to different third features.

[0052] When executing S105 in this embodiment to obtain audio data corresponding to the third feature of each frame according to the third feature of each frame after filling, an optional implementation method that can be adopted is: obtaining audio data of the second feature of each frame after filling; removing the audio data corresponding to the filling feature in the audio data, and obtaining audio data corresponding to the third feature of each frame.

[0053] That is to say, when obtaining the audio data corresponding to the third feature of each frame, this embodiment can also remove the audio data corresponding to the filling feature in the audio data to avoid duplication between the generated audio data corresponding to different third features.

[0054] When executing S105, the present embodiment obtains audio data corresponding to the third feature of each frame based on the third feature of each frame and the adjacent feature frames of the third feature of each frame. The third feature of each frame and the adjacent feature frames of the third feature of each frame can also be input into a pre-trained speech synthesis model, and the voice coder in the speech synthesis model processes the third feature of each frame and the adjacent feature frames of the third feature of each frame to obtain audio data corresponding to the third feature of each frame.

[0055] In addition, when executing S105 to obtain audio data corresponding to the third feature of each frame based on the third feature of each frame and the adjacent feature frames of the third feature of each frame, this embodiment may also include the following contents: obtaining the frame number of the obtained third feature. This embodiment may put the obtained third feature into the second buffer, thereby obtaining the frame number of the third feature stored in the second buffer; when it is determined that the obtained number of frames meets the second preset requirement, obtaining audio data corresponding to the third feature of each frame based on the third feature of each frame and the adjacent feature frames of the third feature of each frame.

[0056] That is to say, in this embodiment, the step of obtaining audio data according to the third feature is performed only when the number of frames of the obtained third feature meets the second preset requirement. Since the number of third features is relatively sufficient, the accuracy of the obtained audio data can be improved.

[0057] In this embodiment, when executing S105, when it is determined that the number of frames of the obtained third feature exceeds the second frame number threshold, it can be determined that the number of frames of the obtained third feature meets the second preset requirement.

[0058] After executing S105 to obtain the audio data corresponding to the third feature of each frame, this embodiment can return the obtained audio data to the input end in sequence. Since the purpose of streaming speech synthesis is achieved, this embodiment can reduce the response time of speech synthesis and improve the speed of speech synthesis.

[0059] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure. Figure 2 As shown, the training method of the speech synthesis model of this embodiment specifically includes the following steps:

[0060] S201, obtaining a data set, wherein the data set includes a plurality of training texts and training audio data of the plurality of training texts;

[0061] S202, constructing a neural network model including an encoder, a decoder, a posterior network layer and a vocoder, wherein the encoder is used to obtain a predicted first feature according to a phoneme sequence of a training text, the decoder is used to obtain a multi-frame predicted second feature according to the predicted first feature, the posterior network layer is used to obtain a predicted third feature corresponding to the predicted second feature of each frame according to the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame, and the vocoder is used to obtain predicted audio data corresponding to the predicted third feature of each frame according to the predicted third feature of each frame and the adjacent feature frames of the predicted third feature of each frame;

[0062] S203. Train the neural network model based on the multiple training texts and the training audio data of the multiple training texts until the neural network model converges to obtain the speech synthesis model.

[0063] The training method of the speech synthesis model of this embodiment, the posterior network layer and the vocoder in the constructed neural network model, obtain the predicted third feature and the predicted audio data by combining adjacent feature frames, which can improve the accuracy of the obtained predicted third feature and the coherence between different predicted audio data, and generate corresponding audio data according to different feature frames, so that the speech synthesis model can achieve the purpose of streaming speech synthesis, thereby enhancing the synthesis effect of the speech synthesis model.

[0064] After executing S201 to obtain training audio data including multiple training texts and the multiple training texts, this embodiment can also divide the data set into a training set, an evaluation set and a test set; wherein the training set is used to train the neural network model, the evaluation set is used to evaluate the neural network model during the training process, and the test set is used to test the effect of the trained neural network model.

[0065] After executing S201 to obtain the data set, this embodiment can also extract training acoustic features from the training audio data, and then align the phoneme sequence of the training text with the extracted training acoustic features; the purpose of extracting training acoustic features from the training audio data is to calculate the loss function of the encoder and decoder in the neural network model, and then update the parameters in the encoder and decoder according to the loss function.

[0066] After executing S201 to obtain a data set, this embodiment executes S202 to construct a neural network model including an encoder, a decoder, a posterior network layer and a vocoder.

[0067] In the neural network model constructed by executing S202 in this embodiment, the encoder, decoder and posterior network layer constitute an acoustic model, which is used to convert the input text into acoustic features, such as Mel spectrum features; the structure of the acoustic model in this embodiment is based on the Tacotron2 structure.

[0068] In the neural network model constructed by executing S202 in this embodiment, the encoder is used to obtain a predicted first feature based on the phoneme sequence of the training text, and the predicted first feature is the encoder feature.

[0069] In the neural network model constructed by executing S202 in this embodiment, the decoder is used to obtain a multi-frame predicted second feature based on the predicted first feature, and the predicted second feature is, for example, an acoustic feature of a Mel spectrum.

[0070] In the neural network model constructed by executing S202 in this embodiment, the decoder outputs the predicted second feature of the current frame and the decoder state of the current frame based on the predicted first feature, the predicted second feature of the previous frame and the decoder state of the previous frame, that is, the decoder in this embodiment outputs the predicted second feature frame by frame.

[0071] In the neural network model constructed by executing S202 in this embodiment, the posterior network layer is used to obtain a predicted third feature corresponding to the predicted second feature of each frame based on the second feature predicted for each frame and the adjacent feature frames of the second feature predicted for each frame; the posterior network layer in this embodiment can be a Post Net (posterior network) composed of a 5-layer convolutional network.

[0072] Specifically, in the neural network model constructed by executing S202 in this embodiment, when the posterior network layer obtains the predicted third feature corresponding to the predicted second feature of each frame based on the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame, an optional implementation method that can be adopted is: determining the adjacent feature frames of the predicted second feature of each frame from the predicted second features of multiple frames; intercepting the filling feature from the adjacent feature frames of the predicted second feature of each frame, and filling the intercepted filling feature into the predicted second feature of each frame; obtaining the predicted third feature corresponding to the predicted second feature of each frame based on the predicted second feature of each frame after filling.

[0073] In the neural network model constructed by executing S202 in this embodiment, when the posterior network layer predicts the second feature of each frame after padding and obtains the predicted third feature corresponding to the predicted second feature of each frame, an optional implementation method that can be adopted is: obtaining the posterior features of the predicted second feature of each frame after padding, the posterior features in this embodiment are the features obtained by the posterior network layer processing the input predicted second features after padding; removing the posterior features corresponding to the padding features in the posterior features, and obtaining the predicted third features corresponding to the predicted second features of each frame.

[0074] In addition, in the neural network model constructed by executing S202 in this embodiment, when the posterior network layer obtains the predicted third feature corresponding to the predicted second feature of each frame based on the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame, it can also include the following contents: obtaining the number of frames of the obtained predicted second feature; when it is determined that the obtained number of frames meets the first preset requirement, obtaining the predicted third feature corresponding to the predicted second feature of each frame based on the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame.

[0075] The posterior network layer of this embodiment can determine that the obtained number of frames of the predicted second feature meets the first preset requirement when it is determined that the obtained number of frames of the predicted second feature exceeds the first frame number threshold.

[0076] In the neural network model constructed by executing S202 in this embodiment, the vocoder is used to obtain predicted audio data corresponding to the predicted third feature of each frame based on the predicted third feature of each frame and the adjacent feature frames of the predicted third feature of each frame.

[0077] Specifically, in the neural network model constructed by executing S202 of this embodiment, when the voice coder obtains the predicted audio data corresponding to the predicted third feature of each frame based on the predicted third feature of each frame and the adjacent feature frames of the predicted third feature of each frame, an optional implementation method that can be adopted is: determining the adjacent feature frames of the predicted third feature of each frame from multiple frames of predicted third features; intercepting filling features from the adjacent feature frames of the predicted third feature of each frame, and filling the intercepted filling features into the predicted third feature of each frame; and obtaining the predicted audio data corresponding to the predicted third feature of each frame based on the predicted third feature of each frame after filling.

[0078] In the neural network model constructed by executing S202 in this embodiment, when the voice coder predicts the third feature based on each frame after filling and obtains the predicted audio data corresponding to the predicted third feature of each frame, an optional implementation method that can be adopted is: obtaining the predicted audio data of the third feature predicted for each frame after filling; removing the predicted audio data corresponding to the filled feature in the predicted audio data, and obtaining the predicted audio data corresponding to the predicted third feature of each frame.

[0079] In addition, in the neural network model constructed by executing S202 in this embodiment, when the voice coder obtains the audio data corresponding to the predicted third feature of each frame based on the predicted third feature of each frame and the adjacent feature frames of the predicted third feature of each frame, it can also include the following content: obtaining the number of frames of the obtained predicted third feature; when it is determined that the obtained number of frames meets the second preset requirement, obtaining the audio data corresponding to the predicted third feature of each frame based on the predicted third feature of each frame and the adjacent feature frames of the predicted third feature of each frame.

[0080] The vocoder may determine that the obtained number of frames of the predicted third feature meets the second preset requirement when determining that the obtained number of frames of the predicted third feature exceeds the second frame number threshold.

[0081] After executing S202 to construct a neural network model including an encoder, a decoder, a posterior network layer and a vocoder, this embodiment executes S203 to train the neural network model based on multiple training texts and training audio data of the multiple training texts until the neural network model converges to obtain a speech synthesis model.

[0082] Specifically, when executing S203 to train a neural network model based on multiple training texts and training audio data of multiple training texts, the present embodiment may adopt an optional implementation method as follows: respectively inputting the phoneme sequences of multiple training texts into the neural network model, and obtaining predicted audio data output by the neural network model for each training text; calculating a first loss function value based on the training audio data of multiple training texts and the predicted audio data; and adjusting the parameters of the encoder, decoder, posterior network layer and vocoder in the neural network model based on the calculated first loss function value until the first loss function value converges to obtain a speech synthesis model.

[0083] In addition, in the data set obtained by executing S201 in the present embodiment, if the training acoustic features corresponding to the phoneme sequences of the training texts are also obtained, then when the present embodiment executes S203 to train the neural network model based on multiple training texts and the training audio data of multiple training texts, an optional implementation method that can be adopted is: inputting the phoneme sequences of the multiple training texts into the neural network model respectively, and obtaining the predicted second features and predicted audio data output by the neural network model for each training text; calculating the first loss function value according to the training audio data and the predicted audio data of the multiple training texts, and calculating the second loss function value according to the predicted second features and the training acoustic features of the multiple training texts; adjusting the parameters of the vocoder in the neural network model according to the calculated first loss function value, and adjusting the parameters of the encoder, decoder and posterior network layer in the neural network model according to the calculated second loss function value, until the first loss function value and the second loss function value converge to obtain a speech synthesis model.

[0084] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure. Figure 3 FIG. 4 shows a flow chart of the present embodiment when obtaining the second feature of multiple frames and obtaining the third feature of multiple frames. Figure 3 The M in it represents the second feature output by the decoder, and M' represents the third feature output by the posterior network layer; Figure 3 The second feature output by the decoder is also placed in the first buffer, and the third feature output by the posterior network layer is placed in the second buffer.

[0085] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure. Figure 4 FIG. 4 shows a flowchart of the present embodiment when obtaining audio data corresponding to multiple frames of the second feature. Figure 4 The M in represents the third feature, and W represents audio data.

[0086] Figure 5 is a schematic diagram according to the fifth embodiment of the present disclosure. Figure 5 As shown, the speech synthesis device 500 of this embodiment includes:

[0087] A first acquisition unit 501 is used to acquire a text to be processed and obtain a phoneme sequence of the text to be processed;

[0088] A first processing unit 502 is used to encode the phoneme sequence to obtain a first feature;

[0089] The second processing unit 503 is used to decode the first feature to obtain multiple frames of second features;

[0090] The third processing unit 504 is used to obtain a third feature corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame;

[0091] The synthesis unit 505 is used to obtain audio data corresponding to the third feature of each frame according to the third feature of each frame and the adjacent feature frames of the third feature of each frame.

[0092] When the first acquiring unit 501 acquires the text to be processed, the text input by the input end or the text selected by the input end may be used as the text to be processed.

[0093] When the first acquisition unit 501 obtains the phoneme sequence of the text to be processed, it can first obtain the phonemes corresponding to each character in the text to be processed, and then construct the phoneme sequence of the text to be processed based on the phonemes corresponding to all the characters; wherein, before the first acquisition unit 501 obtains the phoneme sequence of the text to be processed, the punctuation marks in the text to be processed can also be removed.

[0094] In this embodiment, after the first acquisition unit 501 obtains the phoneme sequence of the text to be processed, the first processing unit 502 encodes the phoneme sequence to obtain the first feature.

[0095] When the first processing unit 502 encodes the phoneme sequence to obtain the first feature, an optional implementation method that can be adopted is: obtaining an encoder feature of the phoneme sequence, and using the obtained encoder feature as the first feature.

[0096] In addition, when the first processing unit 502 encodes the phoneme sequence to obtain the first feature, it can also input the phoneme sequence into a pre-trained speech synthesis model, and the encoder in the speech synthesis model encodes the phoneme sequence, thereby using the output encoder feature as the first feature.

[0097] In this embodiment, after the first processing unit 502 obtains the first feature, the second processing unit 503 decodes the first feature to obtain multiple frames of second features; wherein the second feature obtained by the second processing unit 503 is an acoustic feature, such as a Mel spectrogram.

[0098] When the second processing unit 503 decodes the first feature to obtain multiple frames of second features, an optional implementation method that can be adopted is: obtaining a decoder feature of the first feature, and obtaining multiple frames of second features based on the obtained decoder feature; it can be understood that the decoder feature obtained by the second processing unit 503 can be directly a multiple-frame second feature, or the obtained decoder feature can be segmented and the segmentation result can be used as the multiple-frame second feature.

[0099] When the second processing unit 503 decodes the first feature to obtain a multi-frame second feature, the first feature can be input into a pre-trained speech synthesis model, and the decoder in the speech synthesis model decodes the first feature, thereby using the output multi-frame decoder feature as the multi-frame second feature.

[0100] In this embodiment, after the second processing unit 503 obtains the second features of multiple frames, the third processing unit 504 obtains the third feature corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame.

[0101] Specifically, when the third processing unit 504 obtains the third feature corresponding to the second feature of each frame based on the second feature of each frame and the adjacent feature frames of the second feature of each frame, an optional implementation method may be adopted: determining the adjacent feature frames of the second feature of each frame from multiple frames of second features; intercepting filling features from the adjacent feature frames of the second feature of each frame, and filling the intercepted filling features into the second feature of each frame; and obtaining the third feature corresponding to the second feature of each frame based on the second feature of each frame after filling.

[0102] That is to say, the third processing unit 504 fills the second feature of each frame with the filling feature intercepted from the adjacent feature frame of the second feature of each frame to obtain the third feature corresponding to the second feature of each frame. Since the second feature of each frame after the filling process includes some features in the adjacent feature frame, the boundary of the second feature of each frame can be accurately calculated, thereby improving the accuracy of the third feature corresponding to the second feature of each frame.

[0103] When the third processing unit 504 intercepts filling features from the adjacent feature frames of the second feature of each frame and fills the intercepted filling features into the second feature of each frame, it can intercept the features located at the first preset position of the adjacent feature frame as the filling features based on the positional relationship between the second feature of each frame and the adjacent feature frame, and fill the intercepted filling features into the second preset position of the second feature of each frame.

[0104] In this embodiment, there is a corresponding relationship between the first preset position and the second preset position. If the first preset position is the beginning, the second preset position is the end; if the first preset position is the end, the second preset position is the beginning.

[0105] When the third processing unit 504 obtains the third feature corresponding to the second feature of each frame based on the second feature of each frame after filling, an optional implementation method that can be adopted is: obtaining the posterior features of the second features of each frame after filling; removing the posterior features corresponding to the filling features in the posterior features, and obtaining the third features corresponding to the second features of each frame.

[0106] That is to say, when the third processing unit 504 obtains the third feature corresponding to the second feature of each frame, it can also remove the posterior features corresponding to the filling features in the posterior features of the second feature, so as to improve the accuracy of the obtained third feature and avoid duplication in the different audio data finally generated.

[0107] When the third processing unit 504 obtains the third feature corresponding to the second feature of each frame based on the second feature of each frame and the adjacent feature frames of the second feature of each frame, it can also input the second feature of each frame and the adjacent feature frames of the second feature of each frame into a pre-trained speech synthesis model, and the posterior network layer in the speech synthesis model processes the second feature of each frame and the adjacent feature frames of the second feature of each frame to output the third feature corresponding to the second feature of each frame.

[0108] In addition, when the third processing unit 504 obtains the third feature corresponding to the second feature of each frame based on the second feature of each frame and the adjacent feature frames of the second feature of each frame, it may also include the following contents: obtaining the frame number of the obtained second feature; when it is determined that the obtained number of frames meets the first preset requirement, obtaining the third feature corresponding to the second feature of each frame based on the second feature of each frame and the adjacent feature frames of the second feature of each frame.

[0109] The third processing unit 504 may determine that the number of frames of the obtained second feature meets the first preset requirement when determining that the number of frames of the obtained second feature exceeds the first frame number threshold.

[0110] In this embodiment, after the third processing unit 504 obtains the third feature corresponding to the second feature of each frame, the synthesis unit 505 obtains audio data corresponding to the third feature of each frame based on the third feature of each frame and the adjacent feature frames of the third feature of each frame.

[0111] Specifically, when the synthesis unit 505 obtains the audio data corresponding to the third feature of each frame based on the third feature of each frame and the adjacent feature frames of the third feature of each frame, an optional implementation method that can be adopted is: determining the adjacent feature frames of the third feature of each frame from multiple frames of third features; intercepting the filling feature from the adjacent feature frames of the third feature of each frame, and filling the intercepted filling feature into the third feature of each frame; and obtaining the audio data corresponding to the third feature of each frame based on the third feature of each frame after filling.

[0112] Similarly, when the synthesis unit 505 intercepts filling features from the adjacent feature frames of the third feature of each frame and fills the intercepted filling features into the third feature of each frame, it can intercept the features located at the first preset position of the adjacent feature frame as the filling features based on the positional relationship between the third feature of each frame and the adjacent feature frame, and fill the intercepted filling features into the second preset position of the third feature of each frame; there is a corresponding relationship between the first preset position and the second preset position.

[0113] That is to say, the synthesis unit 505 uses the filling features intercepted from the adjacent feature frames of the third feature of each frame to fill the third feature of each frame, so as to obtain the audio data corresponding to the third feature of each frame. Since the third feature of each frame after the filling process contains some features in the adjacent feature frames, the boundary of the third feature of each frame can be accurately calculated, thereby improving the coherence between the audio data corresponding to different third features.

[0114] When the synthesis unit 505 obtains the audio data corresponding to the third feature of each frame based on the third feature of each frame after filling, an optional implementation method that can be adopted is: obtaining the audio data of the second feature of each frame after filling; removing the audio data corresponding to the filling feature in the audio data, and obtaining the audio data corresponding to the third feature of each frame.

[0115] That is to say, when the synthesis unit 505 obtains the audio data corresponding to the third feature of each frame, it can also remove the audio data corresponding to the filling feature in the audio data to avoid duplication between the generated audio data corresponding to different third features.

[0116] When the synthesis unit 505 obtains the audio data corresponding to the third feature of each frame based on the third feature of each frame and the adjacent feature frames of the third feature of each frame, it can also input the third feature of each frame and the adjacent feature frames of the third feature of each frame into a pre-trained speech synthesis model, and the voice coder in the speech synthesis model processes the third feature of each frame and the adjacent feature frames of the third feature of each frame to obtain the audio data corresponding to the third feature of each frame.

[0117] In addition, when the synthesis unit 505 obtains the audio data corresponding to the third feature of each frame based on the third feature of each frame and the adjacent feature frames of the third feature of each frame, it may also include the following contents: obtaining the frame number of the obtained third feature; when it is determined that the obtained number of frames meets the second preset requirement, obtaining the audio data corresponding to the third feature of each frame based on the third feature of each frame and the adjacent feature frames of the third feature of each frame.

[0118] The synthesis unit 505 may determine that the number of frames of the obtained third feature meets the second preset requirement when determining that the number of frames of the obtained third feature exceeds the second frame number threshold.

[0119] After obtaining the audio data corresponding to the third feature of each frame, the synthesis unit 505 can return the obtained audio data to the input end in sequence. Since the purpose of streaming speech synthesis is achieved, this embodiment can reduce the response time of speech synthesis and improve the speed of speech synthesis.

[0120] Figure 6 is a schematic diagram according to the sixth embodiment of the present disclosure. Figure 6 As shown, the speech synthesis model training device 600 of this embodiment includes:

[0121] A second acquisition unit 601 is used to acquire a data set, wherein the data set includes a plurality of training texts and training audio data of the plurality of training texts;

[0122] A construction unit 602 is used to construct a neural network model including an encoder, a decoder, a posterior network layer and a vocoder, wherein the encoder is used to obtain a predicted first feature according to a phoneme sequence of a training text, the decoder is used to obtain a multi-frame predicted second feature according to the predicted first feature, the posterior network layer is used to obtain a predicted third feature corresponding to the predicted second feature of each frame according to the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame, and the vocoder is used to obtain predicted audio data corresponding to the predicted third feature of each frame according to the predicted third feature of each frame and the adjacent feature frames of the predicted third feature of each frame;

[0123] The training unit 603 is used to train the neural network model based on the multiple training texts and the training audio data of the multiple training texts until the neural network model converges to obtain the speech synthesis model.

[0124] After the second acquisition unit 601 acquires the training audio data including multiple training texts and the multiple training texts, the data set can also be divided into a training set, an evaluation set and a test set; wherein the training set is used to train the neural network model, the evaluation set is used to evaluate the neural network model during the training process, and the test set is used to test the effect of the trained neural network model.

[0125] After acquiring the data set, the second acquisition unit 601 can also extract training acoustic features from the training audio data, and then align the phoneme sequence of the training text with the extracted training acoustic features; the purpose of extracting training acoustic features from the training audio data is to calculate the loss function of the encoder and decoder in the neural network model, and then update the parameters in the encoder and decoder according to the loss function.

[0126] In this embodiment, after the second acquisition unit 601 acquires the data set, the construction unit 602 constructs a neural network model including an encoder, a decoder, a posterior network layer and a vocoder.

[0127] In the neural network model constructed by the construction unit 602, the encoder, decoder and posterior network layer constitute an acoustic model, which is used to convert the input text into acoustic features, such as Mel spectrum features; the structure of the acoustic model in this embodiment is based on the Tacotron2 structure.

[0128] In the neural network model constructed by the construction unit 602, the encoder is used to obtain a predicted first feature based on the phoneme sequence of the training text, and the predicted first feature is the encoder feature.

[0129] In the neural network model constructed by the construction unit 602, the decoder is used to obtain a multi-frame predicted second feature based on the predicted first feature, and the predicted second feature is, for example, an acoustic feature of a Mel spectrum.

[0130] In the neural network model constructed by the construction unit 602, the decoder outputs the predicted second feature of the current frame and the decoder state of the current frame based on the predicted first feature, the predicted second feature of the previous frame and the decoder state of the previous frame, that is, the decoder in this embodiment outputs the predicted second feature frame by frame.

[0131] In the neural network model constructed by the construction unit 602, the posterior network layer is used to obtain a predicted third feature corresponding to the predicted second feature of each frame based on the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame; the posterior network layer in this embodiment can be a Post Net (posterior network) composed of a 5-layer convolutional network.

[0132] Specifically, in the neural network model constructed by the construction unit 602, when the posterior network layer obtains the predicted third feature corresponding to the predicted second feature of each frame based on the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame, an optional implementation method that can be adopted is: determining the adjacent feature frames of the predicted second feature of each frame from the second features predicted by multiple frames; intercepting the filling feature from the adjacent feature frames of the predicted second feature of each frame, and filling the intercepted filling feature into the predicted second feature of each frame; obtaining the predicted third feature corresponding to the predicted second feature of each frame based on the predicted second feature of each frame after filling.

[0133] In the neural network model constructed by the construction unit 602, when the posterior network layer predicts the second feature of each frame after filling and obtains the predicted third feature corresponding to the predicted second feature of each frame, an optional implementation method that can be adopted is: obtaining the posterior features of the predicted second feature of each frame after filling, the posterior features in this embodiment are the features obtained by the posterior network layer processing the input predicted second features after filling; removing the posterior features corresponding to the filling features in the posterior features, and obtaining the predicted third features corresponding to the predicted second features of each frame.

[0134] In addition, in the neural network model constructed by the construction unit 602, when the posterior network layer obtains the predicted third feature corresponding to the predicted second feature of each frame based on the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame, it can also include the following contents: obtaining the number of frames of the obtained predicted second feature; when it is determined that the obtained number of frames meets the first preset requirement, obtaining the predicted third feature corresponding to the predicted second feature of each frame based on the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame.

[0135] The posterior network layer of this embodiment can determine that the obtained number of frames of the predicted second feature meets the first preset requirement when it is determined that the obtained number of frames of the predicted second feature exceeds the first frame number threshold.

[0136] In the neural network model constructed by the construction unit 602, the vocoder is used to obtain predicted audio data corresponding to the predicted third feature of each frame based on the predicted third feature of each frame and the adjacent feature frames of the predicted third feature of each frame.

[0137] Specifically, in the neural network model constructed by the construction unit 602, when the voice coder obtains the predicted audio data corresponding to the predicted third feature of each frame based on the predicted third feature of each frame and the adjacent feature frames of the predicted third feature of each frame, an optional implementation method that can be adopted is: determining the adjacent feature frames of the predicted third feature of each frame from multiple frames of predicted third features; intercepting filling features from the adjacent feature frames of the predicted third feature of each frame, and filling the intercepted filling features into the predicted third feature of each frame; and obtaining the predicted audio data corresponding to the predicted third feature of each frame based on the predicted third feature of each frame after filling.

[0138] In the neural network model constructed by the construction unit 602, when the voice coder predicts the third feature based on each frame after filling and obtains the predicted audio data corresponding to the predicted third feature of each frame, an optional implementation method that can be adopted is: obtaining the predicted audio data of the third feature predicted for each frame after filling; removing the predicted audio data corresponding to the filling feature in the predicted audio data, and obtaining the predicted audio data corresponding to the predicted third feature of each frame.

[0139] In addition, in the neural network model constructed by the construction unit 602, when the voice coder obtains the audio data corresponding to the predicted third feature of each frame based on the predicted third feature of each frame and the adjacent feature frames of the predicted third feature of each frame, it can also include the following content: obtaining the number of frames of the obtained predicted third feature; when it is determined that the obtained number of frames meets the second preset requirement, obtaining the audio data corresponding to the predicted third feature of each frame based on the predicted third feature of each frame and the adjacent feature frames of the predicted third feature of each frame.

[0140] The vocoder may determine that the obtained number of frames of the predicted third feature meets the second preset requirement when determining that the obtained number of frames of the predicted third feature exceeds the second frame number threshold.

[0141] In this embodiment, after the construction unit 602 constructs a neural network model including an encoder, a decoder, a posterior network layer and a vocoder, the training unit 603 trains the neural network model based on multiple training texts and training audio data of the multiple training texts until the neural network model converges to obtain a speech synthesis model.

[0142] Specifically, when the training unit 603 trains a neural network model based on multiple training texts and training audio data of multiple training texts, an optional implementation method that can be adopted is: inputting the phoneme sequences of multiple training texts into the neural network model respectively, and obtaining the predicted audio data output by the neural network model for each training text; calculating a first loss function value based on the training audio data of multiple training texts and the predicted audio data; adjusting the parameters of the encoder, decoder, posterior network layer and vocoder in the neural network model according to the calculated first loss function value until the first loss function value converges to obtain a speech synthesis model.

[0143] In addition, in the data set acquired by the second acquisition unit 602, if the training acoustic features corresponding to the phoneme sequence of the training text are also acquired, then when the training unit 603 trains the neural network model based on multiple training texts and the training audio data of multiple training texts, an optional implementation method that can be adopted is: inputting the phoneme sequences of multiple training texts into the neural network model respectively, and acquiring the predicted second features and predicted audio data output by the neural network model for each training text; calculating a first loss function value based on the training audio data and the predicted audio data of the multiple training texts, and calculating a second loss function value based on the predicted second features and the training acoustic features of the multiple training texts; adjusting the parameters of the vocoder in the neural network model based on the calculated first loss function value, and adjusting the parameters of the encoder, decoder and posterior network layer in the neural network model based on the calculated second loss function value, until the first loss function value and the second loss function value converge to obtain a speech synthesis model.

[0144] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0145] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0146] like Figure 7, is a block diagram of an electronic device according to a method for training speech synthesis or speech synthesis models according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0147] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0148] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0149] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as speech synthesis or a training method for a speech synthesis model. For example, in some embodiments, speech synthesis or a training method for a speech synthesis model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708.

[0150] In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the speech synthesis or speech synthesis model training method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform speech synthesis or speech synthesis model training methods in any other appropriate manner (e.g., by means of firmware).

[0151] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable speech synthesis or speech synthesis model training device, so that the program code, when executed by the processor or controller, causes the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0153] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0154] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0155] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0156] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server for a distributed system, or a server combined with a blockchain.

[0157] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0158] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A speech synthesis method, comprising: Acquire a text to be processed and obtain a phoneme sequence of the text to be processed; Encoding the phoneme sequence to obtain a first feature; Decoding the first feature to obtain multiple frames of second features; According to the second feature of each frame and the adjacent feature frames of the second feature of each frame, a third feature corresponding to the second feature of each frame is obtained; According to the third feature of each frame and the adjacent feature frames of the third feature of each frame, audio data corresponding to the third feature of each frame is obtained, wherein the adjacent feature frames of the third feature of each frame are two feature frames adjacent to the third feature of each frame; Wherein, obtaining the third feature corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frame of the second feature of each frame includes: Determine adjacent feature frames of each frame of the second feature from the multiple frames of the second feature; Intercepting a filling feature from a feature frame adjacent to the second feature of each frame, and filling the filling feature into the second feature of each frame; According to the second feature of each frame after filling, the third feature corresponding to the second feature of each frame is obtained.

2. The method according to claim 1, wherein: The obtaining, according to the second feature of each frame and the adjacent feature frames of the second feature of each frame, a third feature corresponding to the second feature of each frame comprises: Obtaining the frame number of the second feature; When it is determined that the number of frames meets the first preset requirement, a third feature corresponding to the second feature of each frame is obtained according to the second feature of each frame and the adjacent feature frames of the second feature of each frame.

3. The method according to claim 1, wherein: The obtaining, according to the filled second feature of each frame, the third feature corresponding to the second feature of each frame comprises: Obtaining a posterior feature of the second feature of each frame after the padding; The a posteriori features corresponding to the filling features are removed from the a posteriori features to obtain the third features corresponding to the second features of each frame.

4. The method according to claim 1, wherein: The obtaining of audio data corresponding to the third feature of each frame according to the third feature of each frame and the adjacent feature frames of the third feature of each frame comprises: Obtaining the frame number of the third feature; When it is determined that the number of frames meets the second preset requirement, audio data corresponding to the third feature of each frame is obtained according to the third feature of each frame and the adjacent feature frames of the third feature of each frame.

5. The method according to claim 1, wherein: The obtaining of audio data corresponding to the third feature of each frame according to the third feature of each frame and the adjacent feature frames of the third feature of each frame comprises: Determine, from the multiple frames of third features, adjacent feature frames of the third feature of each frame; The filling feature is intercepted from the adjacent feature frame of the third feature of each frame, and the filling feature is Fill to the third feature of each frame; According to the third feature of each frame after filling, the audio data corresponding to the third feature of each frame is obtained.

6. The method according to claim 5, wherein: The obtaining, according to the third feature of each frame after filling, the audio data corresponding to the third feature of each frame comprises: Acquire audio data of the third feature of each frame after the padding; The audio data corresponding to the filling feature in the audio data is removed to obtain the audio data corresponding to the third feature of each frame.

7. The method according to claim 1, wherein: The encoding of the phoneme sequence to obtain the first feature comprises: Inputting the phoneme sequence into a speech synthesis model; The phoneme sequence is encoded by an encoder in the speech synthesis model to obtain the first feature output by the encoder.

8. The method according to claim 1, wherein: The decoding of the first feature to obtain multiple frames of second features includes: Inputting the first feature into a speech synthesis model; The first feature is decoded by a decoder in the speech synthesis model to obtain the multiple frames of second features output by the decoder.

9. The method according to claim 1, wherein: The obtaining, according to the second feature of each frame and the adjacent feature frames of the second feature of each frame, a third feature corresponding to the second feature of each frame comprises: Inputting the second feature of each frame and the adjacent feature frames of the second feature of each frame into a speech synthesis model; The posterior network layer in the speech synthesis model processes the second feature of each frame and the adjacent feature frames of the second feature of each frame to obtain the third feature corresponding to the second feature of each frame output by the posterior network layer.

10. The method according to claim 1, wherein: The obtaining of audio data corresponding to the third feature of each frame according to the third feature of each frame and the adjacent feature frames of the third feature of each frame comprises: Inputting the third feature of each frame and adjacent feature frames of the third feature of each frame into a speech synthesis model; The vocoder in the speech synthesis model processes the third feature of each frame and the adjacent feature frames of the third feature of each frame to obtain the audio data corresponding to the third feature of each frame output by the vocoder.

11. A method for training a speech synthesis model, comprising: Acquire a data set, wherein the data set includes a plurality of training texts and training audio data of the plurality of training texts; Constructing a neural network model including an encoder, a decoder, a posterior network layer and a vocoder, wherein the encoder is used to obtain a predicted first feature according to a phoneme sequence of a training text, the decoder is used to obtain a multi-frame predicted second feature according to the predicted first feature, the posterior network layer is used to obtain a predicted third feature corresponding to the predicted second feature of each frame according to the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame, and the vocoder is used to obtain predicted audio data corresponding to the predicted third feature of each frame according to the predicted third feature of each frame and the adjacent feature frames of the predicted third feature of each frame, wherein the adjacent feature frames of the predicted third feature of each frame are two feature frames adjacent to and before the predicted third feature of each frame; Training the neural network model based on the multiple training texts and the training audio data of the multiple training texts until the neural network model converges to obtain the speech synthesis model; The posterior network layer obtains the predicted third feature corresponding to the predicted second feature of each frame according to the predicted second feature of each frame and the adjacent feature frame of the predicted second feature of each frame, including: Determine, from the plurality of frames predicting the second feature, adjacent feature frames for predicting the second feature of each frame; Extracting filling features from adjacent feature frames of the second feature predicted in each frame, and filling the filling features into the second feature predicted in each frame; According to the predicted second feature of each frame after filling, the predicted third feature corresponding to the predicted second feature of each frame is obtained.

12. The method according to claim 11, wherein: The posterior network layer predicts the second feature of each frame after filling to obtain the predicted third feature corresponding to the predicted second feature of each frame, including: Obtaining a posterior feature of the second feature predicted for each frame after the padding; The a posteriori features corresponding to the filling features are removed from the a posteriori features to obtain the predicted third features corresponding to the predicted second features for each frame.

13. The method according to any one of claims 11 to 12, wherein: The vocoder obtains predicted audio data corresponding to the predicted third feature of each frame according to the predicted third feature of each frame and the adjacent feature frames of the predicted third feature of each frame, including: Determine, from the plurality of frames predicting the third feature, adjacent feature frames for predicting the third feature of each frame; Extracting filling features from adjacent feature frames of the third feature predicted in each frame, and filling the filling features into the third feature predicted in each frame; According to the predicted third feature of each frame after filling, the predicted audio data corresponding to the predicted third feature of each frame is obtained.

14. The method according to claim 13, wherein: The vocoder predicts the third feature according to each frame after padding, and obtains the predicted audio data corresponding to the predicted third feature of each frame, including: Obtaining predicted audio data of the third feature of each frame after the padding; The predicted audio data corresponding to the filling feature in the predicted audio data is removed to obtain the predicted audio data corresponding to the third feature predicted for each frame.

15. A speech synthesis device, comprising: A first acquisition unit is used to acquire a text to be processed and obtain a phoneme sequence of the text to be processed; A first processing unit, configured to encode the phoneme sequence to obtain a first feature; A second processing unit is used to decode the first feature to obtain multiple frames of second features; A third processing unit, configured to obtain a third feature corresponding to the second feature of each frame according to the second feature of each frame and an adjacent feature frame of the second feature of each frame; A synthesis unit, used for obtaining audio data corresponding to the third feature of each frame according to the third feature of each frame and the adjacent feature frames of the third feature of each frame, wherein the adjacent feature frames of the third feature of each frame are two feature frames adjacent to the third feature of each frame; Wherein, when the third processing unit obtains the third feature corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame, the third processing unit specifically performs: Determine adjacent feature frames of each frame of the second feature from the multiple frames of the second feature; Intercepting a filling feature from a feature frame adjacent to the second feature of each frame, and filling the filling feature into the second feature of each frame; According to the second feature of each frame after filling, the third feature corresponding to the second feature of each frame is obtained.

16. The device according to claim 15, wherein: When the third processing unit obtains the third feature corresponding to the second feature of each frame according to the second feature of each frame and the adjacent feature frames of the second feature of each frame, the third processing unit specifically performs: Obtaining the frame number of the second feature; When it is determined that the number of frames meets the first preset requirement, a third feature corresponding to the second feature of each frame is obtained according to the second feature of each frame and the adjacent feature frames of the second feature of each frame.

17. The device according to claim 15, wherein: When the third processing unit obtains the third feature corresponding to the second feature of each frame according to the filled second feature of each frame, the third processing unit specifically performs: Obtaining a posterior feature of the second feature of each frame after the padding; The a posteriori features corresponding to the filling features are removed from the a posteriori features to obtain the third features corresponding to the second features of each frame.

18. The device according to claim 15, wherein: When the synthesis unit obtains the audio data corresponding to the third feature of each frame according to the third feature of each frame and the adjacent feature frames of the third feature of each frame, the synthesis unit specifically performs: Obtaining the frame number of the third feature; When it is determined that the number of frames meets the second preset requirement, audio data corresponding to the third feature of each frame is obtained according to the third feature of each frame and the adjacent feature frames of the third feature of each frame.

19. The device according to claim 15, wherein: When the synthesis unit obtains the audio data corresponding to the third feature of each frame according to the third feature of each frame and the adjacent feature frames of the third feature of each frame, the synthesis unit specifically performs: Determine, from the multiple frames of third features, adjacent feature frames of the third feature of each frame; The filling feature is intercepted from the adjacent feature frame of the third feature of each frame, and the filling feature is Fill to the third feature of each frame; According to the third feature of each frame after filling, the audio data corresponding to the third feature of each frame is obtained.

20. The device according to claim 19, wherein When the synthesis unit obtains the audio data corresponding to the third feature of each frame according to the third feature of each frame after filling, the synthesis unit specifically performs: Acquire audio data of the third feature of each frame after the padding; The audio data corresponding to the filling feature in the audio data is removed to obtain the audio data corresponding to the third feature of each frame.

21. A training device for a speech synthesis model, comprising: A second acquisition unit, configured to acquire a data set, wherein the data set includes a plurality of training texts and training audio data of the plurality of training texts; A construction unit is used to construct a neural network model including an encoder, a decoder, a posterior network layer and a vocoder, wherein the encoder is used to obtain a predicted first feature according to a phoneme sequence of a training text, the decoder is used to obtain a multi-frame predicted second feature according to the predicted first feature, the posterior network layer is used to obtain a predicted third feature corresponding to the predicted second feature of each frame according to the predicted second feature of each frame and the adjacent feature frames of the predicted second feature of each frame, and the vocoder is used to obtain predicted audio data corresponding to the predicted third feature of each frame according to the predicted third feature of each frame and the adjacent feature frames of the predicted third feature of each frame, wherein the adjacent feature frames of the predicted third feature of each frame are two feature frames adjacent to the predicted third feature of each frame; A training unit, configured to train the neural network model based on the multiple training texts and the training audio data of the multiple training texts until the neural network model converges to obtain the speech synthesis model; The posterior network layer obtains the predicted third feature corresponding to the predicted second feature of each frame according to the predicted second feature of each frame and the adjacent feature frame of the predicted second feature of each frame, including: Determine, from the plurality of frames predicting the second feature, adjacent feature frames for predicting the second feature of each frame; Extracting filling features from adjacent feature frames of the second feature predicted in each frame, and filling the filling features into the second feature predicted in each frame; According to the predicted second feature of each frame after filling, the predicted third feature corresponding to the predicted second feature of each frame is obtained.

22. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

23. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 14.

24. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images