Speech synthesis method and apparatus, storage medium, electronic device

Abstract

This application discloses a speech synthesis method, apparatus, storage medium, and electronic device. The method includes acquiring text data; obtaining speech audio data synthesized from the text data through a preset NAT processing model, wherein the preset NAT processing model includes an encoder, a Gaussian upsampling module, and a decoder; the encoder employs a unidirectional long short-term memory network and a recurrent neural network with controllable inverse delay; the Gaussian upsampling module performs Gaussian upsampling according to preset blocks; the decoder includes a Mel vocoder based on a neural network and an LPL vocoder based on a neural network; and obtaining the synthesized result of the text data based on the processing results of the Mel vocoder and the LPL vocoder based on the neural network. This application solves the technical problems of latency and inability to synthesize speech in the overall speech synthesis system.

Description

Technical Field

[0001] This application relates to the field of text data to speech processing, and more specifically, to a speech synthesis method, apparatus, storage medium, and electronic device. Background Technology

[0002] Google introduced Tacotron end-to-end speech synthesis technology in 2017, Tacotron 2 end-to-end technology in 2018, and non-attentive Tacotron in 2020, which greatly improved the sound quality and stability of end-to-end speech synthesis models.

[0003] However, because the encoder contains a bidirectional LSTM model, Gaussian upsampling needs to be calculated on all encoder outputs, which cannot meet the requirements of speech synthesis on some processors with limited computing power. Furthermore, the acoustic parameters output by the Tacotron are Mel-spectrum data, which cannot be used in LPC vocoders.

[0004] There is currently no effective solution to the problems of latency and inability to synthesize speech in the overall speech synthesis system of related technologies. Summary of the Invention

[0005] The main objective of this application is to provide a speech synthesis method, apparatus, storage medium, and electronic device to solve the problems of delay and inability to synthesize speech in the overall speech synthesis system.

[0006] To achieve the above objectives, according to one aspect of this application, a speech synthesis method is provided for a client.

[0007] The speech synthesis method according to this application includes: acquiring text data; obtaining speech audio data through a preset NAT processing model, wherein the preset NAT processing model includes an encoder, a Gaussian upsampling module, and a decoder; the encoder employs a unidirectional long short-term memory network and a recurrent neural network with controllable inverse delay; the Gaussian upsampling module performs Gaussian upsampling according to preset blocks; the decoder includes a Mel vocoder based on a neural network and an LPL vocoder based on a neural network; and processing the speech audio data according to the Mel vocoder based on the neural network and the LPL vocoder based on the neural network to obtain a synthesized result of the speech audio data.

[0008] Furthermore, the decoder includes a neural network-based Mel vocoder and a neural network-based LPL vocoder, comprising: determining the neural network-based Mel vocoder and the neural network-based LPL vocoder using predicted Mel feature vectors and predicted LPL feature vectors, to obtain the synthesized result.

[0009] Furthermore, the encoder replaces the bidirectional long short-term memory network with the unidirectional long short-term memory network and the reverse delay-controllable recurrent neural network to control the delay of the encoder.

[0010] Furthermore, the Gaussian upsampling module performs Gaussian upsampling according to a preset block as the input to the encoder, and the input is related to the block size.

[0011] Furthermore, obtaining the speech and audio data synthesized from the text data through the preset NAT processing model includes controlling the delay of the entire system by controlling the delay of the encoder and the Gaussian upsampling module.

[0012] Furthermore, the step of processing the speech audio data according to the Mel vocoder of the neural network and the LPL vocoder based on the neural network to obtain the synthesized result of the speech audio data includes: processing the speech audio data according to the Mel vocoder of the neural network and the LPL vocoder based on the neural network to obtain different synthesized sound qualities or synthesized prosody in the speech audio data.

[0013] To achieve the above objectives, according to another aspect of this application, a speech synthesis method is provided for use on a server.

[0014] The speech synthesis method according to this application includes: receiving text data from a client; parsing the text data to synthesize speech and audio data through a preset NAT processing model, wherein the preset NAT processing model includes an encoder, a Gaussian upsampling module, and a decoder; the encoder employs a unidirectional long short-term memory network and a recurrent neural network with controllable inverse delay; the Gaussian upsampling module performs Gaussian upsampling according to preset blocks; the decoder includes a Mel vocoder based on a neural network and an LPL vocoder based on a neural network; and sending the synthesized result of the speech and audio data obtained by processing the speech and audio data using the Mel vocoder based on the neural network and the LPL vocoder based on the neural network to the client.

[0015] To achieve the above objectives, according to another aspect of this application, a speech synthesis apparatus is provided.

[0016] The speech synthesis apparatus according to this application includes: an acquisition module for acquiring text data; a processing module for obtaining speech audio data synthesized from the text data through a preset NAT processing model, wherein the preset NAT processing model includes an encoder, a Gaussian upsampling module, and a decoder, the encoder employing a unidirectional long short-term memory network and a recurrent neural network with controllable inverse delay, the Gaussian upsampling module performing Gaussian upsampling according to preset blocks, and the decoder including a neural network-based Mel vocoder and a neural network-based LPL vocoder; and a synthesis module for processing the speech audio data according to the neural network-based Mel vocoder and the neural network-based LPL vocoder to obtain a synthesized result of the speech audio data.

[0017] To achieve the above objectives, according to another aspect of this application, a storage medium is also provided, wherein the storage medium stores a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0018] To achieve the above objectives, according to another aspect of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0019] In the embodiments of this application, the speech synthesis method, apparatus, storage medium, and electronic device acquire text data and obtain speech audio data synthesized from the text data through a preset NAT processing model. This achieves the goal of obtaining the synthesis result of the text data based on the Mel vocoder of the neural network and the processing result of the LPL vocoder based on the neural network, thereby realizing the technical effects of controlling latency and being able to synthesize speech, and thus solving the technical problems of latency and inability to synthesize speech in the overall speech synthesis system. Attached Figure Description

[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application. In the drawings:

[0021] Figure 1 This is a schematic diagram of the hardware structure of the speech synthesis method according to an embodiment of this application;

[0022] Figure 2 This is a schematic flowchart of a speech synthesis method according to an embodiment of this application;

[0023] Figure 3This is a schematic diagram of the structure of a speech synthesis device according to an embodiment of this application;

[0024] Figure 4 This is a schematic flowchart of a speech synthesis method according to an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the encoder and upsampling structure of the speech synthesis method according to an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the decoder structure of the speech synthesis method according to an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] In this application, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing this application and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation.

[0030] Furthermore, in addition to indicating location or positional relationship, some of the aforementioned terms may also have other meanings. For example, the term "above" may also be used in some cases to indicate a certain dependency or connection relationship. Those skilled in the art can understand the specific meaning of these terms in this application based on the specific circumstances.

[0031] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linking," and "socketing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral structure; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of these terms in this application based on the specific circumstances.

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] like Figure 1 The diagram shown is a hardware structure schematic of a speech synthesis method according to an embodiment of this application, including a user 100, a NAT processing model 200, a server 300, and a speech synthesis result 400. The user 100 inputs speech, which is then converted into a speech signal and processed by the NAT processing model 200. The server 300 provides the computing power required by the NAT processing model 200. The speech synthesis result 400 is the speech output of the NAT processing model 200.

[0034] like Figure 2 As shown, the method includes the following steps S201 to S203:

[0035] Step S201: Obtain text data;

[0036] Step S202: Obtain speech and audio data synthesized from text data through a preset NAT processing model. The preset NAT processing model includes an encoder, a Gaussian upsampling module, and a decoder. The encoder uses a unidirectional long short-term memory network and a recurrent neural network with controllable inverse delay. The Gaussian upsampling module performs Gaussian upsampling according to preset blocks. The decoder includes a Mel vocoder based on a neural network and an LPL vocoder based on a neural network.

[0037] Step S203: Process the speech audio data according to the Mel vocoder of the neural network and the LPL vocoder based on the neural network to obtain the synthesized result of the speech audio data.

[0038] As can be seen from the above description, this application achieves the following technical effects:

[0039] By acquiring text data and using a preset NAT processing model, speech audio data synthesized from the text data is obtained. This achieves the goal of obtaining the synthesized result of the text data based on the Mel vocoder of the neural network and the LPL vocoder based on the neural network. This realizes the technical effects of controlling latency and synthesizing speech, thereby solving the technical problems of latency and inability to synthesize speech in the overall speech synthesis system.

[0040] In step S201 above, text data can be obtained through various means, including but not limited to real-time text data of users and historical text data of users.

[0041] As an optional implementation, the text data is the data to be used for speech synthesis.

[0042] In one preferred embodiment, the text data is arranged chronologically to obtain a speech sequence.

[0043] In step S202 above, the speech audio data synthesized from the text data is obtained through a preset NAT processing model. The Non-Attentive Tacotron model in related technologies improves the sound quality and stability of the end-to-end speech synthesis model. However, since the encoder contains a bidirectional LSTM model, Gaussian upsampling needs to be calculated on all encoder outputs, which cannot meet the requirements of speech synthesis on some processors with limited computing power. Furthermore, the acoustic parameters output by the Tacotron are Mel-spectrum data, which cannot be used in LPC vocoders. Therefore, the preset NAT processing model includes an improved encoder, a Gaussian upsampling module, and a decoder.

[0044] In a preferred embodiment, the encoder employs a unidirectional long short-term memory network and a recurrent neural network with controllable reverse delay.

[0045] In the encoder, a unidirectional long short-term memory network and a reverse delay-controlled recurrent neural network are used instead of a bidirectional long short-term memory network (developed by Shaofei Xue and Zhijie Yan in 2017 in Improving latency-controlled BLSTM acoustic models for online speech recognition). By using this model for the first time in the encoder part of speech synthesis, the synthesis ensures that the delay of the encoder is controllable.

[0046] Preferably, the encoder uses the unidirectional long short-term memory network and the reverse delay-controllable recurrent neural network instead of the bidirectional long short-term memory network to control the delay of the encoder.

[0047] In a preferred embodiment, the Gaussian upsampling module performs Gaussian upsampling according to a preset block.

[0048] The Gaussian upsampling module calculates the encoder input used by block-by-block Gaussian upsampling, which is only related to the block size, thus achieving controllable delay.

[0049] Preferably, the Gaussian upsampling module performs Gaussian upsampling according to a preset block as the input of the encoder, and the input is related to the block size.

[0050] In a preferred embodiment, the decoder includes a neural network-based Mel vocoder and a neural network-based LPL vocoder.

[0051] The autoregressive decoder employs simultaneous prediction of Mel spectra and linear prediction coefficients, supporting the synthesis of sound using Mel neural network vocoders and LPC vocoders.

[0052] In step S203 above, the speech audio data is processed according to the Mel vocoder of the neural network and the LPL vocoder based on the neural network to obtain the synthesized result of the speech audio data.

[0053] Preferably, the decoder includes a neural network-based Mel vocoder and a neural network-based LPL vocoder, comprising: determining the neural network-based Mel vocoder and the neural network-based LPL vocoder using predicted Mel feature vectors and predicted LPL feature vectors, to obtain a synthesis result.

[0054] In a preferred embodiment, obtaining the speech and audio data synthesized from the text data through a preset NAT processing model includes controlling the delay of the entire system by controlling the delay of the encoder and the Gaussian upsampling module.

[0055] In practice, the delay of the entire system is controlled by controlling the delay of the encoder and each Gaussian upsampling module in upsampling according to a preset block.

[0056] In a preferred embodiment, the step of processing the speech audio data according to the Mel vocoder of the neural network and the LPL vocoder based on the neural network to obtain the synthesized result of the speech audio data includes: processing the speech audio data according to the Mel vocoder of the neural network and the LPL vocoder based on the neural network to obtain different synthesized sound qualities or synthesized prosody in the speech audio data.

[0057] In practice, the speech audio data is processed at the user end or server end according to the Mel vocoder of the neural network and the LPL vocoder based on the neural network to obtain different synthesized sound qualities or synthesized prosody in the speech audio data.

[0058] like Figure 4 As shown, the speech synthesis method is used on the server side, and the method includes the following steps S401 to S403:

[0059] Step S401: Receive text data from the client;

[0060] Step S402: Parse the speech and audio data synthesized from text data using a preset NAT processing model. The preset NAT processing model includes an encoder, a Gaussian upsampling module, and a decoder. The encoder uses a unidirectional long short-term memory network and a recurrent neural network with controllable inverse delay. The Gaussian upsampling module performs Gaussian upsampling according to preset blocks. The decoder includes a Mel vocoder based on a neural network and an LPL vocoder based on a neural network.

[0061] Step S403: The synthesized result of the speech audio data obtained by the Mel vocoder based on the neural network and the LPL vocoder based on the neural network processing the speech audio data is sent to the client.

[0062] In step 401 above, text data can be obtained on the server side through various means, including but not limited to real-time text data of users and historical text data of users.

[0063] As an optional implementation, the text data is the data to be used for speech synthesis.

[0064] In one preferred embodiment, the text data is arranged chronologically to obtain a speech sequence.

[0065] In step S402 above, the server obtains the speech audio data synthesized from the text data through a preset NAT processing model. The Non-Attentive Tacotron model in related technologies improves the sound quality and stability of the end-to-end speech synthesis model. However, since the encoder contains a bidirectional LSTM model, Gaussian upsampling needs to be calculated on all encoder outputs, which cannot meet the requirements of speech synthesis on some processors with limited computing power. Furthermore, the acoustic parameters output by the Tacotron are Mel-spectrum data, which cannot be used in LPC vocoders. Therefore, the preset NAT processing model includes an improved encoder, a Gaussian upsampling module, and a decoder.

[0066] In a preferred embodiment, the encoder employs a unidirectional long short-term memory network and a recurrent neural network with controllable reverse delay.

[0067] In the encoder, a unidirectional long short-term memory network and a reverse delay-controlled recurrent neural network are used instead of a bidirectional long short-term memory network (developed by Shaofei Xue and Zhijie Yan in 2017 in Improving latency-controlled BLSTM acoustic models for online speech recognition). By using this model for the first time in the encoder part of speech synthesis, the synthesis ensures that the delay of the encoder is controllable.

[0068] Preferably, the encoder uses the unidirectional long short-term memory network and the reverse delay-controllable recurrent neural network instead of the bidirectional long short-term memory network to control the delay of the encoder.

[0069] In a preferred embodiment, the Gaussian upsampling module performs Gaussian upsampling according to a preset block.

[0070] The Gaussian upsampling module calculates the encoder input used by block-by-block Gaussian upsampling, which is only related to the block size, thus achieving controllable delay.

[0071] Preferably, the Gaussian upsampling module performs Gaussian upsampling according to a preset block as the input of the encoder, and the input is related to the block size.

[0072] In a preferred embodiment, the decoder includes a neural network-based Mel vocoder and a neural network-based LPL vocoder.

[0073] The autoregressive decoder employs simultaneous prediction of Mel spectra and linear prediction coefficients, supporting the synthesis of sound using Mel neural network vocoders and LPC vocoders.

[0074] In step S403 above, the server sends the synthesized result of the speech audio data obtained by processing the speech audio data using the Mel vocoder based on the neural network and the LPL vocoder based on the neural network to the client.

[0075] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0076] According to an embodiment of this application, a speech synthesis apparatus for implementing the above method is also provided, such as... Figure 3 As shown, the device includes:

[0077] Module 301 is used to acquire text data;

[0078] Processing module 302 is used to obtain speech and audio data synthesized from text data through a preset NAT processing model. The preset NAT processing model includes an encoder, a Gaussian upsampling module, and a decoder. The encoder adopts a unidirectional long short-term memory network and a recurrent neural network with controllable inverse delay. The Gaussian upsampling module performs Gaussian upsampling according to preset blocks. The decoder includes a Mel vocoder based on a neural network and an LPL vocoder based on a neural network.

[0079] The synthesis module 303 is used to process speech audio data according to the Mel vocoder of the neural network and the LPL vocoder based on the neural network to obtain the synthesized result of the speech audio data.

[0080] The acquisition module 301 in this application embodiment can acquire text data in various ways, including but not limited to real-time text data of users and historical text data of users.

[0081] As an optional implementation, the text data is the data to be used for speech synthesis.

[0082] In one preferred embodiment, the text data is arranged chronologically to obtain a speech sequence.

[0083] In the processing module 302 of this embodiment, speech audio data synthesized from text data is obtained through a preset NAT processing model. The Non-Attentive Tacotron model in related technologies improves the sound quality and stability of the end-to-end speech synthesis model. However, since the encoder contains a bidirectional LSTM model, Gaussian upsampling needs to be calculated on all encoder outputs, which cannot meet the requirements of speech synthesis on some processors with limited computing power. Furthermore, the acoustic parameters output by the Tacotron are Mel-spectrum data, which cannot be used in LPC vocoders. Therefore, the preset NAT processing model includes an improved encoder, a Gaussian upsampling module, and a decoder.

[0084] In a preferred embodiment, the encoder employs a unidirectional long short-term memory network and a recurrent neural network with controllable reverse delay.

[0085] In the encoder, a unidirectional long short-term memory network and a reverse delay-controllable recurrent neural network are used instead of a bidirectional long short-term memory network. By using this model for the first time in the encoder part of speech synthesis, the delay of the encoder is guaranteed to be controllable.

[0086] Preferably, the encoder uses the unidirectional long short-term memory network and the reverse delay-controllable recurrent neural network instead of the bidirectional long short-term memory network to control the delay of the encoder.

[0087] In a preferred embodiment, the Gaussian upsampling module performs Gaussian upsampling according to a preset block.

[0088] The Gaussian upsampling module calculates the encoder input used by block-by-block Gaussian upsampling, which is only related to the block size, thus achieving controllable delay.

[0089] Preferably, the Gaussian upsampling module performs Gaussian upsampling according to a preset block as the input of the encoder, and the input is related to the block size.

[0090] In a preferred embodiment, the decoder includes a neural network-based Mel vocoder and a neural network-based LPL vocoder.

[0091] The autoregressive decoder employs simultaneous prediction of Mel spectra and linear prediction coefficients, supporting the synthesis of sound using Mel neural network vocoders and LPC vocoders.

[0092] The synthesis module 303 of this application embodiment is used to synthesize the speech audio data obtained by processing the speech audio data using the Mel vocoder of the neural network and the LPL vocoder based on the neural network.

[0093] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0094] To better understand the above speech synthesis method process, the following explanation of the technical solution is provided in conjunction with preferred embodiments, but it is not intended to limit the technical solution of the embodiments of the present invention.

[0095] The speech synthesis method in this application, by controlling the delay of the encoder and Gaussian upsampling, can ensure high-quality synthesized sound quality and rhythmic effects while controlling the overall system delay. Through a multi-output decoder, different vocoders can be connected after the acoustic model to achieve high-quality, expressive speech synthesis.

[0096] Explanation of technical terms:

[0097] LSTM: Long Short-Term Memory Network

[0098] LLbwdRNN: Delay-Controllable Reverse Recurrent Neural Network

[0099] Positional Embedding: Location Information Encoding

[0100] LPC: Linear Prediction Coefficient

[0101] like Figure 5 The diagram shown is a schematic representation of the encoder and upsampling structure of the speech synthesis method according to an embodiment of this application. The specific implementation process includes the following steps:

[0102] Specifically, the Gaussian upsampling module calculates the encoder input used by block-by-block Gaussian upsampling, which is only related to the block size, thus achieving controllable delay.

[0103] In the encoder, a unidirectional long short-term memory network and a reverse delay-controllable recurrent neural network are used instead of a bidirectional long short-term memory network. By using this model for the first time in the encoder part of speech synthesis, the delay of the encoder is guaranteed to be controllable.

[0104] like Figure 6 The diagram shown is a schematic diagram of the decoder structure of the speech synthesis method according to an embodiment of this application. The autoregressive decoder uses simultaneous prediction of Mel spectrum and linear prediction coefficients to support the synthesis of sound using Mel neural network vocoders and LPC vocoders.

[0105] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A speech synthesis method, characterized in that, For use on a client, the method includes: Get text data; The text data is synthesized into speech audio data through a preset NAT processing model. The preset NAT processing model includes an encoder, a Gaussian upsampling module, and a decoder. The encoder adopts a unidirectional long short-term memory network and a recurrent neural network with controllable inverse delay. The Gaussian upsampling module performs Gaussian upsampling according to preset blocks. The decoder includes a Mel vocoder based on a neural network and an LPL vocoder based on a neural network. The speech audio data is processed by the Mel vocoder of the neural network and the LPL vocoder based on the neural network to obtain the synthesized result of the speech audio data; The decoder includes a neural network-based Mel vocoder and a neural network-based LPL vocoder, including: By using the predicted Mel feature vector and the predicted LPL feature vector; A neural network-based Mel vocoder and a neural network-based LPL vocoder are determined to obtain the synthesis result.

2. The method according to claim 1, characterized in that, The encoder replaces the bidirectional long short-term memory network with the unidirectional long short-term memory network and the reverse delay controllable recurrent neural network to control the delay of the encoder.

3. The method according to claim 1, characterized in that, The Gaussian upsampling module performs Gaussian upsampling according to a preset block as the input of the encoder, and the input is related to the block size.

4. The method according to claim 1, characterized in that, The process of obtaining speech and audio data synthesized from text data through a preset NAT processing model includes: The delay of the entire system is controlled by controlling the delay of the encoder and the Gaussian upsampling module.

5. The method according to claim 1, characterized in that, The process of processing speech audio data using the Mel vocoder of the neural network and the LPL vocoder based on the neural network to obtain the synthesized result of the speech audio data includes: The speech audio data is processed by the Mel vocoder of the neural network and the LPL vocoder based on the neural network to obtain different synthesized sound qualities or synthesized prosody in the speech audio data.

6. A speech synthesis method, characterized in that, For use on the server side, the method includes: Receive text data from the client; The text data is parsed to synthesize speech and audio data through a preset NAT processing model. The preset NAT processing model includes an encoder, a Gaussian upsampling module, and a decoder. The encoder adopts a unidirectional long short-term memory network and a recurrent neural network with controllable inverse delay. The Gaussian upsampling module performs Gaussian upsampling according to preset blocks. The decoder includes a Mel vocoder based on a neural network and an LPL vocoder based on a neural network. The decoder includes a neural network-based Mel vocoder and a neural network-based LPL vocoder, including: By using the predicted Mel feature vector and the predicted LPL feature vector; A neural network-based Mel vocoder and a neural network-based LPL vocoder are determined to obtain the synthesis result; The synthesized result of the speech and audio data obtained by processing the speech and audio data using the Mel vocoder based on the neural network and the LPL vocoder based on the neural network is sent to the client.

7. A speech synthesis device, characterized in that, include: The acquisition module is used to acquire text data; The processing module is used to obtain speech and audio data synthesized from text data through a preset NAT processing model. The preset NAT processing model includes an encoder, a Gaussian upsampling module, and a decoder. The encoder adopts a unidirectional long short-term memory network and a recurrent neural network with controllable inverse delay. The Gaussian upsampling module performs Gaussian upsampling according to preset blocks. The decoder includes a Mel vocoder based on a neural network and an LPL vocoder based on a neural network. The decoder includes a neural network-based Mel vocoder and a neural network-based LPL vocoder, including: By using the predicted Mel feature vector and the predicted LPL feature vector; A neural network-based Mel vocoder and a neural network-based LPL vocoder are determined to obtain the synthesis result; The synthesis module is used to process speech audio data according to the Mel vocoder of the neural network and the LPL vocoder based on the neural network to obtain the synthesized result of the speech audio data.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to perform the method according to any one of claims 1 to 6 when it is run.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.

Images