A single-stage speech synthesis method, device, equipment and storage medium

Abstract

The application relates to the technical field of data storage, and discloses a single-stage speech synthesis method, device, equipment and storage medium, which comprises the following steps: receiving inputted to-be-processed text through a single-stage TTS model, performing semantic enhancement processing on the to-be-processed text through a semantic knowledge distillation algorithm, and capturing deep semantic features in the to-be-processed text; generating audio according to the to-be-processed text after the semantic enhancement processing, and randomly masking part of the audio signals by using an audio mask modeling algorithm in the audio generation process to generate masked audio signals; and predicting the audio signals that are not masked according to the to-be-processed text after the semantic enhancement processing and the masked audio signals, so as to generate audio signals that conform to the deep semantic features and have natural emotions. The single-stage TTS model architecture is adopted, the semantic knowledge distillation algorithm and the audio mask modeling algorithm are combined, the generation process is simplified, the reasoning speed is improved, and high-quality audio can be generated in a real-time scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing, and in particular to a single-stage speech synthesis method, device, equipment and storage medium. BACKGROUND

[0002] Speech synthesis, also known as text-to-speech (TTS) technology, can convert any text information into standard fluent speech in real time, and is a frontier technology in the field of Chinese information processing. At present, TTS technology has shown great potential in many application fields such as voice assistants, content creation and voice navigation. Most existing TTS technologies adopt a two-stage or multi-stage architecture, that is, acoustic features (such as mel-spectrogram) are first generated, and then a neural vocoder is used to convert the acoustic features into audio. The inventors found that although this multi-stage processing process improves the quality of speech generation, there are still the following shortcomings in model complexity, inference efficiency, naturalness and diversity of generation, etc.:

[0003] 1) The existing TTS system adopts a series of multiple models (i.e. multi-stage architecture), which introduces additional complexity, increases training time and inference time, and especially in real-time speech generation scenarios, the speech generation efficiency is significantly limited.

[0004] 2) Since each stage in the multi-stage model needs to be trained independently, the generated speech may be affected in consistency, especially the coherence of prosody, voice quality and emotion, and the error propagation between stages may also cause the quality of the speech to decline.

[0005] 3) The existing TTS model usually only relies on text input for speech generation, ignoring the depth understanding of the text semantics, resulting in the model being unable to generate natural speech that meets the semantic context when processing complex sentence structures or meanings.

[0006] 4) The existing TTS system relies on a large amount of annotated speech data for training, especially speech data related to specific styles, emotions or tones, making the TTS technology perform poorly in data-limited scenarios. In addition, obtaining high-quality labeled data is costly, limiting the widespread deployment of TTS systems. SUMMARY

[0007] The present application provides a single-stage speech synthesis method, device, equipment and storage medium to solve the technical problems of model architecture complexity, affecting speech consistency, inability to generate natural speech that meets the semantic context, and model training relying on labeled speech data in existing TTS technology.

[0008] In a first aspect, a single-stage speech synthesis method is provided, comprising:

[0009] The single-stage TTS model receives input to-be-processed text, and performs semantic enhancement processing on the to-be-processed text through a semantic knowledge distillation algorithm to capture deep semantic features in the to-be-processed text.

[0010] Audio generation is performed according to the to-be-processed text after the semantic enhancement processing, and during the audio generation, an audio mask modeling algorithm is used to randomly mask part of the audio signal to generate a masked audio signal.

[0011] Audio signals that conform to the deep semantic features and have natural emotions are generated according to the to-be-processed text after the semantic enhancement processing and the masked audio signal.

[0012] In a second aspect, a single-stage speech synthesis device is provided, comprising:

[0013] A semantic knowledge distillation module is configured to perform semantic enhancement processing on input to-be-processed text through a semantic knowledge distillation algorithm to capture deep semantic features in the to-be-processed text.

[0014] An audio mask modeling module is configured to perform audio generation according to the to-be-processed text after the semantic enhancement processing, and during the audio generation, an audio mask modeling algorithm is used to randomly mask part of the audio signal to generate a masked audio signal.

[0015] An audio generation module is configured to generate audio signals that conform to the deep semantic features and have natural emotions according to the to-be-processed text after the semantic enhancement processing and the masked audio signal.

[0016] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the single-stage speech synthesis method described above when executing the computer program.

[0017] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program, and the computer program implements the steps of the single-stage speech synthesis method described above when executed by a processor.

[0018] In the scheme implemented by the single-stage speech synthesis method, device, computer equipment and storage medium, a single-stage TTS model architecture is adopted, a semantic knowledge distillation algorithm is used for semantic enhancement processing on the text to capture deep semantic features of the text and improve semantic analysis capability of the single-stage TTS model on the text; in the audio generation process, an audio mask modeling algorithm is used for random masking of the audio signal, the semantic knowledge distillation algorithm and the audio mask modeling algorithm are combined, so that the TTS model can directly generate an audio signal meeting semantics and having natural speech characteristics and emotional performance from the text in a single stage. The audio output is directly generated from the text input, which not only reduces complexity of the model, simplifies the generation process, but also improves inference speed, so that the model can generate high-quality audio in a real-time scene. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Figure 1 is an architecture diagram of a single-stage speech synthesis system in an embodiment of the present application;

[0021] Figure 2 is a flowchart of a single-stage speech synthesis method in the first embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a training process of the audio mask modeling algorithm in an embodiment of the present application;

[0023] Figure 4 is a flowchart of a single-stage speech synthesis method in the second embodiment of the present application;

[0024] Figure 5 is a structural schematic diagram of a single-stage speech synthesis device in an embodiment of the present application;

[0025] Figure 6 is a structural schematic diagram of a computer equipment in an embodiment of the present application;

[0026] Figure 7 is another structural schematic diagram of a computer equipment in an embodiment of the present application. DETAILED DESCRIPTION

[0027] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described, obviously, the described embodiments are a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0028] The single-stage speech synthesis method provided by the embodiments of the present application can be applied in application environments such as Figure 1 The client communicates with the server through a network. The server can receive an inputted to-be-processed text, receive the inputted to-be-processed text through a single-stage TTS model, and perform semantic enhancement processing on the to-be-processed text through a semantic knowledge distillation algorithm to capture deep semantic features in the to-be-processed text. Audio is generated according to the to-be-processed text after the semantic enhancement processing, and in the audio generation process, part of the audio signal is randomly masked using an audio mask modeling algorithm to generate a masked audio signal. The audio signal that is not masked is predicted according to the to-be-processed text after the semantic enhancement processing and the masked audio signal, and an audio signal that conforms to the deep semantic features and has natural emotion is generated, and the audio signal is returned to the client. The present application is suitable for speech synthesis in the financial field. For speech synthesis in the financial field, the semantic knowledge distillation result and the audio mask modeling can be combined, so that the TTS model can directly generate an audio signal that conforms to the semantics and has natural speech characteristics and emotional performance from the text in a single stage, to simplify the audio generation process and improve the quality and consistency of audio generation. The client can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The present application will be described in detail through specific embodiments.

[0029] Please refer to Figure 2 , which is a flowchart of a single-stage speech synthesis method in the first embodiment of the present application. The single-stage speech synthesis method provided by the first embodiment of the present application includes the following steps:

[0030] S100: receiving an inputted to-be-processed text, and performing semantic enhancement processing on the to-be-processed text through a semantic knowledge distillation algorithm to capture deep semantic features in the to-be-processed text;

[0031] In this step, semantic knowledge distillation is a technology for migrating knowledge in a complex model to a smaller model. In the embodiment of the present application, the semantic knowledge distillation algorithm is used to perform semantic enhancement processing on the to-be-processed text to capture deep semantic features of the to-be-processed text and improve the semantic analysis capability of the TTS model for the to-be-processed text.

[0032] Specifically, in the embodiment of the present application, the semantic knowledge distillation algorithm uses a pre-trained language model such as BERT (Bidirectional Encoder Representations from Transformers) or GPT (Generative Pre-Trained Transformer) as a teacher model to capture deep semantic features such as sentiment, syntactic structure, and semantic stress in the to-be-processed text. Then, the captured deep semantic features are distilled into the TTS model, and the deep semantic features captured by the teacher model guide the TTS model (i.e., the student model) to generate audio signals. When generating audio signals, the TTS model can more accurately understand deep semantic information such as sentiment, syntactic structure, and semantic stress in the to-be-processed text, thereby generating audio signals that are more consistent with the content of the text and improving the naturalness of audio generation. Taking the BERT model as an example, BERT can better understand the semantics of each word in a sentence by using the encoder structure of the Transformer to learn context information from both the left and right of the text, thereby avoiding the limitations of traditional one-way language models.

[0033] It can be understood that by introducing the semantic knowledge distillation algorithm, the TTS model can better capture deep semantic information in the input text when generating audio signals. The generated audio signals not only maintain a high degree of consistency with the input text in terms of content, but also exhibit detailed features such as emotional color and semantic stress in the input text, thereby enhancing the semantic understanding and generation consistency of the TTS model.

[0034] S110: Perform audio generation according to the to-be-processed text after semantic enhancement processing, and in the audio generation process, use an audio mask modeling algorithm to randomly mask part of the audio signals to generate a masked audio signal;

[0035] In this step, the audio mask modeling (Masked Audio Token Modeling) is a model training strategy in natural language processing. The algorithm is based on the masked language modeling in natural language processing. By randomly masking the input audio signal, the model can better learn the global features of the audio and enhance the understanding and generation ability of the overall text information during the training process, and generate more natural audio signals.

[0036] Specifically, as shown in FIG. 1, it is a training process diagram of the audio mask modeling algorithm in the embodiment of the present application. First, the semantic encoder (Semantic Encoder) understands the meaning of the input audio and converts it into a readable form. The speech discrete cosine transform encoder (Speech Discrete Cosine Transform Encoder, DCE) converts the audio signal into a discrete time spectrum by calculating the discrete cosine transform. Then, the input audio signal is converted into a series of discrete tokens by the speech discrete token (Speech Discrete Token), and the discrete tokens are randomly masked. Finally, the masked discrete tokens and the text embedding are used as the input of the Transformer model. Figure 3

[0037] It can be understood that by introducing the audio mask modeling algorithm, the model can process complex phonology, rhythm, rhythm and pause information during audio generation, and increase the diversity of audio generation. Especially in processing long sentences and complex syntax, the naturalness and diversity of audio generation are improved.

[0038] S120: predicting the unmasked audio signal according to the semantic enhanced text to be processed and the masked audio signal, and generating an audio signal conforming to deep semantic features and having natural emotions;

[0039] In this step, by combining the semantic knowledge distillation result and the audio mask modeling, the TTS model can directly generate an audio signal conforming to the semantics and having natural voice features and emotional performance from the text in a single stage.

[0040] It can be understood that by combining the semantic knowledge distillation result and the audio mask modeling, the model does not need to rely on a large amount of labeled data during audio generation, reducing the demand for labeled speech data, so that excellent generation performance can be maintained in the scene with limited data.

[0041] ​It can be seen that in the above scheme, the single-stage speech synthesis method provided by the embodiment of the present application adopts a single-stage TTS model architecture, uses a semantic knowledge distillation algorithm to perform semantic enhancement processing on the text, so as to capture deep semantic features of the text, and improve the semantic analysis capability of the single-stage TTS model on the text; in the audio generation process, an audio mask modeling algorithm is used to randomly mask the audio signal, and the semantic knowledge distillation algorithm and the audio mask modeling algorithm are combined, so that the TTS model can directly generate an audio signal that conforms to semantics and has natural speech characteristics and emotional performance from the text in a single stage. The embodiment of the present application directly generates audio output from text input, not only reduces the complexity of the model, simplifies the generation process, but also improves the inference speed, so that the model can generate high-quality audio in a real-time scene.

[0042] Referring to Figure 4 , a flowchart of a single-stage speech synthesis method in the second embodiment of the present application. The single-stage speech synthesis method provided by the second embodiment of the present application includes the following steps:

[0043] S200: receiving an input text to be processed by a single-stage TTS model, and preprocessing the text to be processed to convert the text to be processed into a fixed-size text embedding;

[0044] In this step, the TTS model adopts a single-stage architecture (single-stage), which can directly generate audio from text, avoiding the multi-stage processing process of the existing multi-stage TTS model, not only reducing the complexity of the model, but also simplifying the audio generation process and improving the inference speed, so that the model can generate high-quality audio in a real-time scene.

[0045] In the embodiment of the present application, the preprocessing of the text to be processed includes word segmentation, character encoding and text embedding (Text Embedding), which represents words, phrases or sentences in the text to be processed as fixed-length text vectors.

[0046] S210: performing semantic enhancement processing on the converted text embedding by a semantic knowledge distillation algorithm to capture deep semantic features in the text to be processed;

[0047] In this step, the semantic knowledge distillation algorithm is the same as S100 in the first embodiment, and will not be described here to avoid redundancy.

[0048] S220: generating audio based on the text embedding after semantic enhancement processing, and using an audio mask modeling algorithm to randomly mask part of the audio signal in the audio generation process to generate a masked audio signal;

[0049] In this step, the audio mask modeling algorithm is the same as S110 in the first embodiment, and will not be described here to avoid redundancy.

[0050] S230: predicting the audio signal not covered by the mask according to the enhanced text embedding and the mask audio signal, to generate an audio signal conforming to deep semantic features and having natural emotion;

[0051] In this step, the audio signal generation process is the same as S120 in the first embodiment, and to avoid redundancy, it will not be repeated here.

[0052] S240: post-processing the generated audio signal to generate an optimized audio signal;

[0053] In this step, the post-processing of the audio signal includes but is not limited to removing noise, adjusting pitch and rhythm, etc., so that the finally generated audio signal is clearer and more natural, and more consistent with the text to be processed in terms of semantics and style.

[0054] It can be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0055] In an embodiment, a single-stage speech synthesis device is provided, which corresponds to the single-stage speech synthesis method in the above embodiment. As shown in the figure, the single-stage speech synthesis device includes a preprocessing module 101, a semantic knowledge distillation module 102, an audio mask modeling module 103, an audio generation module 104, and a post-processing module 105. The functions of each module are described in detail as follows: Figure 5

[0056] Preprocessing module 101: used to receive the input text to be processed by the single-stage TTS model, and pre-process the text to be processed, and convert the text to be processed into a fixed-size text embedding; wherein the TTS model adopts a single-stage architecture (single-stage), which can directly generate audio from text, avoiding the multi-stage processing process of the existing multi-stage TTS model, not only reducing the complexity of the model, but also simplifying the audio generation process, improving the inference speed, so that the model can generate high-quality audio in real-time scenarios.

[0057] In the embodiment of the present application, the preprocessing of the text to be processed by the preprocessing module 101 includes word segmentation, character encoding and text embedding (Text Embedding), which represents words, phrases or sentences in the text to be processed as fixed-length text vectors.

[0058] ​The semantic knowledge distillation module 102 is configured to perform semantic enhancement processing on the text embedding by using a semantic knowledge distillation algorithm, so as to capture deep semantic features in the text to be processed. The semantic knowledge distillation is a technology for migrating knowledge in a complex model to a smaller model. In the embodiment of the present application, the semantic knowledge distillation algorithm is used to perform semantic enhancement processing on the text to be processed, so as to capture deep semantic features of the text to be processed, and improve the semantic analysis capability of the TTS model for the text to be processed.

[0059] Specifically, in the embodiment of the present application, the semantic knowledge distillation algorithm uses a pre-trained language model such as BERT (Bidirectional Encoder Representations from Transformers) or GPT (Generative Pre-Trained Transformer) as a teacher model to capture deep semantic features such as sentiment, syntactic structure, and semantic stress in the text to be processed. Then, the captured deep semantic features are distilled into the TTS model. The deep semantic features captured by the teacher model guide the TTS model (i.e., a student model) to generate an audio signal, and the TTS model can more accurately understand deep semantic information such as sentiment, syntactic structure, and semantic stress in the text to be processed when generating the audio signal, thereby generating an audio signal that is more consistent with the content of the text and improving the naturalness of the audio generation. Taking the BERT model as an example, the BERT can better understand the semantics of each word in a sentence by using the encoder structure of the Transformer to learn context information from both left and right directions of the text, thereby avoiding the limitations of traditional one-way language models.

[0060] It can be understood that, by introducing the semantic knowledge distillation algorithm, the TTS model can better capture deep semantic information in the input text when generating an audio signal, and the generated audio signal can not only be highly consistent with the input text in content, but also can show detailed features such as emotional color and semantic stress in the input text, thereby enhancing the semantic understanding and generation consistency of the TTS model.

[0061] Audio Token Modeling Module 103: This module generates audio based on the semantically enhanced text embedding. During the audio generation process, it randomly masks part of the audio signal using an audio token modeling algorithm to generate a masked audio signal. Audio token modeling is a model training strategy in natural language processing. The algorithm is based on masked language modeling in natural language processing. By randomly masking the input audio signal, the masked audio signal is inferred from the text and the unmasked audio signal during training. This allows the model to better learn the global features of the audio, enhance its understanding and generation capabilities of the overall text information, and generate more natural audio signals.

[0062] Specifically, such as Figure 3 The diagram illustrates the training process of the audio masking modeling algorithm in this embodiment of the invention. First, a semantic encoder understands the meaning of the input audio and converts it into a readable form. Then, a speech discrete cosine transform (DCE) converts the audio signal into a discrete-time spectrum, which is encoded by calculating the discrete cosine transform. Next, a speech discrete token converts the input audio signal into a series of discrete tokens. These discrete tokens are then randomly masked, and the masked discrete tokens and the embedded text are used as input to the Transformer model.

[0063] It is understood that by introducing an audio mask modeling algorithm, this invention enables the model to process complex phonological, rhythmic, and pause information during the audio generation process, and increases the diversity of audio generation, especially when dealing with long sentences and complex syntax, thereby improving the naturalness and diversity of audio generation.

[0064] Audio generation module 104: It is used to predict the unmasked audio signal based on the semantically enhanced text embedding and masked audio signal, and generate an audio signal that conforms to deep semantic features and has natural emotion; wherein, the audio generation module 104 combines the semantic knowledge distillation results and audio mask modeling, so that the TTS model can directly generate an audio signal that conforms to semantics and has natural speech features and emotional expression from the text in a single stage.

[0065] It can be understood that the present application combines semantic knowledge distillation results and audio mask modeling, so that the model does not need to rely on a large amount of labeled data in the audio generation process, reduces the demand for labeled speech data, and still maintains excellent generation performance in a limited data scenario.

[0066] The post-processing module 105 is configured to post-process the generated audio signal to generate an optimized audio signal. The post-processing of the audio signal includes, but is not limited to, removing noise, adjusting pitch and rhythm, etc., so that the finally generated audio signal is clearer and more natural, and is more consistent with the text to be processed in terms of semantics and style.

[0067] The single-stage speech synthesis device provided by the present application adopts a single-stage TTS model architecture, uses a semantic knowledge distillation algorithm to perform semantic enhancement processing on text, to capture deep semantic features of the text and improve the semantic analysis capability of the single-stage TTS model on the text. In the audio generation process, an audio mask modeling algorithm is used to randomly mask the audio signal. The semantic knowledge distillation algorithm and the audio mask modeling algorithm are combined, so that the TTS model can directly generate an audio signal that meets the semantics and has natural speech characteristics and emotional performance from the text in a single stage. The present application directly generates audio output from text input, not only reduces the complexity of the model and simplifies the generation process, but also improves the inference speed, so that the model can generate high-quality audio in a real-time scenario.

[0068] The specific limitations of the single-stage speech synthesis device can be referred to the limitations of the single-stage speech synthesis method in the above, which will not be repeated here. Each module in the single-stage speech synthesis device described above can be realized by software, hardware and their combination in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0069] In one embodiment, a computer device is provided, which can be a server, and its internal structure diagram can be as shown in Figure 6 The computer device includes a processor, a memory, a network interface and a database connected by a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external client through the network connection. The computer program is executed by the processor to implement the functions or steps of a single-stage speech synthesis method server side.

[0070] In one embodiment, a computer device is provided, which can be a client, and an internal structure diagram thereof can be as shown in Figure 7 The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with an external server through a network connection. The computer program is executed by the processor to implement the functions or steps of a single-stage speech synthesis method on the client side.

[0071] In one embodiment, a computer device is provided, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0072] The input text to be processed is received by a single-stage TTS model, and the semantic enhancement processing is performed on the text to be processed by a semantic knowledge distillation algorithm to capture deep semantic features in the text to be processed;

[0073] According to the text to be processed after the semantic enhancement processing, audio generation is performed, and in the audio generation process, part of the audio signal is randomly masked using an audio mask modeling algorithm to generate a masked audio signal;

[0074] According to the text to be processed after the semantic enhancement processing and the masked audio signal, the audio signal that is not masked is predicted to generate an audio signal that conforms to the deep semantic features and has natural emotions.

[0075] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0076] The input text to be processed is received by a single-stage TTS model, and the semantic enhancement processing is performed on the text to be processed by a semantic knowledge distillation algorithm to capture deep semantic features in the text to be processed;

[0077] According to the text to be processed after the semantic enhancement processing, audio generation is performed, and in the audio generation process, part of the audio signal is randomly masked using an audio mask modeling algorithm to generate a masked audio signal;

[0078] According to the text to be processed after the semantic enhancement processing and the masked audio signal, the audio signal that is not masked is predicted to generate an audio signal that conforms to the deep semantic features and has natural emotions.

[0079] It should be noted that the above functions or steps that can be implemented by the computer readable storage medium or the computer device can correspond to the above-mentioned related descriptions of the server side and the client side in the method embodiments, and to avoid repetition, they will not be described one by one here.

[0080] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of the method. Any reference to memory, storage, database or other storage medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0081] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified. In actual application, the above-mentioned functions can be completed by different functional units or modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0082] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A single stage speech synthesis method characterized by, The method comprises the following steps: receiving input text to be processed by a single-stage TTS model, and performing semantic enhancement processing on the text to be processed by a semantic knowledge distillation algorithm to capture deep semantic features in the text to be processed; generating audio according to the text to be processed after semantic enhancement processing, and randomly masking part of the audio signal during audio generation using an audio mask modeling algorithm to generate a masked audio signal; predicting the audio signal that has not been masked according to the text to be processed after semantic enhancement processing and the masked audio signal, and generating an audio signal that conforms to the deep semantic features and has natural emotions.

2. The single-stage speech synthesis method of claim 1, wherein, The semantic enhancement processing of the text to be processed by the semantic knowledge distillation algorithm is specifically: The semantic knowledge distillation algorithm uses a pre-trained language model as a teacher model to capture deep semantic features in the text to be processed and distill the deep semantic features into a single-stage TTS model; wherein the deep semantic features include emotions, grammatical structures, and semantic stress.

3. The single-stage speech synthesis method of claim 2, wherein, The audio generation according to the text to be processed after semantic enhancement processing and the random masking of part of the audio signal during audio generation using the audio mask modeling algorithm are specifically: The audio mask modeling algorithm understands the meaning of the audio signal through a semantic encoder and converts it into a readable form, converts the audio signal into a discrete time spectrum through a speech discrete encoder, encodes it through a discrete cosine transform, and converts the audio signal into a discrete token through a speech discrete token, and then randomly masks the discrete token.

4. The single-stage speech synthesis method according to any one of claims 1 to 3, wherein, Before the semantic enhancement processing of the text to be processed by the semantic knowledge distillation algorithm, the method further comprises: preprocessing the text to be processed to convert the text to be processed into a fixed-size text embedding.

5. The single-stage speech synthesis method of claim 4, wherein, The preprocessing of the text to be processed comprises: segmenting, character encoding, and text embedding the text to be processed to represent words, phrases, or sentences in the text to be processed as fixed-length text vectors.

6. The single-stage speech synthesis method of claim 4, wherein, After the prediction of the audio signal that has not been masked according to the text to be processed after semantic enhancement processing and the masked audio signal, and the generation of an audio signal that conforms to the deep semantic features and has natural emotions, the method further comprises: post-processing the audio signal to generate an optimized audio signal.

7. The single-stage speech synthesis method of claim 6, wherein, The post-processing of the audio signal is specifically: noise removal, pitch adjustment, and rhythm processing of the audio signal.

8. A single stage speech synthesis apparatus characterized by comprising: The method comprises: a semantic knowledge distillation module for performing semantic enhancement processing on input text to be processed by a semantic knowledge distillation algorithm to capture deep semantic features in the text to be processed; an audio mask modeling module for generating audio according to the text to be processed after semantic enhancement processing, and randomly masking part of the audio signal during audio generation using an audio mask modeling algorithm to generate a masked audio signal; an audio generation module for predicting the audio signal that has not been masked according to the text to be processed after semantic enhancement processing and the masked audio signal, and generating an audio signal that conforms to the deep semantic features and has natural emotions.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the steps of the single-stage speech synthesis method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the single-stage speech synthesis method according to any one of claims 1 to 7.

Images