Single-stage speech synthesis method and device, equipment and storage medium

Abstract

The invention relates to the technical field of data storage, and discloses a single-stage speech synthesis method, device and equipment and a storage medium, and the method comprises the steps: receiving an input to-be-processed text through a single-stage TTS model, carrying out the semantic enhancement processing of the to-be-processed text through a semantic knowledge distillation algorithm, and capturing the deep semantic features in the to-be-processed text; performing audio generation according to the to-be-processed text subjected to semantic enhancement processing, and randomly covering a part of audio signals by using an audio mask modeling algorithm in the audio generation process to generate mask audio signals; and according to the to-be-processed text after semantic enhancement processing and the mask audio signal, predicting an unmasked audio signal, and generating an audio signal which accords with the deep semantic features and has natural emotion. According to the method, a single-stage TTS model architecture is adopted, a semantic knowledge distillation algorithm and an audio mask modeling algorithm are combined, the generation process is simplified, the reasoning speed is increased, and high-quality audio can be generated in a real-time scene.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a single-stage speech synthesis method, device, equipment and storage medium. Background Art

[0002] Speech synthesis, also known as text-to-speech (TTS) technology, can convert any text information into standard and fluent speech in real time, and is a cutting-edge technology in the field of Chinese information processing. At present, TTS technology has shown great potential in many application fields such as voice assistants, content creation, and voice navigation. Most existing TTS technologies adopt a two-stage or multi-stage architecture, that is, first generate acoustic features (such as Mel-spectrograms), and then use a neural vocoder to convert the acoustic features into audio. The inventors found that although this multi-stage processing process improves the quality of speech generation, it still has the following deficiencies in terms of model complexity, reasoning efficiency, generation naturalness, and diversity:

[0003] 1) Existing TTS systems use a series connection of multiple models (i.e., a multi-stage architecture), which introduces additional complexity and increases training and inference time. Especially in scenarios where real-time speech generation is required, the efficiency of speech generation is significantly limited.

[0004] 2) Since each stage in the multi-stage model needs to be trained independently, the consistency of the generated speech may be affected, especially the coherence of phonology, sound quality and emotion. Error propagation between stages may also lead to a decline in speech quality.

[0005] 3) Existing TTS models usually rely only on text input for speech generation, ignoring the deep understanding of text semantics, resulting in the model being unable to generate natural speech that conforms to the semantic context when processing complex sentence structures or meanings.

[0006] 4) Existing TTS systems rely on a large amount of accurately annotated speech data for training, especially speech data related to specific styles, emotions or intonations, which makes TTS technology perform poorly in scenarios with limited data. In addition, the high cost of obtaining high-quality annotated data limits the widespread deployment of TTS systems. Summary of the invention

[0007] The present invention provides a single-stage speech synthesis method, apparatus, device and storage medium to solve the technical problems existing in the existing TTS technology, such as complex model architecture, impact on speech consistency, inability to generate natural speech that conforms to the semantic context, and reliance on annotated speech data for model training.

[0008] In a first aspect, a single-stage speech synthesis method is provided, comprising:

[0009] Receiving an input text to be processed through a single-stage TTS model, and performing semantic enhancement processing on the text to be processed through a semantic knowledge distillation algorithm to capture deep semantic features in the text to be processed;

[0010] Generating audio according to the text to be processed after the semantic enhancement processing, and in the audio generation process, randomly masking part of the audio signal using an audio mask modeling algorithm to generate a masked audio signal;

[0011] The unmasked audio signal is predicted based on the text to be processed after the semantic enhancement processing and the masked audio signal, so as to generate an audio signal that conforms to the deep semantic features and has natural emotions.

[0012] In a second aspect, a single-stage speech synthesis device is provided, comprising:

[0013] Semantic knowledge distillation module: used to perform semantic enhancement processing on the input text to be processed through the semantic knowledge distillation algorithm to capture the deep semantic features in the text to be processed;

[0014] An audio mask modeling module is used to generate audio according to the text to be processed after the semantic enhancement processing, and in the audio generation process, randomly mask part of the audio signal using the audio mask modeling algorithm to generate a masked audio signal;

[0015] Audio generation module: used to predict the unmasked audio signal based on the text to be processed and the masked audio signal after the semantic enhancement processing, and generate an audio signal that conforms to the deep semantic features and has natural emotions.

[0016] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned single-stage speech synthesis method when executing the computer program.

[0017] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned single-stage speech synthesis method are implemented.

[0018] In the scheme implemented by the above-mentioned single-stage speech synthesis method, device, computer equipment and storage medium, a single-stage TTS model architecture is adopted, and a semantic knowledge distillation algorithm is used to perform semantic enhancement processing on the text to capture the deep semantic features of the text and improve the semantic parsing ability of the single-stage TTS model for the text; in the audio generation process, an audio mask modeling algorithm is used to randomly mask the audio signal, and the semantic knowledge distillation algorithm and the audio mask modeling algorithm are combined, so that the TTS model can directly generate an audio signal that is semantically consistent and has natural speech features and emotional expressions from the text in a single stage. The embodiment of the present invention directly generates audio output from text input, which not only reduces the complexity of the model and simplifies the generation process, but also improves the reasoning speed, so that the model can generate high-quality audio in real-time scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0020] Figure 1 is an architectural diagram of a single-stage speech synthesis system in one embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of a flow chart of a single-stage speech synthesis method in a first embodiment of the present invention;

[0022] Figure 3 A schematic diagram of a training process of an audio mask modeling algorithm according to an embodiment of the present invention;

[0023] Figure 4 is a flow chart of a single-stage speech synthesis method in a second embodiment of the present invention;

[0024] Figure 5 is a schematic diagram of the structure of a single-stage speech synthesis device in one embodiment of the present invention;

[0025] Figure 6 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0026] Figure 7 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The single-stage speech synthesis method provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the client communicates with the server through a network. The server can receive the input text to be processed, receive the input text to be processed through a single-stage TTS model, and perform semantic enhancement processing on the text to be processed through a semantic knowledge distillation algorithm to capture the deep semantic features in the text to be processed; generate audio according to the text to be processed after the semantic enhancement processing, and in the audio generation process, use an audio mask modeling algorithm to randomly mask part of the audio signal to generate a masked audio signal; predict the unmasked audio signal according to the text to be processed after the semantic enhancement processing and the masked audio signal, generate an audio signal that conforms to the deep semantic features and has natural emotions, and return the audio signal to the client. The present invention is applicable to speech synthesis in the financial field. For speech synthesis in the financial field, the TTS model can generate an audio signal that conforms to the semantics and has natural speech features and emotional expressions directly from the text in a single stage by combining the semantic knowledge distillation results and the audio mask modeling, so as to simplify the audio generation process and improve the quality and consistency of audio generation. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server side can be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0029] See also Figure 2 , is a flow chart of a single-stage speech synthesis method in the first embodiment of the present invention. The single-stage speech synthesis method provided by the first embodiment of the present invention comprises the following steps:

[0030] S100: receiving an input text to be processed, and performing semantic enhancement processing on the text to be processed by using a semantic knowledge distillation algorithm to capture deep semantic features in the text to be processed;

[0031] In this step, semantic knowledge distillation is a technology that transfers knowledge from a complex model to a smaller model. In the embodiment of the present invention, the semantic knowledge distillation algorithm is used to perform semantic enhancement processing on the text to be processed to capture the deep semantic features of the text to be processed and improve the semantic parsing ability of the TTS model for the text to be processed.

[0032] Specifically, in an embodiment of the present invention, the semantic knowledge distillation algorithm uses a pre-trained language model such as BERT (Bidirectional Encoder Representations from Transformers) or GPT (Generative Pre-Trained Transformer) as a teacher model to capture deep semantic features such as sentiment, grammatical structure, and semantic stress in the text to be processed. Then, the captured deep semantic features are distilled into the TTS model, and the deep semantic features captured by the teacher model guide the TTS model (i.e., the student model) to generate an audio signal, and when generating an audio signal, it can more accurately understand the deep semantic information such as sentiment, syntactic structure, and semantic stress in the text to be processed, thereby generating an audio signal that is more in line with the text content and improving the naturalness of audio generation. Among them, taking the BERT model as an example, BERT can better understand the semantics of each word in a sentence by using the encoder structure of the Transformer and learning contextual information from both the left and right directions of the text, thereby avoiding the limitations of traditional unidirectional language models.

[0033] It can be understood that the present invention introduces a semantic knowledge distillation algorithm, so that the TTS model can better capture the deep semantic information in the input text when generating audio signals. The generated audio signal can not only be highly consistent with the input text in content, but also can express detailed features such as emotional color and semantic stress in the input text, thereby enhancing the semantic understanding and generation consistency of the TTS model.

[0034] S110: generating audio according to the text to be processed after the semantic enhancement processing, and in the audio generation process, randomly masking part of the audio signal by using an audio mask modeling algorithm to generate a masked audio signal;

[0035] In this step, Masked Audio Token Modeling is a model training strategy in natural language processing. The algorithm is based on the Masked Language Modeling in natural language processing. It randomly masks the input audio signal. During the training process, the masked audio signal is inferred based on the text and the unmasked audio signal, so that the model can better learn the global features of the audio, enhance the understanding and generation capabilities of the overall text information, and generate more natural audio signals.

[0036] Specifically, Figure 3 As shown, it is a schematic diagram of the training process of the audio mask modeling algorithm in the embodiment of the present invention. First, the semantic encoder understands the meaning of the input audio and converts it into a readable form, and the audio signal is converted into a discrete time spectrum by the speech discrete encoder (Speech Discrete Cosine Transform Encoder, DCE), and encoded by calculating the discrete cosine transform, and then the input audio signal is converted into a series of discrete tokens (tokens) by the speech discrete token (SpeechDiscrete Token), and the discrete tokens are randomly masked, and the masked discrete tokens and text embedding are used as the input of the Transformer model.

[0037] It can be understood that the present invention introduces the audio mask modeling algorithm, which enables the model to process complex information such as phonetics, rhythms and pauses during the audio generation process, and increases the diversity of audio generation, especially when processing long sentences and complex syntax, thereby improving the naturalness and diversity of audio generation.

[0038] S120: predicting an unmasked audio signal based on the text to be processed and the masked audio signal after the semantic enhancement processing, and generating an audio signal that conforms to the deep semantic features and has natural emotions;

[0039] In this step, by combining the results of semantic knowledge distillation and audio mask modeling, the TTS model can generate audio signals that are semantically consistent and have natural speech features and emotional expressions directly from text in a single stage.

[0040] It can be understood that the present invention combines the results of semantic knowledge distillation with audio mask modeling, so that the model does not need to rely on a large amount of labeled data during the audio generation process, reducing the demand for labeled speech data, thereby maintaining excellent generation performance in scenarios with limited data.

[0041] It can be seen that in the above scheme, the single-stage speech synthesis method provided by the embodiment of the present invention adopts a single-stage TTS model architecture, and uses a semantic knowledge distillation algorithm to perform semantic enhancement processing on the text to capture the deep semantic features of the text and improve the semantic parsing ability of the single-stage TTS model for the text; in the audio generation process, the audio signal is randomly masked using an audio mask modeling algorithm, and the semantic knowledge distillation algorithm and the audio mask modeling algorithm are combined, so that the TTS model can directly generate an audio signal that is semantically consistent and has natural speech features and emotional expressions from the text in a single stage. The embodiment of the present invention directly generates audio output from text input, which not only reduces the complexity of the model and simplifies the generation process, but also improves the reasoning speed, so that the model can generate high-quality audio in real-time scenarios.

[0042] See also Figure 4 , is a flow chart of a single-stage speech synthesis method in the second embodiment of the present invention. The single-stage speech synthesis method provided by the second embodiment of the present invention comprises the following steps:

[0043] S200: receiving input text to be processed through a single-stage TTS model, preprocessing the text to be processed, and converting the text to be processed into a text embedding of a fixed size;

[0044] In this step, the TTS model adopts a single-stage architecture and can generate audio directly from text, avoiding the multi-stage processing process of the existing multi-stage TTS model. This not only reduces the complexity of the model, but also simplifies the audio generation process and improves the inference speed, enabling the model to generate high-quality audio in real-time scenarios.

[0045] In the embodiment of the present invention, the preprocessing of the text to be processed includes word segmentation, character encoding and text embedding, and the words, phrases or sentences in the text to be processed are represented as text vectors of fixed length.

[0046] S210: Perform semantic enhancement on the converted text embedding through the semantic knowledge distillation algorithm to capture the deep semantic features in the text to be processed;

[0047] In this step, the semantic knowledge distillation algorithm is the same as S100 in the first embodiment, and will not be described again here to avoid redundancy.

[0048] S220: generating audio based on the text embedding after semantic enhancement processing, and in the audio generation process, randomly masking part of the audio signal using an audio mask modeling algorithm to generate a masked audio signal;

[0049] In this step, the audio mask modeling algorithm is the same as S110 in the first embodiment, and will not be described again here to avoid redundancy.

[0050] S230: predicting an unmasked audio signal according to the enhanced text embedding and the masked audio signal, and generating an audio signal that conforms to deep semantic features and has natural emotions;

[0051] In this step, the audio signal generation process is the same as S120 in the first embodiment, and will not be described again here to avoid redundancy.

[0052] S240: performing post-processing on the generated audio signal to generate an optimized audio signal;

[0053] In this step, post-processing of the audio signal includes but is not limited to removing noise, adjusting pitch and rhythm, etc., so that the final generated audio signal is clearer and more natural, and more consistent with the text to be processed in terms of semantics and style.

[0054] It can be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0055] In one embodiment, a single-stage speech synthesis device is provided, which corresponds one-to-one to the single-stage speech synthesis method in the above embodiment. Figure 5 As shown, the single-stage speech synthesis device includes a pre-processing module 101, a semantic knowledge distillation module 102, an audio mask modeling module 103, an audio generation module 104 and a post-processing module 105. The functional modules are described in detail as follows:

[0056] Preprocessing module 101: used to receive input text to be processed through a single-stage TTS model, preprocess the text to be processed, and convert the text to be processed into a fixed-size text embedding; wherein the TTS model adopts a single-stage architecture (single-stage), which can directly generate audio from text, avoiding the multi-stage processing process of the existing multi-stage TTS model, which not only reduces the complexity of the model, but also simplifies the audio generation process and improves the reasoning speed, so that the model can generate high-quality audio in real-time scenarios.

[0057] In the embodiment of the present invention, the preprocessing module 101 performs preprocessing on the text to be processed, including word segmentation, character encoding and text embedding, and represents the words, phrases or sentences in the text to be processed as text vectors of fixed length.

[0058] Semantic knowledge distillation module 102: used to perform semantic enhancement processing on text embeddings through a semantic knowledge distillation algorithm to capture deep semantic features in the text to be processed; wherein, semantic knowledge distillation (Semantic Knowledge Distillation) is a technology that transfers knowledge from a complex model to a smaller model. In an embodiment of the present invention, a semantic knowledge distillation algorithm is used to perform semantic enhancement processing on the text to be processed to capture deep semantic features of the text to be processed, thereby improving the semantic parsing capability of the TTS model for the text to be processed.

[0059] Specifically, in an embodiment of the present invention, the semantic knowledge distillation algorithm uses a pre-trained language model such as BERT (Bidirectional Encoder Representations from Transformers) or GPT (Generative Pre-Trained Transformer) as a teacher model to capture deep semantic features such as sentiment, grammatical structure, and semantic stress in the text to be processed. Then, the captured deep semantic features are distilled into the TTS model, and the deep semantic features captured by the teacher model guide the TTS model (i.e., the student model) to generate an audio signal, and when generating an audio signal, it can more accurately understand the deep semantic information such as sentiment, syntactic structure, and semantic stress in the text to be processed, thereby generating an audio signal that is more in line with the text content and improving the naturalness of audio generation. Among them, taking the BERT model as an example, BERT can better understand the semantics of each word in a sentence by using the encoder structure of the Transformer and learning contextual information from both the left and right directions of the text, thereby avoiding the limitations of traditional unidirectional language models.

[0060] It can be understood that the present invention introduces a semantic knowledge distillation algorithm, so that the TTS model can better capture the deep semantic information in the input text when generating audio signals. The generated audio signal can not only be highly consistent with the input text in content, but also can express detailed features such as emotional color and semantic stress in the input text, thereby enhancing the semantic understanding and generation consistency of the TTS model.

[0061] Audio mask modeling module 103: used to generate audio according to the text embedding after semantic enhancement processing, and in the audio generation process, use the audio mask modeling algorithm to randomly mask part of the audio signal to generate a masked audio signal; wherein, audio mask modeling (Masked Audio Token Modeling) is a model training strategy in natural language processing. The algorithm is based on the masked language model (Masked Language Modeling) in natural language processing. By randomly masking the input audio signal, during the training process, the masked audio signal is inferred based on the text and the unmasked audio signal, so that the model can better learn the global characteristics of the audio, enhance the understanding and generation capabilities of the overall text information, and generate a more natural audio signal.

[0062] Specifically, Figure 3 As shown, it is a schematic diagram of the training process of the audio mask modeling algorithm in the embodiment of the present invention. First, the semantic encoder understands the meaning of the input audio and converts it into a readable form, and the audio signal is converted into a discrete time spectrum by the speech discrete encoder (Speech Discrete Cosine Transform Encoder, DCE), and encoded by calculating the discrete cosine transform, and then the input audio signal is converted into a series of discrete tokens (tokens) by the speech discrete token (SpeechDiscrete Token), and the discrete tokens are randomly masked, and the masked discrete tokens and text embedding are used as the input of the Transformer model.

[0063] It can be understood that the present invention introduces the audio mask modeling algorithm, which enables the model to process complex information such as phonetics, rhythms and pauses during the audio generation process, and increases the diversity of audio generation, especially when processing long sentences and complex syntax, thereby improving the naturalness and diversity of audio generation.

[0064] Audio generation module 104: used to predict the unmasked audio signal based on the text embedding and masked audio signal after semantic enhancement processing, and generate an audio signal that conforms to deep semantic features and has natural emotions; wherein, the audio generation module 104 combines the semantic knowledge distillation results and audio mask modeling, so that the TTS model can directly generate an audio signal that conforms to semantics and has natural speech features and emotional expressions from the text in a single stage.

[0065] It can be understood that the present invention combines the results of semantic knowledge distillation with audio mask modeling, so that the model does not need to rely on a large amount of labeled data during the audio generation process, reducing the demand for labeled speech data, thereby maintaining excellent generation performance in scenarios with limited data.

[0066] Post-processing module 105: used to post-process the generated audio signal to generate an optimized audio signal; wherein the post-processing of the audio signal includes but is not limited to removing noise, adjusting pitch and rhythm, etc., so that the final generated audio signal is clearer and more natural, and more consistent with the text to be processed in terms of semantics and style.

[0067] The single-stage speech synthesis device provided by the present invention adopts a single-stage TTS model architecture, and uses a semantic knowledge distillation algorithm to perform semantic enhancement processing on the text to capture the deep semantic features of the text and improve the semantic parsing ability of the single-stage TTS model for the text; in the audio generation process, the audio signal is randomly masked using an audio mask modeling algorithm, and the semantic knowledge distillation algorithm and the audio mask modeling algorithm are combined, so that the TTS model can directly generate an audio signal that is semantically consistent and has natural voice features and emotional expressions from the text in a single stage. The embodiment of the present invention directly generates audio output from text input, which not only reduces the complexity of the model and simplifies the generation process, but also improves the reasoning speed, so that the model can generate high-quality audio in real-time scenarios.

[0068] For the specific definition of the single-stage speech synthesis device, please refer to the definition of the single-stage speech synthesis method above, which will not be repeated here. Each module in the above-mentioned single-stage speech synthesis device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0069] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a single-stage speech synthesis method.

[0070] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it implements the functions or steps of the client side of a single-stage speech synthesis method.

[0071] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0072] Receiving an input text to be processed through a single-stage TTS model, and performing semantic enhancement processing on the text to be processed through a semantic knowledge distillation algorithm to capture deep semantic features in the text to be processed;

[0073] Generating audio according to the text to be processed after the semantic enhancement processing, and in the audio generation process, randomly masking part of the audio signal using an audio mask modeling algorithm to generate a masked audio signal;

[0074] The unmasked audio signal is predicted based on the text to be processed after the semantic enhancement processing and the masked audio signal, so as to generate an audio signal that conforms to the deep semantic features and has natural emotions.

[0075] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0076] Receiving an input text to be processed through a single-stage TTS model, and performing semantic enhancement processing on the text to be processed through a semantic knowledge distillation algorithm to capture deep semantic features in the text to be processed;

[0077] Generating audio according to the text to be processed after the semantic enhancement processing, and in the audio generation process, randomly masking part of the audio signal using an audio mask modeling algorithm to generate a masked audio signal;

[0078] The unmasked audio signal is predicted based on the text to be processed after the semantic enhancement processing and the masked audio signal, so as to generate an audio signal that conforms to the deep semantic features and has natural emotions.

[0079] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0080] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other storage medium used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0081] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0082] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A single-stage speech synthesis method, characterized in that: include: Receiving an input text to be processed through a single-stage TTS model, and performing semantic enhancement processing on the text to be processed through a semantic knowledge distillation algorithm to capture deep semantic features in the text to be processed; Generating audio according to the text to be processed after the semantic enhancement processing, and in the audio generation process, randomly masking part of the audio signal using an audio mask modeling algorithm to generate a masked audio signal; The unmasked audio signal is predicted based on the text to be processed after the semantic enhancement processing and the masked audio signal, so as to generate an audio signal that conforms to the deep semantic features and has natural emotions.

2. The single-stage speech synthesis method according to claim 1, characterized in that: The semantic enhancement processing of the to-be-processed text by using the semantic knowledge distillation algorithm is specifically as follows: The semantic knowledge distillation algorithm uses a pre-trained language model as a teacher model to capture deep semantic features in the text to be processed and distills the deep semantic features into a single-stage TTS model; wherein the deep semantic features include sentiment, grammatical structure and semantic stress.

3. The single-stage speech synthesis method according to claim 2, characterized in that: The audio is generated according to the text to be processed after the semantic enhancement processing, and in the audio generation process, a part of the audio signal is randomly masked by using an audio mask modeling algorithm, specifically: The audio mask modeling algorithm understands the meaning of the audio signal through a semantic encoder and converts it into a readable form, converts the audio signal into a discrete time spectrum through a speech discrete encoder, encodes it by calculating a discrete cosine transform, and converts the audio signal into discrete tokens through speech discrete tokens, and then randomly masks the discrete tokens.

4. The single-stage speech synthesis method according to any one of claims 1 to 3, characterized in that: Before the semantic enhancement processing is performed on the text to be processed by the semantic knowledge distillation algorithm, the method further includes: The text to be processed is preprocessed to convert the text to be processed into a text embedding of a fixed size.

5. The single-stage speech synthesis method according to claim 4, characterized in that: The preprocessing of the to-be-processed text comprises: The text to be processed is segmented, character encoded and embedded, and the words, phrases or sentences in the text to be processed are represented as text vectors of fixed length.

6. The single-stage speech synthesis method according to claim 4, characterized in that: After predicting the unmasked audio signal according to the to-be-processed text after the semantic enhancement processing and the masked audio signal to generate an audio signal that conforms to the deep semantic features and has natural emotions, the method further includes: The audio signal is post-processed to generate an optimized audio signal.

7. The single-stage speech synthesis method according to claim 6, characterized in that: The post-processing of the audio signal is specifically: The audio signal is processed by removing noise, adjusting pitch and rhythm.

8. A single-stage speech synthesis device, characterized in that: include: Semantic knowledge distillation module: used to perform semantic enhancement processing on the input text to be processed through the semantic knowledge distillation algorithm to capture the deep semantic features in the text to be processed; An audio mask modeling module is used to generate audio according to the text to be processed after the semantic enhancement processing, and in the audio generation process, randomly mask part of the audio signal using the audio mask modeling algorithm to generate a masked audio signal; Audio generation module: used to predict the unmasked audio signal based on the text to be processed and the masked audio signal after the semantic enhancement processing, and generate an audio signal that conforms to the deep semantic features and has natural emotions.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the single-stage speech synthesis method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the single-stage speech synthesis method according to any one of claims 1 to 7 are implemented.

Images