A speech synthesis method, device, equipment and storage medium thereof

Abstract

The embodiment of the present application belongs to the field of speech processing technology and is applied to text-to-speech scenarios. It relates to a speech synthesis method, apparatus, device and storage medium thereof. The method obtains a synthesized reference speech and text data to be synthesized; performs text encoding and decoding on the text data to obtain a target text character sequence; inputs the target text character sequence into a trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; acoustically decodes the target text character sequence based on the phoneme features and pronunciation start and end time positions to obtain a target output speech. By adopting a model prediction method, the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence are predicted, and each text character is decoded one by one to synthesize the target output speech, thereby achieving fast and high-quality speech synthesis output.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and is applied to text-to-speech scenarios, and in particular to a speech synthesis method, apparatus, device, and storage medium thereof. Background Art

[0002] Speech synthesis technology involves synthesizing the pronunciation of a target speaker from a given text. Currently, speech synthesis technology has been rapidly developing, with end-to-end models and large speech models significantly driving progress in this field.

[0003] However, end-to-end speech synthesis in practical applications faces the following challenges: the model is too large, conventional text encoding uses a method of encoding one sentence at a time based on sequence position, resulting in excessive computational complexity, and traditional decoding methods often decode an entire text sentence at once, resulting in inaccurate decoding and reduced speech synthesis quality. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to propose a speech synthesis method, apparatus, device and storage medium thereof to solve the problem that in existing speech synthesis, the encoding calculation is too much and the decoding is not accurate enough, which reduces the quality of speech synthesis.

[0005] In order to solve the above technical problems, the embodiments of the present application provide a speech synthesis method, which adopts the following technical solutions:

[0006] A speech synthesis method comprises the following steps:

[0007] Obtaining a synthesized reference speech and text data to be synthesized;

[0008] Inputting the text data into a preset text processor and performing text encoding and decoding processing to obtain a target text character sequence, wherein the preset text processor is composed of an N-layer transformer structure, and the position encoding mode of the text processor is set to a rotation position encoding mode, where N is a positive integer greater than 1;

[0009] Using the synthesized reference speech to train a preset phoneme prediction model to obtain a trained phoneme prediction model;

[0010] Inputting the target text character sequence into the trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence;

[0011] The target text character sequence is acoustically decoded according to the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence to obtain the target output speech.

[0012] Furthermore, the step of obtaining the synthesized reference speech and the text data to be synthesized specifically includes:

[0013] Filtering the target object's voice package from the reference voice package provided by the target speech synthesis end by screening;

[0014] Extracting batches of speech segments from the speech package of the target object as the synthesized reference speech;

[0015] The text data sent or imported by the speech synthesis request end is obtained as the text data to be speech synthesized.

[0016] Furthermore, the step of inputting the text data into a preset text processor, performing text encoding and decoding processing, and obtaining a target text character sequence specifically includes:

[0017] Utilizing the encoder layer in the transformer structure included in the text processor to convert all text in the text data into vector representations of target dimensions;

[0018] Using a rotational position encoding method to add position information to the vector representations corresponding to all the texts to obtain vector representations with embedded position information;

[0019] Utilizing the multi-head attention mechanism included in the encoder layer, capturing the long-term dependencies and semantic relationships between all texts in the text data from the vector representation of the embedded position information;

[0020] In combination with the long-term dependencies and semantic relationships between all texts in the text data, the decoder layer in the transformer structure included in the text processor is used to decode the vector representation of the embedded position information to obtain the target text character sequence.

[0021] Furthermore, the phoneme prediction model includes a speech-to-text component, an acoustic feature extraction component, and a Viterbi algorithm component. The step of using the synthesized reference speech to train the preset phoneme prediction model to obtain the trained phoneme prediction model specifically includes:

[0022] Using the speech-to-text component to convert the synthesized reference speech into text, obtaining text character sequences corresponding to all speech segments in the synthesized reference speech;

[0023] Using the acoustic feature extraction component to extract acoustic features from all speech segments in the synthesized reference speech, respectively, to obtain phoneme features contained in all speech segments, wherein the phoneme features include pronunciation length information, timbre, tone, and pitch corresponding to each phonetic character;

[0024] Inputting the text character sequences corresponding to all the speech segments and the phoneme features contained in all the speech segments into the Viterbi algorithm component;

[0025] Calculating the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments through the Viterbi algorithm component;

[0026] The pronunciation length information corresponding to the text characters in all the speech segments in different speech segments is deployed as prediction knowledge information into the preset phoneme prediction model to obtain the trained phoneme prediction model.

[0027] Furthermore, the phoneme prediction model further includes a prediction result output component, wherein the step of inputting the target text character sequence into the trained phoneme prediction model and predicting the phoneme features and start and end time positions corresponding to each character in the target text character sequence specifically includes:

[0028] Identifying the speech segment corresponding to the target text character sequence;

[0029] Determining the timbre, tone, and pitch corresponding to each voice character in the target text character sequence according to the voice segment corresponding to the target text character sequence;

[0030] Determining the pronunciation length information corresponding to all text characters in the target text character sequence according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments;

[0031] Based on the pronunciation length information corresponding to all text characters in the target text character sequence, predicting the pronunciation start and end time positions of each character in the target text character sequence in the target output speech;

[0032] The timbre, tone, and pitch corresponding to each phonetic character in the target text character sequence, as well as the pronunciation start and end time positions of each character in the target text character sequence in the target output speech are output via the prediction result output component.

[0033] Furthermore, the step of determining the pronunciation length information corresponding to all text characters in the target text character sequence according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments specifically includes:

[0034] Identifying all text characters included in the target text character sequence as dynamic programming characters;

[0035] Generate a pronunciation length information screening domain corresponding to each of the dynamic programming characters according to the pronunciation length information corresponding to each of the text characters in each of the voice segments in different voice segments;

[0036] According to the speech segment corresponding to the target text character sequence, a recursive screening method is adopted to sequentially screen out the pronunciation length information corresponding to all dynamically programmed characters from the pronunciation length information screening domains corresponding to all dynamically programmed characters;

[0037] The pronunciation length information corresponding to all the dynamically planned characters is determined as the pronunciation length information corresponding to the corresponding text characters in the target text character sequence.

[0038] Furthermore, the step of performing acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence to obtain the target output speech specifically includes:

[0039] Inputting the timbre, tone, and pitch corresponding to each phonetic character in the target text character sequence, as well as the start and end time positions of each character in the target text character sequence in the target output speech, into a preset acoustic decoder, wherein the acoustic decoder includes a diffusion model-like step-by-step decoder;

[0040] According to the sequence position of each voice character in the target text character sequence, the acoustic decoder is used to gradually perform speech synthesis processing on each voice character to obtain the target output speech.

[0041] In order to solve the above technical problems, the embodiment of the present application also provides a speech synthesis device, which adopts the following technical solutions:

[0042] A speech synthesis device, comprising:

[0043] A synthesis data acquisition module is used to obtain synthesis reference speech and text data to be synthesized;

[0044] a text encoding and decoding processing module, configured to input the text data into a preset text processor, perform text encoding and decoding processing, and obtain a target text character sequence, wherein the preset text processor is composed of an N-layer transformer structure, and the position encoding mode of the text processor is set to a rotation position encoding mode, where N is a positive integer greater than 1;

[0045] A phoneme prediction model training module is used to train a preset phoneme prediction model using the synthesized reference speech to obtain a trained phoneme prediction model;

[0046] A phoneme prediction model prediction module is used to input the target text character sequence into the trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence;

[0047] The acoustic decoding processing module is used to perform acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence to obtain the target output speech.

[0048] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0049] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned speech synthesis method when executing the computer-readable instructions.

[0050] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0051] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech synthesis method described above.

[0052] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0053] The speech synthesis method described in the embodiment of the present application obtains a synthesized reference speech and text data to be synthesized; inputs the text data into a preset text processor, performs text encoding and decoding processing, and obtains a target text character sequence; inputs the target text character sequence into a trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; performs acoustic decoding processing on the target text character sequence based on the phoneme features and pronunciation start and end time positions to obtain a target output speech. By adopting a model prediction method, the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence are predicted, and each text character is decoded one by one to synthesize the target output speech, thereby achieving fast and high-quality speech synthesis output. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0055] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0056] Figure 2 is a flow chart of an embodiment of a speech synthesis method according to the present application;

[0057] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 201 is shown;

[0058] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 202 is shown;

[0059] Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 203 is shown;

[0060] Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 204 is shown;

[0061] Figure 7 yes Figure 6 A flowchart of a specific embodiment of step 603 is shown;

[0062] Figure 8 yes Figure 2 A flowchart of a specific embodiment of step 205 is shown

[0063] Figure 9 is a structural diagram of an embodiment of a speech synthesis device according to the present application;

[0064] Figure 10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0066] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0067] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0068] like Figure 1 As shown, system architecture 100 may include a terminal device 101, a network 102, and a server 103. Terminal device 101 may be a laptop computer 1011, a tablet computer 1012, or a mobile phone 1013. Network 102 is a medium for providing a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0069] The user can use the terminal device 101 to interact with the server 103 via the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0070] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0071] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0072] It should be noted that the speech synthesis method provided in the embodiments of the present application is generally executed by a terminal device, and accordingly, the speech synthesis device is generally provided in the terminal device.

[0073] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0074] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech synthesis method according to the present application. The speech synthesis method comprises the following steps:

[0075] Step 201: Acquire a synthetic reference speech and text data to be subjected to speech synthesis;

[0076] Specifically, the synthesized reference speech refers to the spoken speech package of the target speech object, and the text data to be speech synthesized includes pre-prepared speech text data. Applying the speech synthesis method described in this application to AI intelligent speech synthesis or automated customer service speech synthesis scenarios can greatly save manpower consumption and be more intelligent and automated.

[0077] Step 202: Input the text data into a preset text processor to perform text encoding and decoding processing to obtain a target text character sequence, wherein the preset text processor is composed of an N-layer transformer structure, and the position encoding mode of the text processor is set to a rotation position encoding mode, where N is a positive integer greater than 1;

[0078] In this embodiment, by setting the position encoding mode of the text processor to the rotation position encoding mode, Rotary Position Embedding (RoPE) is a technology used when processing sequence data in natural language processing, which aims to encode position information into the input representation by rotation, so that the model can better understand the position relationship of elements in the sequence. The core idea of ​​RoPE is to integrate position information into the representation of the word unit by applying a rotation matrix to the embedding of each word unit at each position. This encoding method allows the model to better utilize position information when processing sequence data, thereby improving the quality of semantic understanding and language generation. Complex multiplication is usually used to achieve rotation transformation. Specifically, a two-dimensional vector can be rotated by multiplying it by a complex number, namely cosθ or sinθ, and a combination of cosθ and sinθ, where θ is the rotation angle.

[0079] Specifically, the present application adopts a rotational position encoding mode, which aims to encode position information into the input representation by rotation, so that the text processor can better understand the positional relationship of elements in the sequence, improve semantic understanding and the generation quality of the target text character sequence.

[0080] Step 203: using the synthesized reference speech to train a preset phoneme prediction model to obtain a trained phoneme prediction model;

[0081] In this embodiment, the use of the synthesized reference speech to learn and train the preset phoneme prediction model is actually to use the correspondence between the speech text and the acoustic features in the synthesized reference speech to learn and train the preset phoneme prediction model, in the hope of learning the correspondence between text characters and phoneme features.

[0082] By learning and training the preset phoneme prediction model, a trained phoneme prediction model is obtained, so that when the phoneme feature correspondence relationship of the target text character sequence is subsequently predicted, the trained phoneme prediction model can be directly used, which is more automated.

[0083] Step 204: input the target text character sequence into the trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence;

[0084] In this embodiment, the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence are predicted, so that the target output speech synthesis can be performed by combining the phoneme features and pronunciation start and end time positions corresponding to each character.

[0085] Step 205 , performing acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence to obtain a target output speech.

[0086] In this embodiment, a synthetic reference speech and text data to be synthesized are obtained; the text data is input into a preset text processor for text encoding and decoding to obtain a target text character sequence; the target text character sequence is input into a trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; and the target text character sequence is acoustically decoded based on the phoneme features and pronunciation start and end time positions to obtain a target output speech. By using a model prediction approach to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence, and decoding each text character one by one to synthesize the target output speech, fast and high-quality speech synthesis output is achieved.

[0087] Continue to refer Figure 3 , in some optional implementations, Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 201 shown includes the following steps:

[0088] Step 301, filtering out a target object's voice packet from the reference voice packets provided by the target speech synthesis terminal by screening;

[0089] Step 302: extracting batches of speech segments from the target subject's speech packet as the synthesized reference speech;

[0090] Step 303: Acquire the text data sent or imported by the speech synthesis requesting end as the text data to be speech synthesized.

[0091] Specifically, the target object's voice package is selected from the reference voice package provided by the target voice synthesis terminal by screening, thereby ensuring that the subsequent actual voice synthesis is synthesized based on the preset real voice as a reference.

[0092] Continue to refer Figure 4 , in some optional implementations, Figure 4 yes Figure 2 The flowchart of a specific embodiment of step 202 shown includes the following steps:

[0093] Step 401, using the encoder layer in the transformer structure included in the text processor to convert all text in the text data into vector representations of target dimensions;

[0094] Step 402: adding position information to the vector representations corresponding to all the texts using a rotational position encoding method to obtain vector representations with embedded position information;

[0095] The rotational position encoding mode is designed to encode position information into the input representation by rotation, so that the text processor can better understand the positional relationship of elements in the sequence, improve semantic understanding and the generation quality of the target text character sequence. Compared with the previous encoding mode that strictly follows the character sequence, it improves the efficiency of position encoding, thereby improving the processing efficiency of the entire text processor.

[0096] Step 403: using a multi-head attention mechanism included in the encoder layer to capture long-term dependencies and semantic relationships between all texts in the text data from the vector representation of the embedded position information;

[0097] Step 404: Combining the long-term dependencies and semantic relationships among all texts in the text data, the decoder layer in the transformer structure included in the text processor is used to decode the vector representation of the embedded position information to obtain the target text character sequence.

[0098] By adopting the encoder layer and decoder layer in the multi-layer transformer structure, combined with the rotation position encoding method, it is ensured that when the text data to be subjected to speech synthesis is processed, a higher quality target text character sequence is obtained while ensuring the processing efficiency of the text processor.

[0099] In this embodiment, the phoneme prediction model includes a speech-to-text component, an acoustic feature extraction component, a Viterbi algorithm component, and a prediction result output component. Specifically, the speech-to-text component is used to first convert the speech signal of the synthesized reference speech into a digital signal, and then generate the corresponding speech text by combining the digital signal with natural language recognition technology. The acoustic feature extraction component is used to extract acoustic features from the synthesized reference speech and further analyze the acoustic features to obtain phoneme features, such as pronunciation duration information, timbre, tone, pitch, etc. The Viterbi algorithm component, combined with its built-in Viterbi algorithm, forcibly aligns the speech text and phoneme features in the synthesized reference speech, that is, determines the pronunciation duration information, timbre, tone, and pitch corresponding to each text character. The prediction result output component is used to output the phoneme features corresponding to each text character in the target text character sequence during subsequent actual prediction, including pronunciation duration information, timbre, tone, and pitch.

[0100] Continue to refer Figure 5, in some optional implementations, Figure 5 yes Figure 2 The flowchart of a specific embodiment of step 203 shown includes the following steps:

[0101] Step 501: Using the speech-to-text component to convert the synthesized reference speech into text, and obtaining text character sequences corresponding to all speech segments in the synthesized reference speech;

[0102] Step 502: Using the acoustic feature extraction component, extract acoustic features from all speech segments in the synthesized reference speech to obtain phoneme features contained in all speech segments, wherein the phoneme features include pronunciation length information, timbre, tone, and pitch corresponding to each phonetic character;

[0103] Step 503: inputting the text character sequences corresponding to all the speech segments and the phoneme features contained in all the speech segments into the Viterbi algorithm component;

[0104] Step 504: Calculate the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments through the Viterbi algorithm component;

[0105] The Viterbi algorithm is a forced alignment algorithm: given audio and text, it can determine the specific start and end time positions of each text character in the audio. Furthermore, the Viterbi algorithm accurately finds the specific start and end time positions of each text character through recursive iteration while maintaining computational efficiency.

[0106] Step 505 : deploying the pronunciation length information corresponding to the text characters in all the speech segments in different speech segments as prediction knowledge information into the preset phoneme prediction model to obtain the trained phoneme prediction model.

[0107] Through the Viterbi algorithm component, the pronunciation length information corresponding to the text characters in all speech segments in the synthesized reference speech in different speech segments is calculated and deployed as predictive knowledge information into the preset phoneme prediction model. This allows the phoneme features of each text character in the target output speech, namely, pronunciation duration information, timbre, tone, and pitch, to be predicted in advance when subsequently performing speech synthesis on the text data to be synthesized, thereby ensuring fast and efficient speech synthesis.

[0108] Continue to refer Figure 6 , in some optional implementations, Figure 6 yes Figure 2The flowchart of a specific embodiment of step 204 shown includes the following steps:

[0109] Step 601, identifying the speech segment corresponding to the target text character sequence;

[0110] Step 602, determining the timbre, tone, and pitch corresponding to each voice character in the target text character sequence based on the voice segment corresponding to the target text character sequence;

[0111] Step 603, determining the pronunciation length information corresponding to all text characters in the target text character sequence according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments;

[0112] Step 604: predicting the pronunciation start and end time positions of each character in the target text character sequence in the target output speech based on the pronunciation length information corresponding to all text characters in the target text character sequence;

[0113] Step 605: The timbre, tone, and pitch corresponding to each voice character in the target text character sequence, as well as the pronunciation start and end time positions of each character in the target text character sequence in the target output voice are output via the prediction result output component.

[0114] Specifically, by predicting the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence, the target output speech synthesis is performed by combining the phoneme features and pronunciation start and end time positions corresponding to each character.

[0115] Continue to refer Figure 7 , in some optional implementations, Figure 7 yes Figure 6 The flowchart of a specific embodiment of step 603 shown includes the following steps:

[0116] Step 701, identifying all text characters included in the target text character sequence as dynamic programming characters;

[0117] Step 702: generating a pronunciation length information screening domain corresponding to all dynamically programmed characters according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments;

[0118] Step 703, based on the speech segment corresponding to the target text character sequence, a recursive screening method is used to sequentially filter out the pronunciation length information corresponding to all dynamically programmed characters from the pronunciation length information screening domains corresponding to all dynamically programmed characters;

[0119] Step 704: determine the pronunciation length information corresponding to all the dynamically programmed characters as the pronunciation length information corresponding to the corresponding text characters in the target text character sequence.

[0120] Specifically, Figure 7 The detailed steps shown in the figure are to determine the pronunciation length information corresponding to all text characters in the target text character sequence according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments. In fact, the Viterbi algorithm in the Viterbi algorithm component uses a recursive iteration method to accurately find the specific start and end time position corresponding to each text character from the screening domain. The Viterbi algorithm can accurately find the specific start and end time position corresponding to each text character through a recursive iteration method while maintaining computational efficiency, thereby ensuring efficient and fast target output speech synthesis.

[0121] Continue to refer Figure 8 , in some optional implementations, Figure 8 yes Figure 2 The flowchart of a specific embodiment of step 205 shown includes the following steps:

[0122] Step 801: Inputting the timbre, tone, and pitch corresponding to each phonetic character in the target text character sequence, as well as the start and end time positions of each character in the target text character sequence in the target output speech, into a preset acoustic decoder, wherein the acoustic decoder includes a diffusion model-like step-by-step decoder;

[0123] Step 802: According to the sequence position of each voice character in the target text character sequence, the acoustic decoder is used to perform speech synthesis processing on each voice character step by step to obtain the target output speech.

[0124] By limiting the acoustic decoder to a diffusion model-like step-by-step decoder, and using the acoustic decoder to gradually perform speech synthesis processing on each speech character according to the sequence position of each speech character in the target text character sequence, the target output speech is obtained. Compared with the previous one-step generation mode, a new idea is used, and multi-step step-by-step generation is used to reconstruct all the details in the phoneme features corresponding to each text character, thereby ensuring the output of high-quality target synthesized speech.

[0125] This application obtains a synthetic reference speech and text data to be synthesized; inputs the text data into a preset text processor, performs text encoding and decoding processing, and obtains a target text character sequence; inputs the target text character sequence into a trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; acoustically decodes the target text character sequence based on the phoneme features and pronunciation start and end time positions to obtain the target output speech. By adopting a model prediction method, the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence are predicted, and each text character is decoded one by one to synthesize the target output speech, thereby achieving fast and high-quality speech synthesis output.

[0126] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0127] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0128] In an embodiment of the present application, a synthetic reference speech and text data to be synthesized are obtained; the text data is input into a preset text processor to perform text encoding and decoding processing to obtain a target text character sequence; the target text character sequence is input into a trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; the target text character sequence is acoustically decoded according to the phoneme features and pronunciation start and end time positions to obtain a target output speech. By adopting a model prediction method, the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence are predicted, and each text character is decoded one by one to synthesize the target output speech, thereby achieving fast and high-quality speech synthesis output.

[0129] Further references Figure 9 , as a response to the above Figure 2 The present application provides an embodiment of a speech synthesis device, which is similar to the embodiment of the present invention. Figure 2Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0130] like Figure 9 As shown, the speech synthesis device 900 of this embodiment includes: a synthesis data acquisition module 901, a text encoding and decoding processing module 902, a phoneme prediction model training module 903, a phoneme prediction model prediction module 904 and an acoustic decoding processing module 905. Among them:

[0131] The synthesis data acquisition module 901 is used to obtain the synthesis reference speech and the text data to be synthesized;

[0132] A text encoding and decoding processing module 902 is configured to input the text data into a preset text processor, perform text encoding and decoding processing, and obtain a target text character sequence, wherein the preset text processor is composed of an N-layer transformer structure, and the position encoding mode of the text processor is set to a rotation position encoding mode, where N is a positive integer greater than 1;

[0133] The phoneme prediction model training module 903 is used to train a preset phoneme prediction model using the synthesized reference speech to obtain a trained phoneme prediction model;

[0134] The phoneme prediction model prediction module 904 is used to input the target text character sequence into the trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence;

[0135] The acoustic decoding processing module 905 is used to perform acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence to obtain the target output speech.

[0136] This application obtains a synthetic reference speech and text data to be synthesized; inputs the text data into a preset text processor, performs text encoding and decoding processing, and obtains a target text character sequence; inputs the target text character sequence into a trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; acoustically decodes the target text character sequence based on the phoneme features and pronunciation start and end time positions to obtain the target output speech. By adopting a model prediction method, the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence are predicted, and each text character is decoded one by one to synthesize the target output speech, thereby achieving fast and high-quality speech synthesis output.

[0137] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware via computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0138] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0139] To solve the above technical problems, the present application also provides a computer device. Figure 10 , Figure 10 This is a basic structural block diagram of the computer device in this embodiment.

[0140] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c that are interconnected via a system bus. Figure 10 Only a computer device 10 having components such as a memory 10a, a processor 10b, and a network interface 10c is shown. However, it should be understood that implementation of all shown components is not required, and more or fewer components may be implemented instead. Those skilled in the art will understand that a computer device herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, and the like.

[0141] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0142] The memory 10a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10a may be an internal storage unit of the computer device 10, such as a hard disk or memory of the computer device 10. In other embodiments, the memory 10a may also be an external storage device of the computer device 10, such as a plug-in hard disk equipped on the computer device 10, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc. Of course, the memory 10a may also include both the internal storage unit of the computer device 10 and its external storage device. In this embodiment, the memory 10a is generally used to store an operating system and various application software installed on the computer device 10, such as computer-readable instructions for a speech synthesis method. In addition, the memory 10a can also be used to temporarily store various data that has been output or is to be output.

[0143] The processor 10b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 10b is generally used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to run computer-readable instructions or process data stored in the memory 10a, such as computer-readable instructions for running the speech synthesis method.

[0144] The network interface 10c may include a wireless network interface or a wired network interface. The network interface 10c is generally used to establish a communication connection between the computer device 10 and other electronic devices.

[0145] The computer device proposed in this embodiment belongs to the field of speech processing technology and is applied to text-to-speech scenarios. This application obtains a synthesized reference speech and text data to be synthesized; inputs the text data into a preset text processor, performs text encoding and decoding processing, and obtains a target text character sequence; inputs the target text character sequence into a trained phoneme prediction model, and predicts the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; performs acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions to obtain the target output speech. By adopting a model prediction method, the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence are predicted, and each text character is decoded one by one to synthesize the target output speech, thereby achieving fast and high-quality speech synthesis output.

[0146] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the speech synthesis method as described above.

[0147] The computer-readable storage medium proposed in this embodiment belongs to the field of speech processing technology and is applied to text-to-speech scenarios. This application obtains a synthesized reference speech and text data to be synthesized; inputs the text data into a preset text processor, performs text encoding and decoding processing, and obtains a target text character sequence; inputs the target text character sequence into a trained phoneme prediction model, and predicts the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; performs acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions to obtain the target output speech. By adopting a model prediction method, the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence are predicted, and each text character is decoded one by one to synthesize the target output speech, thereby achieving fast and high-quality speech synthesis output.

[0148] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0149] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A speech synthesis method, characterized in that: The steps include: Obtaining a synthesized reference speech and text data to be synthesized; Inputting the text data into a preset text processor and performing text encoding and decoding processing to obtain a target text character sequence, wherein the preset text processor is composed of an N-layer transformer structure, and the position encoding mode of the text processor is set to a rotation position encoding mode, where N is a positive integer greater than 1; Using the synthesized reference speech to train a preset phoneme prediction model to obtain a trained phoneme prediction model; Inputting the target text character sequence into the trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; Performing acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence to obtain the target output speech, The phoneme prediction model includes a speech-to-text component, an acoustic feature extraction component, and a Viterbi algorithm component. The step of using the synthesized reference speech to train the preset phoneme prediction model to obtain the trained phoneme prediction model specifically includes: Using the speech-to-text component to convert the synthesized reference speech into text, obtaining text character sequences corresponding to all speech segments in the synthesized reference speech; Using the acoustic feature extraction component to extract acoustic features from all speech segments in the synthesized reference speech, respectively, to obtain phoneme features contained in all speech segments, wherein the phoneme features include pronunciation length information, timbre, tone, and pitch corresponding to each phonetic character; Inputting the text character sequences corresponding to all the speech segments and the phoneme features contained in all the speech segments into the Viterbi algorithm component; Calculating the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments through the Viterbi algorithm component; Deploying the pronunciation length information corresponding to the text characters in all the speech segments in different speech segments as prediction knowledge information into the preset phoneme prediction model to obtain the trained phoneme prediction model; The phoneme prediction model further includes a prediction result output component. The step of inputting the target text character sequence into the trained phoneme prediction model and predicting the phoneme features and start and end time positions corresponding to each character in the target text character sequence specifically includes: Identifying the speech segment corresponding to the target text character sequence; Determining the timbre, tone, and pitch corresponding to each voice character in the target text character sequence according to the voice segment corresponding to the target text character sequence; Determining the pronunciation length information corresponding to all text characters in the target text character sequence according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments; Based on the pronunciation length information corresponding to all text characters in the target text character sequence, predicting the pronunciation start and end time positions of each character in the target text character sequence in the target output speech; The timbre, tone, and pitch corresponding to each phonetic character in the target text character sequence, as well as the pronunciation start and end time positions of each character in the target text character sequence in the target output speech are output via the prediction result output component.

2. The speech synthesis method according to claim 1, wherein: The step of obtaining the synthesized reference speech and the text data to be subjected to speech synthesis specifically includes: Filtering the target object's voice package from the reference voice package provided by the target speech synthesis end by screening; Extracting batches of speech segments from the speech package of the target object as the synthesized reference speech; The text data sent or imported by the speech synthesis request end is obtained as the text data to be speech synthesized.

3. The speech synthesis method according to claim 1, wherein: The step of inputting the text data into a preset text processor, performing text encoding and decoding processing, and obtaining a target text character sequence specifically includes: Utilizing the encoder layer in the transformer structure included in the text processor to convert all text in the text data into vector representations of target dimensions; Using a rotational position encoding method to add position information to the vector representations corresponding to all the texts to obtain vector representations with embedded position information; Utilizing the multi-head attention mechanism included in the encoder layer, capturing the long-term dependencies and semantic relationships between all texts in the text data from the vector representation of the embedded position information; In combination with the long-term dependencies and semantic relationships between all texts in the text data, the decoder layer in the transformer structure included in the text processor is used to decode the vector representation of the embedded position information to obtain the target text character sequence.

4. The speech synthesis method according to claim 1, wherein: The step of determining the pronunciation length information corresponding to all text characters in the target text character sequence according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments specifically includes: Identifying all text characters included in the target text character sequence as dynamic programming characters; Generate a pronunciation length information screening domain corresponding to each of the dynamic programming characters according to the pronunciation length information corresponding to each of the text characters in each of the voice segments in different voice segments; According to the speech segment corresponding to the target text character sequence, a recursive screening method is adopted to sequentially screen out the pronunciation length information corresponding to all dynamically programmed characters from the pronunciation length information screening domains corresponding to all dynamically programmed characters; The pronunciation length information corresponding to all the dynamically planned characters is determined as the pronunciation length information corresponding to the corresponding text characters in the target text character sequence.

5. The speech synthesis method according to claim 1, wherein: The step of performing acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence to obtain the target output speech specifically includes: Inputting the timbre, tone, and pitch corresponding to each phonetic character in the target text character sequence, as well as the start and end time positions of each character in the target text character sequence in the target output speech, into a preset acoustic decoder, wherein the acoustic decoder includes a diffusion model-like step-by-step decoder; According to the sequence position of each voice character in the target text character sequence, the acoustic decoder is used to gradually perform speech synthesis processing on each voice character to obtain the target output speech.

6. A speech synthesis device, characterized in that: include: A synthesis data acquisition module is used to obtain synthesis reference speech and text data to be synthesized; a text encoding and decoding processing module, configured to input the text data into a preset text processor, perform text encoding and decoding processing, and obtain a target text character sequence, wherein the preset text processor is composed of an N-layer transformer structure, and the position encoding mode of the text processor is set to a rotation position encoding mode, where N is a positive integer greater than 1; A phoneme prediction model training module is used to train a preset phoneme prediction model using the synthesized reference speech to obtain a trained phoneme prediction model; A phoneme prediction model prediction module is used to input the target text character sequence into the trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; The acoustic decoding processing module is used to perform acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence to obtain the target output speech. The phoneme prediction model includes a speech-to-text component, an acoustic feature extraction component, and a Viterbi algorithm component. The step of using the synthesized reference speech to train the preset phoneme prediction model to obtain the trained phoneme prediction model specifically includes: Using the speech-to-text component to convert the synthesized reference speech into text, obtaining text character sequences corresponding to all speech segments in the synthesized reference speech; Using the acoustic feature extraction component to extract acoustic features from all speech segments in the synthesized reference speech, respectively, to obtain phoneme features contained in all speech segments, wherein the phoneme features include pronunciation length information, timbre, tone, and pitch corresponding to each phonetic character; Inputting the text character sequences corresponding to all the speech segments and the phoneme features contained in all the speech segments into the Viterbi algorithm component; Calculating the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments through the Viterbi algorithm component; Deploying the pronunciation length information corresponding to the text characters in all the speech segments in different speech segments as prediction knowledge information into the preset phoneme prediction model to obtain the trained phoneme prediction model; The phoneme prediction model further includes a prediction result output component. The step of inputting the target text character sequence into the trained phoneme prediction model and predicting the phoneme features and start and end time positions corresponding to each character in the target text character sequence specifically includes: Identifying the speech segment corresponding to the target text character sequence; Determining the timbre, tone, and pitch corresponding to each voice character in the target text character sequence according to the voice segment corresponding to the target text character sequence; Determining the pronunciation length information corresponding to all text characters in the target text character sequence according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments; Based on the pronunciation length information corresponding to all text characters in the target text character sequence, predicting the pronunciation start and end time positions of each character in the target text character sequence in the target output speech; The timbre, tone, and pitch corresponding to each phonetic character in the target text character sequence, as well as the pronunciation start and end time positions of each character in the target text character sequence in the target output speech are output via the prediction result output component.

7. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech synthesis method according to any one of claims 1 to 5 when executing the computer-readable instructions.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech synthesis method according to any one of claims 1 to 5.

Images