Speech synthesis method, device and equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of speech processing, is applied to a text-to-speech scene, and relates to a speech synthesis method, device and equipment and a storage medium thereof. Performing text coding and decoding processing on the text data to obtain a target text character sequence; inputting the target text character sequence into the trained phoneme prediction model, and predicting phoneme features and pronunciation starting and ending time positions respectively corresponding to each character in the target text character sequence; and performing acoustic decoding processing on the target text character sequence according to the phoneme characteristics and the pronunciation starting and ending time positions to obtain a target output voice. According to the method, the phoneme feature and the pronunciation starting and ending time position corresponding to each character in the target text character sequence are predicted in a model prediction mode, decoding is carried out one by one according to each text character, so that the target output voice is synthesized, and rapid and high-quality voice synthesis output is realized.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and is applied to text-to-speech scenarios, and in particular to a speech synthesis method, apparatus, device, and storage medium thereof. Background Art

[0002] Speech synthesis technology refers to synthesizing the pronunciation of the target speaker from a specified text. Currently, speech synthesis technology has been rapidly developed, and end-to-end models and large speech models have greatly promoted the development of this field.

[0003] However, in practical applications, end-to-end speech synthesis faces the following challenges: the model is too large, and conventional text encoding uses a method of encoding each sentence one by one according to the sequence position, which results in excessive computation. Traditional decoding methods often decode an entire text sentence at one time, resulting in inaccurate decoding and reduced speech synthesis quality. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a speech synthesis method, apparatus, device and storage medium thereof to solve the problem that in the existing speech synthesis, the amount of calculation is too much during encoding and the decoding is not accurate enough, which reduces the quality of speech synthesis.

[0005] In order to solve the above technical problems, the present application provides a speech synthesis method, which adopts the following technical solutions:

[0006] A speech synthesis method comprises the following steps:

[0007] Obtaining synthetic reference speech and text data to be synthesized;

[0008] Input the text data into a preset text processor, perform text encoding and decoding processing, and obtain a target text character sequence, wherein the preset text processor is composed of an N-layer transformer structure, and the position encoding mode of the text processor is set to a rotation position encoding mode, and N is a positive integer greater than 1;

[0009] Using the synthesized reference speech to learn and train a preset phoneme prediction model to obtain a trained phoneme prediction model;

[0010] Inputting the target text character sequence into the trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence;

[0011] The target text character sequence is acoustically decoded according to the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence to obtain the target output speech.

[0012] Furthermore, the step of obtaining the synthesized reference speech and the text data to be synthesized includes:

[0013] Filtering the speech package of the target object from the reference speech package provided by the target speech synthesis end by screening;

[0014] Extracting batches of speech segments from the speech packet of the target object as the synthesized reference speech;

[0015] The text data sent or imported by the speech synthesis request end is obtained as the text data to be subjected to speech synthesis.

[0016] Furthermore, the step of inputting the text data into a preset text processor, performing text encoding and decoding processing, and obtaining a target text character sequence specifically includes:

[0017] Using the encoder layer in the transformer structure included in the text processor to convert all text in the text data into a vector representation of a target dimension;

[0018] Using a rotation position encoding method to add position information to the vector representations corresponding to all the texts to obtain vector representations with embedded position information;

[0019] Utilizing the multi-head attention mechanism included in the encoder layer, capturing the long-term dependencies and semantic relationships between all texts in the text data from the vector representation of the embedded position information;

[0020] In combination with the long-term dependencies and semantic relationships among all the texts in the text data, the decoder layer in the transformer structure included in the text processor is used to decode the vector representation of the embedded position information to obtain the target text character sequence.

[0021] Furthermore, the phoneme prediction model includes a speech-to-text component, an acoustic feature extraction component, and a Viterbi algorithm component. The step of using the synthesized reference speech to learn and train the preset phoneme prediction model to obtain the trained phoneme prediction model specifically includes:

[0022] The speech-to-text component is used to convert the synthesized reference speech into text, so as to obtain text character sequences corresponding to all speech segments in the synthesized reference speech;

[0023] The acoustic feature extraction component is used to extract acoustic features of all speech segments in the synthesized reference speech to obtain phoneme features contained in all speech segments, wherein the phoneme features include pronunciation length information, timbre, tone, and pitch corresponding to each phonetic character;

[0024] Inputting the text character sequences corresponding to all the speech segments and the phoneme features respectively contained in all the speech segments into the Viterbi algorithm component;

[0025] By using the Viterbi algorithm component, the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments is calculated;

[0026] The pronunciation length information corresponding to the text characters in all the speech segments in different speech segments is deployed as prediction knowledge information into the preset phoneme prediction model to obtain the trained phoneme prediction model.

[0027] Furthermore, the phoneme prediction model further includes a prediction result output component, wherein the step of inputting the target text character sequence into the trained phoneme prediction model and predicting the phoneme features and start and end time positions corresponding to each character in the target text character sequence specifically includes:

[0028] Identifying the speech segment corresponding to the target text character sequence;

[0029] Determine the timbre, tone, and pitch corresponding to each voice character in the target text character sequence according to the voice segment corresponding to the target text character sequence;

[0030] Determine the pronunciation length information corresponding to all the text characters in the target text character sequence according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments;

[0031] Based on the pronunciation length information corresponding to all the text characters in the target text character sequence, predict the pronunciation start and end time position of each character in the target text character sequence in the target output speech;

[0032] The timbre, tone, pitch corresponding to each voice character in the target text character sequence, and the pronunciation start and end time positions of each character in the target text character sequence in the target output voice are output via the prediction result output component.

[0033] Furthermore, the step of determining the pronunciation length information corresponding to all the text characters in the target text character sequence respectively according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments specifically includes:

[0034] Identify all text characters included in the target text character sequence as dynamic programming characters;

[0035] Generate a pronunciation length information screening domain corresponding to all dynamic programming characters respectively according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments;

[0036] According to the speech segment corresponding to the target text character sequence, a recursive screening method is adopted to sequentially screen out the pronunciation length information corresponding to all the dynamically programmed characters from the pronunciation length information screening domains corresponding to all the dynamically programmed characters;

[0037] The pronunciation length information corresponding to all the dynamically planned characters is determined as the pronunciation length information corresponding to the corresponding text characters in the target text character sequence.

[0038] Furthermore, the step of performing acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence to obtain the target output speech specifically includes:

[0039] Inputting the timbre, tone, and pitch corresponding to each voice character in the target text character sequence, as well as the start and end time positions of pronunciation of each character in the target text character sequence in the target output voice into a preset acoustic decoder, wherein the acoustic decoder includes a diffusion model-like step-by-step decoder;

[0040] According to the sequence position of each voice character in the target text character sequence, the acoustic decoder is used to gradually perform speech synthesis processing on each voice character to obtain the target output speech.

[0041] In order to solve the above technical problems, the embodiment of the present application also provides a speech synthesis device, which adopts the following technical solution:

[0042] A speech synthesis device, comprising:

[0043] A synthesis data acquisition module, used to acquire synthesis reference speech and text data to be synthesized;

[0044] A text encoding and decoding processing module, used for inputting the text data into a preset text processor, performing text encoding and decoding processing, and obtaining a target text character sequence, wherein the preset text processor is composed of an N-layer transformer structure, and the position encoding mode of the text processor is set to a rotation position encoding mode, and N is a positive integer greater than 1;

[0045] A phoneme prediction model training module, used to use the synthesized reference speech to learn and train a preset phoneme prediction model to obtain a trained phoneme prediction model;

[0046] A phoneme prediction model prediction module, used to input the target text character sequence into the trained phoneme prediction model, and predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence;

[0047] The acoustic decoding processing module is used to perform acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence to obtain the target output speech.

[0048] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0049] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned speech synthesis method when executing the computer-readable instructions.

[0050] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0051] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech synthesis method as described above.

[0052] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0053] The speech synthesis method described in the embodiment of the present application obtains a synthesized reference speech and text data to be synthesized; inputs the text data into a preset text processor, performs text encoding and decoding processing, and obtains a target text character sequence; inputs the target text character sequence into a trained phoneme prediction model, predicts the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; performs acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions, and obtains the target output speech. By adopting a model prediction method, the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence are predicted, and each text character is decoded one by one to synthesize the target output speech, thereby achieving fast and high-quality speech synthesis output. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0056] Figure 2 is a flow chart of an embodiment of a speech synthesis method according to the present application;

[0057] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 201 is shown;

[0058] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 202 is shown;

[0059] Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 203 is shown;

[0060] Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 204 is shown;

[0061] Figure 7 yes Figure 6 A flowchart of a specific embodiment of step 603 is shown;

[0062] Figure 8 yes Figure 2 A flowchart of a specific embodiment of step 205 is shown

[0063] Fig. 9 is a structural schematic diagram of an embodiment of a speech synthesis device according to the present application;

[0064] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0066] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0067] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0068] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0069] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0070] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0071] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0072] It should be noted that the speech synthesis method provided in the embodiments of the present application is generally executed by a terminal device, and accordingly, the speech synthesis device is generally arranged in the terminal device.

[0073] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0074] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech synthesis method according to the present application. The speech synthesis method comprises the following steps:

[0075] Step 201, obtaining a synthetic reference speech and text data to be subjected to speech synthesis;

[0076] Specifically, the synthesized reference speech refers to the spoken speech package of the target speech object, and the text data to be speech synthesized includes pre-prepared speech text data. Applying the speech synthesis method described in this application to AI intelligent speech synthesis or automated customer service speech synthesis scenarios can greatly save manpower consumption and be more intelligent and automated.

[0077] Step 202, inputting the text data into a preset text processor, performing text encoding and decoding processing, and obtaining a target text character sequence, wherein the preset text processor is composed of an N-layer transformer structure, and the position encoding mode of the text processor is set to a rotation position encoding mode, and N is a positive integer greater than 1;

[0078] In this embodiment, by setting the position encoding mode of the text processor to the rotation position encoding mode, rotation position encoding (Rotary Position Embedding, RoPE) is a technology used in natural language processing when processing sequence data, which aims to encode the position information into the input representation by rotation, so that the model can better understand the position relationship of the elements in the sequence. The core idea of ​​RoPE is to integrate the position information into the representation of the word unit by applying a rotation matrix to the embedding of each word unit at each position. This encoding method allows the model to better utilize the position information when processing sequence data, and improve the quality of semantic understanding and language generation. Complex multiplication is usually used to achieve rotation transformation. Specifically, a two-dimensional vector can be rotated by multiplying it by a complex number, namely cosθ or sinθ, and the combined complex number of cosθ and sinθ, where θ is the rotation angle.

[0079] Specifically, the present application adopts a rotational position encoding mode, which aims to encode the position information into the input representation by rotation, so that the text processor can better understand the positional relationship of the elements in the sequence, improve the semantic understanding and the generation quality of the target text character sequence.

[0080] Step 203, using the synthesized reference speech to train a preset phoneme prediction model to obtain a trained phoneme prediction model;

[0081] In this embodiment, the use of the synthetic reference speech to learn and train the preset phoneme prediction model is actually to use the correspondence between the speech text and the acoustic features in the synthetic reference speech to learn and train the preset phoneme prediction model in the hope of learning the correspondence between text characters and phoneme features.

[0082] By learning and training the preset phoneme prediction model, a trained phoneme prediction model is obtained, so that when the phoneme feature correspondence relationship of the target text character sequence is subsequently predicted, the trained phoneme prediction model can be directly used, which is more automated.

[0083] Step 204, inputting the target text character sequence into the trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence;

[0084] In this embodiment, the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence are predicted, so that the target output speech synthesis can be performed by combining the phoneme features and pronunciation start and end time positions corresponding to each character.

[0085] Step 205, performing acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence, to obtain the target output speech.

[0086] In this embodiment, a synthetic reference speech and text data to be synthesized are obtained; the text data is input into a preset text processor to perform text encoding and decoding processing to obtain a target text character sequence; the target text character sequence is input into a trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; the target text character sequence is acoustically decoded according to the phoneme features and pronunciation start and end time positions to obtain a target output speech. By adopting a model prediction method, the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence are predicted, and each text character is decoded one by one to synthesize the target output speech, thereby achieving fast and high-quality speech synthesis output.

[0087] Continue to refer Figure 3 , in some optional implementations, Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 201 shown includes the following steps:

[0088] Step 301, filtering out a speech packet of a target object from reference speech packets provided by a target speech synthesis terminal by screening;

[0089] Step 302, extracting batches of speech segments from the speech package of the target object as the synthesized reference speech;

[0090] Step 303: Acquire text data sent or imported by the speech synthesis request end as the text data to be speech synthesized.

[0091] Specifically, the speech package of the target object is screened out from the reference speech package provided by the target speech synthesis end by screening, thereby ensuring that the subsequent actual speech synthesis is synthesized based on the preset real speech as a reference.

[0092] Continue to refer Figure 4 , in some optional implementations, Figure 4 yes Figure 2 The flowchart of a specific embodiment of step 202 shown includes the following steps:

[0093] Step 401, using the encoder layer in the transformer structure included in the text processor to convert all texts in the text data into vector representations of target dimensions;

[0094] Step 402, adding position information to the vector representations corresponding to all the texts by using a rotation position encoding method to obtain vector representations with embedded position information;

[0095] The rotational position encoding mode is designed to encode the position information into the input representation by rotation, so that the text processor can better understand the positional relationship of the elements in the sequence, improve the semantic understanding and the generation quality of the target text character sequence. Compared with the previous encoding mode that strictly follows the character sequence, it improves the position encoding efficiency, thereby improving the processing efficiency of the entire text processor.

[0096] Step 403, using the multi-head attention mechanism included in the encoder layer to capture the long-term dependencies and semantic relationships between all texts in the text data from the vector representation of the embedded position information;

[0097] Step 404, combining the long-term dependencies and semantic relationships between all texts in the text data, using the decoder layer in the transformer structure included in the text processor to decode the vector representation of the embedded position information to obtain the target text character sequence.

[0098] By adopting the encoder layer and the decoder layer in the multi-layer transformer structure, combined with the rotation position encoding method, it is ensured that when the text data to be subjected to speech synthesis is processed, a higher quality target text character sequence is obtained while ensuring the processing efficiency of the text processor.

[0099] In this embodiment, the phoneme prediction model includes a speech-to-text component, an acoustic feature extraction component, a Viterbi algorithm component and a prediction result output component. Specifically, the speech-to-text component is used to first perform speech signal conversion processing on the synthesized reference speech, and then generate corresponding speech text by combining the digitized signal with natural language recognition technology; the acoustic feature extraction component is used to extract acoustic features of the synthesized reference speech, and further analyze the acoustic features to obtain phoneme features, such as pronunciation duration information, timbre, tone, pitch, etc.; the Viterbi algorithm component, combined with its built-in Viterbi algorithm, forces alignment of the speech text and phoneme features in the synthesized reference speech, that is, determines the pronunciation duration information, timbre, tone, and pitch corresponding to each text character; the prediction result output component is used for subsequent actual prediction to output the phoneme features corresponding to each text character in the target text character sequence, including pronunciation duration information, timbre, tone, and pitch.

[0100] Continue to refer Figure 5, in some optional implementations, Figure 5 yes Figure 2 The flowchart of a specific embodiment of step 203 shown includes the following steps:

[0101] Step 501, using the speech-to-text component to convert the synthesized reference speech into text, to obtain text character sequences corresponding to all speech segments in the synthesized reference speech;

[0102] Step 502, using the acoustic feature extraction component to extract acoustic features from all speech segments in the synthesized reference speech, to obtain phoneme features contained in all speech segments, wherein the phoneme features include pronunciation length information, timbre, tone, and pitch corresponding to each phonetic character;

[0103] Step 503, inputting the text character sequences corresponding to all the voice segments and the phoneme features contained in all the voice segments into the Viterbi algorithm component;

[0104] Step 504, using the Viterbi algorithm component, calculating the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments;

[0105] The Viterbi algorithm is a forced alignment algorithm: given audio and text, the forced alignment algorithm can determine the specific start and end time positions corresponding to each text character in the audio. Moreover, the Viterbi algorithm can accurately find the specific start and end time positions corresponding to each text character through recursive iteration while maintaining computational efficiency.

[0106] Step 505: deploy the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments as prediction knowledge information into the preset phoneme prediction model to obtain the trained phoneme prediction model.

[0107] Through the Viterbi algorithm component, the pronunciation length information corresponding to the text characters in all the speech segments in the synthesized reference speech in different speech segments is calculated, and deployed as predictive knowledge information into the preset phoneme prediction model, so that when the text data to be speech synthesized is subsequently subjected to speech synthesis, the phoneme features of each text character in the target output speech, namely, the pronunciation duration information, timbre, tone, and pitch, can be predicted in advance, thereby ensuring fast and efficient speech synthesis.

[0108] Continue to refer Figure 6 , in some optional implementations, Figure 6 yes Figure 2The flowchart of a specific embodiment of step 204 shown includes the following steps:

[0109] Step 601, identifying the speech segment corresponding to the target text character sequence;

[0110] Step 602, determining the timbre, tone, and pitch corresponding to each voice character in the target text character sequence according to the voice segment corresponding to the target text character sequence;

[0111] Step 603, determining the pronunciation length information corresponding to all the text characters in the target text character sequence according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments;

[0112] Step 604, predicting the pronunciation start and end time positions of each character in the target text character sequence in the target output speech based on the pronunciation length information corresponding to all text characters in the target text character sequence;

[0113] Step 605, the timbre, tone, pitch corresponding to each voice character in the target text character sequence, and the pronunciation start and end time positions of each character in the target text character sequence in the target output voice are output through the prediction result output component.

[0114] Specifically, by predicting the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence, the target output speech synthesis is performed by combining the phoneme features and pronunciation start and end time positions corresponding to each character.

[0115] Continue to refer Figure 7 , in some optional implementations, Figure 7 yes Figure 6 The flowchart of a specific embodiment of step 603 shown includes the following steps:

[0116] Step 701, identifying all text characters included in the target text character sequence as dynamic programming characters;

[0117] Step 702, generating a pronunciation length information screening domain corresponding to all dynamic programming characters respectively according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments;

[0118] Step 703, according to the speech segment corresponding to the target text character sequence, a recursive screening method is used to sequentially screen out the pronunciation length information corresponding to all the dynamically programmed characters from the pronunciation length information screening domains corresponding to all the dynamically programmed characters;

[0119] Step 704: determine the pronunciation length information corresponding to all the dynamically programmed characters as the pronunciation length information corresponding to the corresponding text characters in the target text character sequence.

[0120] Specifically, Figure 7 The detailed steps shown in the figure are to determine the pronunciation length information corresponding to all the text characters in the target text character sequence according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments. In fact, the Viterbi algorithm in the Viterbi algorithm component uses a recursive iteration method to accurately find the specific start and end time positions corresponding to each text character from the screening domain. The Viterbi algorithm can accurately find the specific start and end time positions corresponding to each text character through a recursive iteration method while maintaining computational efficiency, thereby ensuring efficient and fast target output speech synthesis.

[0121] Continue to refer Figure 8 , in some optional implementations, Figure 8 yes Figure 2 The flowchart of a specific embodiment of step 205 shown includes the following steps:

[0122] Step 801, inputting the timbre, tone, pitch corresponding to each voice character in the target text character sequence, and the pronunciation start and end time positions of each character in the target text character sequence in the target output voice into a preset acoustic decoder, wherein the acoustic decoder includes a diffusion model-like step-by-step decoder;

[0123] Step 802: According to the sequence position of each voice character in the target text character sequence, the acoustic decoder is used to gradually perform speech synthesis processing on each voice character to obtain the target output speech.

[0124] By limiting the acoustic decoder to a diffusion model-like step-by-step decoder, according to the sequence position of each phonetic character in the target text character sequence, the acoustic decoder is used to gradually perform speech synthesis processing on each phonetic character to obtain the target output speech. Compared with the previous one-step generation mode, a new idea is used, using multi-step step-by-step generation, which realizes the reconstruction of all the details in the phoneme features corresponding to each text character, thus ensuring the output of high-quality target synthesized speech.

[0125] The present application obtains a synthetic reference speech and text data to be synthesized; inputs the text data into a preset text processor, performs text encoding and decoding processing, and obtains a target text character sequence; inputs the target text character sequence into a trained phoneme prediction model, predicts the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; performs acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions, and obtains the target output speech. By adopting a model prediction method, the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence are predicted, and each text character is decoded one by one, thereby synthesizing the target output speech, achieving fast and high-quality speech synthesis output.

[0126] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0127] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0128] In the embodiment of the present application, a synthetic reference speech and text data to be synthesized are obtained; the text data is input into a preset text processor to perform text encoding and decoding processing to obtain a target text character sequence; the target text character sequence is input into a trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; the target text character sequence is acoustically decoded according to the phoneme features and pronunciation start and end time positions to obtain a target output speech. By adopting a model prediction method, the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence are predicted, and each text character is decoded one by one to synthesize the target output speech, thereby achieving fast and high-quality speech synthesis output.

[0129] Further references Fig. 9 , as a response to the above Figure 2 The present application provides an embodiment of a speech synthesis device, and the device embodiment is similar to Figure 2Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0130] like Fig. 9 As shown, the speech synthesis device 900 described in this embodiment includes: a synthesis data acquisition module 901, a text encoding and decoding processing module 902, a phoneme prediction model training module 903, a phoneme prediction model prediction module 904 and an acoustic decoding processing module 905. Among them:

[0131] The synthesis data acquisition module 901 is used to acquire the synthesis reference speech and the text data to be synthesized;

[0132] The text encoding and decoding processing module 902 is used to input the text data into a preset text processor, perform text encoding and decoding processing, and obtain a target text character sequence, wherein the preset text processor is composed of an N-layer transformer structure, and the position encoding mode of the text processor is set to a rotation position encoding mode, and N is a positive integer greater than 1;

[0133] The phoneme prediction model training module 903 is used to use the synthesized reference speech to learn and train the preset phoneme prediction model to obtain a trained phoneme prediction model;

[0134] The phoneme prediction model prediction module 904 is used to input the target text character sequence into the trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence;

[0135] The acoustic decoding processing module 905 is used to perform acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence to obtain the target output speech.

[0136] The present application obtains a synthetic reference speech and text data to be synthesized; inputs the text data into a preset text processor, performs text encoding and decoding processing, and obtains a target text character sequence; inputs the target text character sequence into a trained phoneme prediction model, predicts the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; performs acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions, and obtains the target output speech. By adopting a model prediction method, the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence are predicted, and each text character is decoded one by one, thereby synthesizing the target output speech, achieving fast and high-quality speech synthesis output.

[0137] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0138] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0139] To solve the above technical problems, the present application also provides a computer device. Fig.10 , Fig.10 This is a basic structural block diagram of the computer device in this embodiment.

[0140] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c which are interconnected and communicated via a system bus. It should be noted that: Fig.10 Only a computer device 10 having a component memory 10a, a processor 10b, and a network interface 10c is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0141] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0142] The memory 10a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10a can be an internal storage unit of the computer device 10, such as a hard disk or memory of the computer device 10. In other embodiments, the memory 10a can also be an external storage device of the computer device 10, such as a plug-in hard disk equipped on the computer device 10, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10a can also include both the internal storage unit of the computer device 10 and its external storage device. In this embodiment, the memory 10a is generally used to store an operating system and various application software installed on the computer device 10, such as a computer-readable instruction of a speech synthesis method, etc. In addition, the memory 10a can also be used to temporarily store various data that have been output or will be output.

[0143] The processor 10b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 10b is generally used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to run the computer-readable instructions or process data stored in the memory 10a, such as the computer-readable instructions for running the speech synthesis method.

[0144] The network interface 10c may include a wireless network interface or a wired network interface. The network interface 10c is generally used to establish a communication connection between the computer device 10 and other electronic devices.

[0145] The computer device proposed in this embodiment belongs to the field of speech processing technology and is applied to text-to-speech scenarios. This application obtains a synthesized reference speech and text data to be synthesized for speech; inputs the text data into a preset text processor, performs text encoding and decoding processing, and obtains a target text character sequence; inputs the target text character sequence into a trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; performs acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions to obtain the target output speech. By adopting a model prediction method, the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence are predicted, and each text character is decoded one by one to synthesize the target output speech, thereby achieving fast and high-quality speech synthesis output.

[0146] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the speech synthesis method as described above.

[0147] The computer-readable storage medium proposed in this embodiment belongs to the field of speech processing technology and is applied to text-to-speech scenarios. This application obtains a synthesized reference speech and text data to be synthesized for speech; inputs the text data into a preset text processor, performs text encoding and decoding processing, and obtains a target text character sequence; inputs the target text character sequence into a trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; performs acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions to obtain the target output speech. By adopting a model prediction method, the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence are predicted, and each text character is decoded one by one to synthesize the target output speech, thereby achieving fast and high-quality speech synthesis output.

[0148] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0149] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A speech synthesis method, characterized in that: The steps include: Obtaining synthetic reference speech and text data to be synthesized; Input the text data into a preset text processor, perform text encoding and decoding processing, and obtain a target text character sequence, wherein the preset text processor is composed of an N-layer transformer structure, and the position encoding mode of the text processor is set to a rotation position encoding mode, and N is a positive integer greater than 1; Using the synthesized reference speech to learn and train a preset phoneme prediction model to obtain a trained phoneme prediction model; Inputting the target text character sequence into the trained phoneme prediction model to predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; The target text character sequence is acoustically decoded according to the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence to obtain the target output speech.

2. The speech synthesis method according to claim 1, characterized in that: The step of obtaining the synthesized reference speech and the text data to be subjected to speech synthesis specifically comprises: Filtering the speech package of the target object from the reference speech package provided by the target speech synthesis end by screening; Extracting batches of speech segments from the speech packet of the target object as the synthesized reference speech; The text data sent or imported by the speech synthesis request end is obtained as the text data to be subjected to speech synthesis.

3. The speech synthesis method according to claim 1, characterized in that: The step of inputting the text data into a preset text processor, performing text encoding and decoding processing, and obtaining a target text character sequence specifically includes: Using the encoder layer in the transformer structure included in the text processor to convert all text in the text data into a vector representation of a target dimension; Using a rotation position encoding method to add position information to the vector representations corresponding to all the texts to obtain vector representations with embedded position information; Utilizing the multi-head attention mechanism included in the encoder layer, capturing the long-term dependencies and semantic relationships between all texts in the text data from the vector representation of the embedded position information; In combination with the long-term dependencies and semantic relationships among all the texts in the text data, the decoder layer in the transformer structure included in the text processor is used to decode the vector representation of the embedded position information to obtain the target text character sequence.

4. The speech synthesis method according to claim 1, characterized in that: The phoneme prediction model includes a speech-to-text component, an acoustic feature extraction component, and a Viterbi algorithm component. The step of using the synthetic reference speech to learn and train the preset phoneme prediction model to obtain the trained phoneme prediction model specifically includes: The speech-to-text component is used to convert the synthesized reference speech into text, so as to obtain text character sequences corresponding to all speech segments in the synthesized reference speech; The acoustic feature extraction component is used to extract acoustic features of all speech segments in the synthesized reference speech to obtain phoneme features contained in all speech segments, wherein the phoneme features include pronunciation length information, timbre, tone, and pitch corresponding to each phonetic character; Inputting the text character sequences corresponding to all the speech segments and the phoneme features respectively contained in all the speech segments into the Viterbi algorithm component; By using the Viterbi algorithm component, the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments is calculated; The pronunciation length information corresponding to the text characters in all the speech segments in different speech segments is deployed as prediction knowledge information into the preset phoneme prediction model to obtain the trained phoneme prediction model.

5. The speech synthesis method according to claim 4, characterized in that: The phoneme prediction model further includes a prediction result output component, wherein the step of inputting the target text character sequence into the trained phoneme prediction model and predicting the phoneme features and start and end time positions corresponding to each character in the target text character sequence specifically includes: Identifying the speech segment corresponding to the target text character sequence; Determine the timbre, tone, and pitch corresponding to each voice character in the target text character sequence according to the voice segment corresponding to the target text character sequence; Determine the pronunciation length information corresponding to all the text characters in the target text character sequence according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments; Based on the pronunciation length information corresponding to all the text characters in the target text character sequence, predict the pronunciation start and end time position of each character in the target text character sequence in the target output speech; The timbre, tone, pitch corresponding to each voice character in the target text character sequence, and the pronunciation start and end time positions of each character in the target text character sequence in the target output voice are output via the prediction result output component.

6. The speech synthesis method according to claim 5, characterized in that: The step of determining the pronunciation length information corresponding to all the text characters in the target text character sequence according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments specifically includes: Identify all text characters included in the target text character sequence as dynamic programming characters; Generate a pronunciation length information screening domain corresponding to all dynamic programming characters respectively according to the pronunciation length information corresponding to the text characters in all the voice segments in different voice segments; According to the speech segment corresponding to the target text character sequence, a recursive screening method is adopted to sequentially screen out the pronunciation length information corresponding to all the dynamically programmed characters from the pronunciation length information screening domains corresponding to all the dynamically programmed characters; The pronunciation length information corresponding to all the dynamically planned characters is determined as the pronunciation length information corresponding to the corresponding text characters in the target text character sequence.

7. The speech synthesis method according to claim 5, characterized in that: The step of performing acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence to obtain the target output speech specifically includes: Inputting the timbre, tone, and pitch corresponding to each voice character in the target text character sequence, as well as the start and end time positions of pronunciation of each character in the target text character sequence in the target output voice into a preset acoustic decoder, wherein the acoustic decoder includes a diffusion model-like step-by-step decoder; According to the sequence position of each voice character in the target text character sequence, the acoustic decoder is used to gradually perform speech synthesis processing on each voice character to obtain the target output speech.

8. A speech synthesis device, characterized in that: include: A synthesis data acquisition module, used to acquire synthesis reference speech and text data to be synthesized; A text encoding and decoding processing module, used for inputting the text data into a preset text processor, performing text encoding and decoding processing, and obtaining a target text character sequence, wherein the preset text processor is composed of an N-layer transformer structure, and the position encoding mode of the text processor is set to a rotation position encoding mode, and N is a positive integer greater than 1; A phoneme prediction model training module, used to use the synthesized reference speech to learn and train a preset phoneme prediction model to obtain a trained phoneme prediction model; A phoneme prediction model prediction module, used to input the target text character sequence into the trained phoneme prediction model, and predict the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence; The acoustic decoding processing module is used to perform acoustic decoding processing on the target text character sequence according to the phoneme features and pronunciation start and end time positions corresponding to each character in the target text character sequence to obtain the target output speech.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech synthesis method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech synthesis method according to any one of claims 1 to 7 are implemented.

Images