Text-to-speech method, device, computer equipment and storage medium

Abstract

The present invention relates to the field of speech synthesis, and in particular to a text-to-speech method, device, computer equipment and storage medium. The method comprises: obtaining timbre information from a timbre database, obtaining a target text from a client, preprocessing the target text and timbre information, and obtaining first composite information; inputting the first composite information into an attention layer, extracting composite features of the first composite information through an attention network model in the attention layer, and obtaining second composite information output by the attention layer; inputting the second composite information into a timbre classifier, performing timbre recognition on the second composite information through the timbre classifier, and determining whether the second composite information is a target sound; if the second composite information is a target sound, decoding the second composite information through a speech generator, and generating speech information corresponding to the target text. The present invention can make the generated speech information closer to the target sound, and can improve the accuracy of speech synthesis.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis, and in particular to a text-to-speech method, device, computer equipment and storage medium. Background Art

[0002] With the rapid development of the Internet, electronic products such as mobile phones and laptops have become an indispensable part of daily life and work. Staring at the mobile phone screen or computer screen for a long time can easily cause adverse conditions such as eye fatigue and cervical spondylosis.

[0003] Converting text to speech can reduce the time spent staring at a mobile phone or computer screen, thereby reducing adverse conditions such as eye fatigue and cervical spondylosis. In existing technologies for converting text to speech, the neural network obtained through distributed training is prone to bring encoding errors of text information into audio encoding, resulting in low accuracy of text-to-speech. Secondly, in existing technologies, since text encoding and audio encoding are trained simultaneously, it is difficult to synthesize the target sound. Summary of the invention

[0004] Based on this, it is necessary to provide a text-to-speech method, device, computer equipment and storage medium to address the above technical problems, so as to solve the problems in existing speech synthesis technology that the accuracy of text-to-speech is low and it is difficult to synthesize the target sound.

[0005] A text-to-speech method, comprising:

[0006] Acquire timbre information from a timbre database, and acquire a target text input by a user from a client, and pre-process the target text and the timbre information to obtain first composite information;

[0007] Inputting the first composite information into an attention layer, extracting composite features of the first composite information through an attention network model in the attention layer, and obtaining second composite information output by the attention layer;

[0008] inputting the second compound information into a timbre classifier, performing timbre recognition on the second compound information through the timbre classifier, and determining whether the second compound information is a target sound;

[0009] If the second compound information is the target sound, the second compound information is decoded by a speech generator to obtain a mel-spectrogram corresponding to the second compound information, and the mel-spectrogram is inversely processed to generate speech information corresponding to the target text.

[0010] A text-to-speech device, comprising:

[0011] A first composite information module is used to obtain timbre information from a timbre database, obtain a target text input by a user from a client, and pre-process the target text and the timbre information to obtain first composite information;

[0012] A second composite information module is used to input the first composite information into an attention layer, extract composite features of the first composite information through an attention network model in the attention layer, and obtain second composite information output by the attention layer;

[0013] a timbre classifier module, configured to input the second composite information into a timbre classifier, perform timbre recognition on the second composite information through the timbre classifier, and determine whether the second composite information is a target sound;

[0014] The speech information generation module is used to decode the second compound information through a speech generator if the second compound information is a target sound, obtain a Mel-spectrogram corresponding to the second compound information, and inversely process the Mel-spectrogram to generate speech information corresponding to the target text.

[0015] A computer device comprises a memory, a processor and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-mentioned text-to-speech method when executing the computer-readable instructions.

[0016] One or more readable storage media storing computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors execute the text-to-speech method as described above.

[0017] The above-mentioned text-to-speech method, device, computer equipment and storage medium obtain timbre information from a timbre database and obtain a target text input by a user from a client, pre-process the target text and the timbre information to obtain first composite information; input the first composite information into an attention layer, extract the composite features of the first composite information through an attention network model in the attention layer, and obtain second composite information output by the attention layer; input the second composite information into a timbre classifier, perform timbre recognition on the second composite information through the timbre classifier, and determine whether the second composite information is a target sound; if the second composite information is a target sound, decode the second composite information through a speech generator to obtain a mel-spectrogram corresponding to the second composite information, and inversely process the mel-spectrogram to generate speech information corresponding to the target text. The present invention can make the generated speech information closer to the target sound and improve the accuracy of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0019] Figure 1 is a schematic diagram of an application environment of a text-to-speech method in one embodiment of the present invention;

[0020] Figure 2 is a flow chart of a text-to-speech method in one embodiment of the present invention;

[0021] Figure 3 is a structural schematic diagram of a text-to-speech device in one embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The text-to-speech method provided in this embodiment can be applied to Figure 1 In an application environment, a client communicates with a server. The client includes but is not limited to various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0025] In one embodiment, if Figure 2 As shown, a text-to-speech method is provided, and the method is applied in Figure 1 The server in the example is used as an example to illustrate the following steps:

[0026] S10, acquiring timbre information from a timbre database, acquiring a target text input by a user from a client, and preprocessing the target text and the timbre information to obtain first composite information.

[0027] It is understandable that the target text is the text that needs to be converted into speech specified by the user. The target text includes but is not limited to the text typed by the user and the text selected by the user on a certain page. The timbre information is any timbre data in the timbre database. Among them, a timbre data is the information about the timbre of a person. The timbre database contains data of several timbres, that is, the timbre database contains timbre data of multiple people.

[0028] Specifically, any timbre data is obtained from the timbre database, and the target text information specified by the user is obtained. The timbre data and the target text information are preprocessed, that is, the timbre data and the target text information are converted into a voice matrix and a text matrix respectively through an encoder, and then the voice matrix and the text matrix are added and fused to obtain a composite matrix, and the composite matrix is ​​used as the first composite information. Among them, the encoder includes a text encoder and a timbre encoder. The text encoder is used to convert the target text information into a text matrix. The timbre encoder is used to convert the timbre information into a timbre matrix. The text encoder includes a character embedding layer (Character Embedding), a 3-layer convolution and a bidirectional LSTM layer. The timbre encoder can be a pre-set deep network model, for example, DNN.

[0029] S20. Input the first composite information into the attention layer, extract the composite features of the first composite information through the attention network model in the attention layer, and obtain the second composite information output by the attention layer.

[0030] It is understandable that the first composite information includes target text information and timbre information. The attention layer includes an attention network model. Preferably, the attention network model can be a network model based on a hybrid attention mechanism. The attention network model is used to learn the hidden information of the first composite information. That is, the hidden information between the target text information and the timbre information can be learned through the attention layer.

[0031] Specifically, the first composite information is input into the attention network model of the attention layer, and the hidden information in the first composite information is learned through the attention network model. Preferably, each row vector in the first composite information (composite matrix) is combined with the information of its adjacent row vectors, so that each row vector of the first composite information contains the information of other row vectors, and a new composite matrix is ​​generated, which is output from the attention layer as the second composite information. The second composite information integrates the target text information, the timbre information, and the hidden information between the target text information and the timbre information.

[0032] S30, inputting the second compound information into a timbre classifier, performing timbre recognition on the second compound information through the timbre classifier, and determining whether the second compound information is a target sound.

[0033] Understandably, the timbre classifier is a classifier obtained through pre-training, and the timbre classifier is used to identify the timbre of the second composite information and determine whether the second composite information is the target sound. Wherein, the target sound is the timbre that matches the target person. For example, the timbre of a person recorded by the user himself or the timbre of a person selected by the user in the smart client. Preferably, the pre-training of the timbre classifier can be: obtain a number of historical second composite information, select a historical second composite information as the target sound, and mark the historical second composite information as a positive sample (e.g., marked as 1), mark the historical second composite information that is not the target sound as a negative sample (e.g., marked as 0), use the marked several historical second composite information as a sample set, and randomly input the historical second composite information in the sample set into the initial classifier for training. During the training process, if the classification result of the output of the initial classifier is wrong, negative feedback is given to the initial classifier until the classification result of the initial classifier no longer changes or tends to be stable, and the initial classifier when it no longer changes or tends to be stable is used as the timbre classifier.

[0034] Specifically, the second compound information is input into the timbre classifier, and the second compound information is identified by the timbre classifier to determine whether the second compound information is the target sound. When the second compound information is the target sound, the second compound information is input into the speech generator to generate speech information corresponding to the target text information. If the second compound information is not the target sound, feedback information is generated according to the second compound information, and the feedback information is sent to the speech generator, and the speech generator performs negative feedback on the second compound information, so that after receiving the feedback information from the timbre classifier, the speech generator gradually reduces the processing of the second compound information, so that the generated speech information is closer to the target sound, and the accuracy of speech synthesis can be improved.

[0035] S40: If the second compound information is a target sound, the second compound information is decoded by a speech generator to obtain a mel-spectrogram corresponding to the second compound information, and the mel-spectrogram is inversely processed to generate speech information corresponding to the target text.

[0036] It is understandable that the sound signal is originally a one-dimensional time domain signal. The short-time Fourier transform (STFT) is used to perform Fourier transform on the short-time sound signal. A long signal is divided into frames, windowed, and then each frame is Fourier transformed (FFT). Finally, the results of each frame are stacked along another dimension to obtain a two-dimensional signal form similar to a picture, that is, a spectrogram. The spectrogram is often a large picture. In order to obtain the sound features of the appropriate size, it is often transformed into a mel spectrum, that is, a mel spectrogram, through a mel-scale filter bank.

[0037] It can be understood that the target voice is the timbre that matches the target person. For example, the target voice is the timbre of Li San. The speech generator is a pre-trained decoder. The speech generator is used to decode the second composite information, decode the second composite information into a mel-spectrogram corresponding to the target text, and perform inverse processing on the mel-spectrogram through a preset transformation rule to generate speech information corresponding to the target text. Among them, the speech information is the speech expression of the target text. For example, if the target text is "I am Chinese", the speech information is the voice of "I am Chinese", and the timbre of the speech information is the target voice.

[0038] Preferably, the pre-training of the speech generator is as follows: obtaining a number of speech data as training samples, wherein the speech data includes a text sample, a timbre information sample, and a mel-spectrogram sample. The text sample and the timbre information sample are pre-processed to obtain a second composite information sample. The training sample is input into the initial speech generator, and the second composite information sample is decoded by the initial speech generator, and the sample second composite information is decoded into a mel-spectrogram. Wherein, the initial speech generator includes initial parameters. During the training process, the decoded mel-spectrogram is compared with the mel-spectrogram sample, and when there is a difference between the decoded mel-spectrogram and the mel-spectrogram sample, the difference is fed back to the initial speech generator as negative feedback, so that the initial speech generator adjusts the initial parameters until the difference between the decoded mel-spectrogram and the mel-spectrogram sample is less than a preset threshold, and the initial speech generator corresponding to the difference less than the preset threshold is used as the speech generator.

[0039] In steps S10-S40, by obtaining timbre information from a timbre database and obtaining a target text input by a user from a client, the target text and the timbre information are preprocessed to obtain first composite information; the first composite information is input into an attention layer, and the composite features of the first composite information are extracted through an attention network model in the attention layer to obtain second composite information output by the attention layer; the second composite information is input into a timbre classifier, and the timbre classifier is used to perform timbre recognition on the second composite information to determine whether the second composite information is a target sound; if the second composite information is a target sound, the second composite information is decoded through a speech generator to obtain a Mel-spectrogram corresponding to the second composite information, and the Mel-spectrogram is inversely processed to generate speech information corresponding to the target text. The present invention can make the generated speech information closer to the target sound and improve the accuracy of speech synthesis.

[0040] Optionally, in step S10, the target text and the timbre information are preprocessed to obtain first composite information, including:

[0041] S101. Input the target text into a text encoder, and convert the target text into a text matrix through the text encoder;

[0042] S102. Input the timbre information into a preset deep network model, and convert the timbre information into a timbre matrix through the preset deep network model;

[0043] S103. Embed the timbre matrix into the text matrix to obtain first composite information.

[0044] It can be understood that the text encoder is used to convert target text information into a text matrix. The text encoder includes a Character Embedding layer, a three-layer convolution, and a bidirectional LSTM layer. The input target text is encoded into a 512-dimensional character vector, and then passes through a three-layer convolution. Each convolution layer contains 512 5×1 convolutional kernels, that is, each convolutional kernel spans 5 characters. The convolutional layer will perform large-span context modeling on the input character sequence (similar to N-grams). After the convolutional layer, batch normalization is performed, and ReLu (activation function) is used for activation; the output of the last convolutional layer is sent to a bidirectional LSTM (long short term memory) layer to generate encoded features and obtain the text matrix.

[0045] Among them, the process of encoding the target text into a 512-dimensional character vector includes decomposing the target text into corresponding phonemes by using a phoneme correspondence table, converting the phonemes into numbers according to a phoneme lookup table, and then converting the numbers into 512-dimensional character vectors. Among them, a phoneme is the smallest speech unit. For example, the pronunciation "fā" of the character "发" is a syllable and can be decomposed into two phonemes, f and a. The phoneme lookup table is a table containing the numbers corresponding to all phonemes. For example, in the phoneme lookup table, the number corresponding to f is 10.

[0046] It can be understood that the timbre encoder is used to convert timbre information into a timbre matrix. The timbre encoder can be a preset deep network model, such as a DNN (Deep Neural Network). The timbre information is processed through the deep network model to obtain the timbre matrix.

[0047] Specifically, after obtaining the text matrix corresponding to the target text and the timbre matrix corresponding to the timbre information, the timbre matrix is embedded into the text matrix to obtain a composite matrix. This composite matrix contains the features of the target text and the timbre information, and this composite matrix is determined as the first composite information.

[0048] In S101-S103, the target text and timbre information are converted into a text matrix and a timbre matrix respectively, and then the timbre matrix is ​​embedded into the target matrix, so as to embed the timbre information into the target text, which can improve the accuracy of text-to-speech.

[0049] Optionally, in step S10, the preprocessing of the target text and the timbre information further includes:

[0050] S104, receiving a target language setting instruction;

[0051] S105, setting the language type according to the target language setting instruction to obtain a target language, and storing the target language in a memory;

[0052] S106: Convert the original text into the target text corresponding to the target language.

[0053] Understandably, the target language setting instruction is an instruction for setting the target language. The target language setting instruction is an instruction generated after the user selects a certain language as the target language on the smart client. For example, the languages ​​that the user can select on the smart client are Chinese, English, French and Italian. After the user selects Chinese as the target language, a target language setting instruction is generated. According to the target language setting instruction, the language type is set to Chinese, that is, Chinese is the target language, and Chinese is stored in the memory as the target language. As the default language, the user does not need to select the target language every time, which can improve the user experience. After determining the target language, the characters in the original text are converted into characters corresponding to the target language. For example, when the input original text is in English and the target language is Chinese, the original text is translated into Chinese text to obtain the target text. The present invention can generate voice information for texts in multiple languages, has strong applicability, and can improve the user experience.

[0054] Optionally, in step S20, that is, inputting the first composite information into the attention layer, extracting composite features of the first composite information through the attention network model in the attention layer, and obtaining second composite information output by the attention layer, including:

[0055] S201, inputting the first composite information into the attention layer;

[0056] S202. Fusing the contextual semantic information in the first compound information through the attention network model, extracting the compound features of the first compound information, and obtaining the second compound information output by the attention layer.

[0057] It is understandable that the attention layer includes an attention network model. Preferably, the attention network model can be a network model based on a hybrid attention mechanism. The attention network model is used to learn and extract the composite features of the first composite information. The composite features include contextual semantic information. That is, the contextual semantic information of the first composite information can be learned through the attention layer.

[0058] Specifically, the first composite information is input into the attention network model of the attention layer, and the contextual semantic information in the first composite information is learned through the attention network model. Preferably, each row vector in the first composite information (composite matrix) is combined with the information of its adjacent row vectors, so that each row vector of the first composite information contains the information of other row vectors, and a new composite matrix is ​​generated, which is output from the attention layer as the second composite information. The second composite information integrates the contextual semantic information of the first composite text, which can improve the accuracy of text-to-speech conversion.

[0059] Optionally, in step S40, if the second compound information is the target sound, the second compound information is decoded by a speech generator to obtain a Mel-spectrogram corresponding to the second compound information, and the Mel-spectrogram is inversely processed to generate speech information corresponding to the target text, including:

[0060] S401, when the second composite information is a target sound, decoding the second composite information into a mel-spectrogram by a vocoder based on a neural network;

[0061] S402: Perform inverse processing on the mel-spectrogram according to a preset transformation rule to generate speech information corresponding to the target text.

[0062] It is understandable that the sound signal is originally a one-dimensional time domain signal. The short-time Fourier transform (STFT) is used to perform Fourier transform on the short-time sound signal. A long signal is divided into frames and windows, and then a Fourier transform (FFT) is performed on each frame. Finally, the results of each frame are stacked along another dimension to obtain a two-dimensional signal form similar to a picture, that is, a spectrogram. The spectrogram is often a very large picture. In order to obtain the sound features of the appropriate size, it is often transformed into a Mel spectrum, that is, a Mel spectrogram, through a Mel-scale filter bank. The speech generator includes a vocoder based on a neural network. Among them, the vocoder can be a speech synthesizer based on neural networks such as wavenet (sequence generation model), wavernn (single-layer recurrent neural network) and lpcnet (Linear Prediction, linear prediction neural network).

[0063] It can be understood that the target sound is a timbre of a sound specified by the user. The vocoder is a decoder obtained through pre-training. The second composite information is decoded by the vocoder, and the second composite information is decoded into a mel-spectrogram corresponding to the target text. Then, the mel-spectrogram is inversely processed through a preset transformation rule to generate speech information corresponding to the target text.

[0064] Among them, the preset transformation rule is a pre-set transformation rule. The transformation rule includes but is not limited to inverse Fourier transform (IFFT) and inverse discrete cosine transform. Preferably, the transformation rule can be an inverse Fourier transform (IFFT). Through IFFT, the FFT signals of frames are transformed into small segments of time domain signals, and then they are spliced ​​together to obtain sound signals, that is, voice information. Inverse processing refers to the processing process of converting the Mel spectrogram into a sound signal. Specifically, the Mel spectrogram is transformed into a spectrogram through a Mel scale filter group, and then the FFT signal of the spectrogram is transformed into small segments of time domain signals through an inverse Fourier transform (IFFT), and then they are spliced ​​together to obtain a sound signal, that is, voice information.

[0065] Optionally, after step S30, that is, after the timbre classifier performs timbre recognition on the second composite information to determine whether the second composite information is a target sound, the following steps are included:

[0066] S301: if the second composite information is not a target sound, generate feedback information according to the second composite information;

[0067] S302: Send the feedback information to the speech generator, so that the speech generator reduces processing of the second composite information.

[0068] It can be understood that the target sound is a timbre that matches the target. The feedback information is used to indicate that the second composite information is not the target sound.

[0069] Specifically, if the second compound information is not the target sound, feedback information is generated based on the second compound information, and the feedback information is sent to the speech generator. After receiving the feedback information, the speech generator performs negative feedback on the second compound information, so that after receiving the feedback information from the timbre classifier, the speech generator gradually reduces the processing of the second compound information, making the generated speech information closer to the target sound, thereby improving the accuracy of speech synthesis.

[0070] Optionally, after step S30, that is, after performing timbre recognition on the second composite information by the timbre classifier to determine whether the second composite information is a target sound, the method further includes:

[0071] S303: If the second composite information is not the target sound, new timbre information is obtained, and the target text and the new timbre information are preprocessed to obtain new first composite information.

[0072] It is understandable that if the second composite information is not the target sound, the timbre information corresponding to the second composite information is discarded, and a new timbre data is randomly obtained from the timbre database, and the new timbre data is used as the new timbre information. The new timbre information is any timbre information other than the timbre information corresponding to the second composite information.

[0073] Specifically, a new timbre information is randomly obtained from the timbre database, the timbre data and the target text information are preprocessed to obtain a composite matrix, and the composite matrix is ​​used as the new first composite information.

[0074] S304: Input the new first composite information into the attention layer to obtain new second composite information output by the attention layer.

[0075] It is understandable that the first composite information includes the target text information and the new timbre information. The attention network model is used to learn the hidden information of the new first composite information. That is, the hidden information between the target text information and the new timbre information can be learned through the attention layer.

[0076] Specifically, the new first composite information is input into the attention network model of the attention layer, and the hidden information in the new first composite information is learned through the attention network model. Preferably, each row vector in the new first composite information (composite matrix) is combined with the information of its adjacent row vectors, so that each row vector of the new first composite information contains the information of other row vectors, and a new composite matrix is ​​generated, and the new composite matrix is ​​output from the attention layer as the new second composite information. The new second composite information integrates the target text information, the new timbre information, and the hidden information between the target text information and the new timbre information.

[0077] S305: Input the new second composite information into a timbre classifier, and determine whether the new second composite information is a target sound through the timbre classifier.

[0078] Specifically, the new second compound information is input into the timbre classifier, and the timbre classifier identifies the new second compound information to determine whether the new second compound information is the target sound. When the new second compound information is the target sound, the new second compound information is input into the speech generator to generate speech information corresponding to the target text information. If the new second compound information is not the target sound, feedback information is generated according to the new second compound information.

[0079] S306: If the new second compound information is a target sound, the new second compound information is processed by a speech generator to generate speech information corresponding to the target text.

[0080] Specifically, if the new second compound information is the target sound, the new second compound information is decoded by the speech generator, and the new second compound information is decoded into a mel-spectrogram corresponding to the target text. The mel-spectrogram is inversely processed according to a preset transformation rule to generate speech information corresponding to the target text. If the new second compound information is not the target sound, feedback information is generated according to the new second compound information, and the feedback information is sent to the speech generator. After receiving the feedback information, the speech generator performs negative feedback on the new second compound information, so that after receiving the feedback information from the timbre classifier, the speech generator gradually reduces the processing of the new second compound information, so that the generated speech information is closer to the target sound, and the accuracy of speech synthesis can be improved.

[0081] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0082] In one embodiment, a text-to-speech device is provided, which corresponds one-to-one to the text-to-speech method in the above embodiment. Figure 3 As shown, the text-to-speech device includes a first composite information module 10, a second composite information module 20, a timbre classifier module 30 and a speech information generation module 40. Each functional module is described in detail as follows:

[0083] The first composite information module 10 is used to obtain timbre information from a timbre database, obtain a target text input by a user from a client, and pre-process the target text and the timbre information to obtain first composite information;

[0084] The second composite information module 20 is used to input the first composite information into the attention layer, extract the composite features of the first composite information through the attention network model in the attention layer, and obtain the second composite information output by the attention layer;

[0085] The timbre classifier module 30 is used to input the second composite information into a timbre classifier, perform timbre recognition on the second composite information through the timbre classifier, and determine whether the second composite information is a target sound;

[0086] The speech information generating module 40 is used for, if the second compound information is the target sound, decoding the second compound information through a speech generator to obtain a Mel-spectrogram corresponding to the second compound information, and inversely processing the Mel-spectrogram to generate speech information corresponding to the target text.

[0087] Optionally, the first composite information module 10 includes:

[0088] A text matrix unit, used for inputting the target text into a text encoder, and converting the target text into a text matrix through the text encoder;

[0089] A timbre matrix unit, used for inputting the timbre information into a preset deep network model, and converting the timbre information into a timbre matrix through the preset deep network model;

[0090] The first composite information unit is used to embed the timbre matrix into the text matrix to obtain the first composite information.

[0091] Optionally, the first composite information module 10 further includes:

[0092] An instruction receiving unit, used for receiving an instruction for setting a target language;

[0093] A target language unit, configured to set the language type according to the target language setting instruction, obtain the target language, and store the target language in a memory;

[0094] The target language text unit is used to convert the original text into the target text corresponding to the target language.

[0095] Optionally, the second composite information module 20 includes:

[0096] A composite information input unit, used for inputting the first composite information into the attention layer;

[0097] The composite information generating unit is used to fuse the contextual semantic information in the first composite information through the attention network model, extract the composite features of the first composite information, and obtain the second composite information output by the attention layer.

[0098] Optionally, the voice information generating module 40 includes:

[0099] A mel-spectrogram unit, configured to decode the second composite information into a mel-spectrogram by a neural network-based vocoder when the second composite information is a target sound;

[0100] The speech information generating unit is used to perform inverse processing on the mel-spectrogram according to a preset transformation rule to generate speech information corresponding to the target text.

[0101] Optionally, after the first timbre classifier module 30, it includes:

[0102] a feedback information generating unit, configured to generate feedback information according to the second composite information if the second composite information is not a target sound;

[0103] The feedback information sending unit is used to send the feedback information to the speech generator, so that the speech generator reduces the processing of the second composite information.

[0104] Optional,

[0105] The first composite information module is further configured to obtain new timbre information if the second composite information is not the target sound, and pre-process the target text and the new timbre information to obtain new first composite information;

[0106] The second composite information module is further used to input the new first composite information into the attention layer to obtain new second composite information output by the attention layer;

[0107] A timbre classifier module, used for inputting the new second composite information into a timbre classifier, and determining whether the new second composite information is a target sound through the timbre classifier;

[0108] The voice information generating module is used for, if the new second compound information is a target sound, processing the new second compound information through a voice generator to generate voice information corresponding to the target text.

[0109] For the specific definition of the text-to-speech device, please refer to the definition of the text-to-speech method above, which will not be repeated here. Each module in the above-mentioned text-to-speech device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0110] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium and an internal memory. The readable storage medium stores an operating system, computer-readable instructions and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The database of the computer device is used to store data involved in the text-to-speech method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer-readable instructions are executed by the processor, a text-to-speech method is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0111] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor executes the computer-readable instructions to implement the following steps:

[0112] Acquire timbre information from a timbre database, and acquire a target text input by a user from a client, and pre-process the target text and the timbre information to obtain first composite information;

[0113] Inputting the first composite information into an attention layer, extracting composite features of the first composite information through an attention network model in the attention layer, and obtaining second composite information output by the attention layer;

[0114] inputting the second compound information into a timbre classifier, performing timbre recognition on the second compound information through the timbre classifier, and determining whether the second compound information is a target sound;

[0115] If the second compound information is the target sound, the second compound information is decoded by a speech generator to obtain a mel-spectrogram corresponding to the second compound information, and the mel-spectrogram is inversely processed to generate speech information corresponding to the target text.

[0116] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media store computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the following steps are implemented:

[0117] Acquire timbre information from a timbre database, and acquire a target text input by a user from a client, and pre-process the target text and the timbre information to obtain first composite information;

[0118] Inputting the first composite information into an attention layer, extracting composite features of the first composite information through an attention network model in the attention layer, and obtaining second composite information output by the attention layer;

[0119] inputting the second compound information into a timbre classifier, performing timbre recognition on the second compound information through the timbre classifier, and determining whether the second compound information is a target sound;

[0120] If the second compound information is the target sound, the second compound information is decoded by a speech generator to obtain a mel-spectrogram corresponding to the second compound information, and the mel-spectrogram is inversely processed to generate speech information corresponding to the target text.

[0121] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0122] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0123] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A text-to-speech method, characterized in that: include: Acquire timbre information from a timbre database, and acquire a target text input by a user from a client, and pre-process the target text and the timbre information to obtain first composite information; Inputting the first composite information into an attention layer, extracting composite features of the first composite information through an attention network model in the attention layer, and obtaining second composite information output by the attention layer; inputting the second compound information into a timbre classifier, performing timbre recognition on the second compound information through the timbre classifier, and determining whether the second compound information is a target sound; If the second composite information is the target sound, the second composite information is decoded by a speech generator to obtain a mel-spectrogram corresponding to the second composite information, and the mel-spectrogram is inversely processed to generate speech information corresponding to the target text; After the timbre classifier is used to identify the timbre of the second composite information and determines whether the second composite information is a target sound, the method further includes: If the second composite information is not the target sound, new timbre information is obtained, and the target text and the new timbre information are preprocessed to obtain new first composite information; Inputting the new first composite information into the attention layer to obtain new second composite information output by the attention layer; Inputting the new second composite information into a timbre classifier, and determining whether the new second composite information is a target sound by the timbre classifier; If the new second compound information is the target sound, the new second compound information is processed by a speech generator to generate speech information corresponding to the target text.

2. The text-to-speech method according to claim 1, wherein: The preprocessing of the target text and the timbre information to obtain the first composite information includes: Inputting the target text into a text encoder, and converting the target text into a text matrix by the text encoder; Inputting the timbre information into a preset deep network model, and converting the timbre information into a timbre matrix through the preset deep network model; The timbre matrix is ​​embedded in the text matrix to obtain first composite information.

3. The text-to-speech method according to claim 1, wherein: The preprocessing of the target text and the timbre information further includes: receiving a target language setting instruction; Setting the language type according to the target language setting instruction to obtain the target language, and storing the target language in a memory; The original text is converted into the target text corresponding to the target language.

4. The text-to-speech method according to claim 1, wherein: The step of inputting the first composite information into the attention layer, extracting composite features of the first composite information through the attention network model in the attention layer, and obtaining second composite information output by the attention layer includes: Inputting the first composite information into the attention layer; The contextual semantic information in the first composite information is fused through the attention network model, the composite features of the first composite information are extracted, and the second composite information output by the attention layer is obtained.

5. The text-to-speech method according to claim 1, wherein: If the second composite information is the target sound, the second composite information is decoded by a speech generator to obtain a mel-spectrogram corresponding to the second composite information, and the mel-spectrogram is inversely processed to generate speech information corresponding to the target text, including: When the second composite information is a target sound, decoding the second composite information into a mel-spectrogram by a vocoder based on a neural network; The Mel-spectrogram is inversely processed by preset transformation rules to generate speech information corresponding to the target text.

6. The text-to-speech method according to claim 1, wherein: After the timbre classifier performs timbre recognition on the second composite information and determines whether the second composite information is a target sound, the method includes: If the second composite information is not the target sound, generating feedback information according to the second composite information; The feedback information is sent to the speech generator so that the speech generator reduces processing of the second composite information.

7. A text-to-speech device, characterized in that: include: A first composite information module is used to obtain timbre information from a timbre database, obtain a target text input by a user from a client, and pre-process the target text and the timbre information to obtain first composite information; A second composite information module is used to input the first composite information into an attention layer, extract composite features of the first composite information through an attention network model in the attention layer, and obtain second composite information output by the attention layer; a timbre classifier module, configured to input the second composite information into a timbre classifier, perform timbre recognition on the second composite information through the timbre classifier, and determine whether the second composite information is a target sound; A speech information generating module, configured to decode the second compound information through a speech generator to obtain a mel-spectrogram corresponding to the second compound information if the second compound information is a target sound, and to inversely process the mel-spectrogram to generate speech information corresponding to the target text; After the timbre classifier is used to identify the timbre of the second composite information and determines whether the second composite information is a target sound, the method further includes: If the second composite information is not the target sound, new timbre information is obtained, and the target text and the new timbre information are preprocessed to obtain new first composite information; Inputting the new first composite information into the attention layer to obtain new second composite information output by the attention layer; Inputting the new second composite information into a timbre classifier, and determining whether the new second composite information is a target sound by the timbre classifier; If the new second compound information is the target sound, the new second compound information is processed by a speech generator to generate speech information corresponding to the target text.

8. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the computer-readable instructions, the text-to-speech method according to any one of claims 1 to 6 is implemented.

9. One or more readable storage media storing computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors execute the text-to-speech method according to any one of claims 1 to 6.

Images