Speech synthesis method and device, equipment and storage medium

Abstract

The invention provides a speech synthesis method, device and equipment and a storage medium, and belongs to the field of speech synthesis, and the method comprises the steps: obtaining a target text, a reference audio and a preset speech synthesis model; and obtaining a target voice corresponding to the target text according to the target text and the reference audio based on a preset voice synthesis model. According to the method, speech synthesis of whole-sentence-level emotion embedding and frame-level deep emotion embedding is performed on the target text and the reference audio through each level in the preset speech synthesis model, so that the synthesized speech has better tone quality, naturalness and emotion, and the speech synthesis accuracy is greatly improved. The preset speech synthesis model can be stored in a block chain.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and in particular to a speech synthesis method, apparatus, device and storage medium. Background Art

[0002] With the development of speech synthesis models, the quality of synthesized speech is constantly improving, and the similarity of speakers is getting higher and higher. However, speech synthesis has a one-to-many problem in which a text can map multiple speech sounds. Simply synthesizing the target speaker's voice is no longer enough. We need more methods and means to regulate the entire model to make the synthesized speech more controllable.

[0003] In response to this, speech synthesis based on emotion transfer came into being. By inputting the form of speech, the model is forced to synthesize speech that conforms to the current emotional form rather than arbitrarily synthesizing a learned emotion. In the past, speech synthesis models for emotion transfer often used a single acoustic feature such as Mel spectrum to predict emotional information, but the input features of this method were not sufficient and did not take into account the speaker's emotions when speaking, resulting in the synthesized speech having no melody changes and low expressiveness.

[0004] Therefore, how to synthesize speech with emotion embedding is a problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of this application is to provide a speech synthesis method, apparatus, device and storage medium, aiming to improve the emotional naturalness and accuracy of synthesized speech.

[0006] In a first aspect, the present application provides a speech synthesis method, the speech synthesis method comprising the following steps:

[0007] Acquire a target text, a reference audio, and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer, and a decoder layer, wherein the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder, and an embedding fusion module;

[0008] Perform speaker feature recognition on the reference audio by the speaker encoder to obtain a speaker embedding vector, and perform sentence-level emotion embedding recognition on the reference audio by the sentence-level audio encoder to obtain a first emotion vector;

[0009] Performing frame-level deep emotion embedding recognition on the reference audio by the frame-level audio encoder to obtain a second emotion vector;

[0010] Performing vector fusion on the speaker embedding vector, the first emotion vector and the second emotion vector through the embedding fusion module to obtain an emotion embedding vector;

[0011] Performing feature encoding on the target text through the text encoding layer to obtain a text embedding vector, and performing speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector;

[0012] The speech audio vector is outputted via the decoder layer to obtain a target speech corresponding to the target text.

[0013] In a second aspect, the present application further provides a speech synthesis device, the speech synthesis device comprising an acquisition module and a generation module, wherein:

[0014] The acquisition module is used to acquire the target text, the reference audio and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer and a decoder layer, and the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder and an embedding fusion module;

[0015] The generating module is used for:

[0016] Perform speaker feature recognition on the reference audio through a speaker encoder to obtain a speaker embedding vector, and perform sentence-level emotion embedding recognition on the reference audio through a sentence-level audio encoder to obtain a first emotion vector;

[0017] Performing frame-level deep emotion embedding recognition on the reference audio through a frame-level audio encoder to obtain a second emotion vector;

[0018] Performing vector fusion on the speaker embedding vector, the first emotion vector and the second emotion vector through the embedding fusion module to obtain an emotion embedding vector;

[0019] Performing feature encoding on the target text through the text encoding layer to obtain a text embedding vector, and performing speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector;

[0020] The speech audio vector is outputted via the decoder layer to obtain a target speech corresponding to the target text.

[0021] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the speech synthesis method as described above are implemented.

[0022] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the speech synthesis method as described above are implemented.

[0023] The present application provides a speech synthesis method, apparatus, device and storage medium. The present application obtains a target text, a reference audio and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer and a decoder layer, and the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder and an embedding fusion module; the speaker feature recognition is performed on the reference audio by the speaker encoder to obtain a speaker embedding vector, and the whole sentence-level emotion embedding recognition is performed on the reference audio by the sentence-level audio encoder to obtain a first emotion vector; the frame-level deep emotion embedding recognition is performed on the reference audio by the frame-level audio encoder to obtain a second emotion vector; the speaker embedding vector, the first emotion vector and the second emotion vector are vector-fused by the embedding fusion module to obtain an emotion embedding vector; the target text is feature-encoded by the text coding layer to obtain a text embedding vector, and the text embedding vector and the emotion embedding vector are subjected to speech audio synthesis by the speech prediction layer to generate a speech audio vector; the speech audio vector is speech-outputted by the decoder layer to obtain a target speech corresponding to the target text. In this application, speech synthesis is performed by performing sentence-level emotion embedding and frame-level deep emotion embedding on the target text and reference audio through various levels in the preset speech synthesis model, so that the synthesized speech has better sound quality, naturalness and emotion, greatly improving the accuracy of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A flowchart of a speech synthesis method provided in an embodiment of the present application;

[0026] Figure 2 A schematic block diagram of a speech synthesis model provided in an embodiment of the present application;

[0027] Figure 3 A flowchart of another speech synthesis method provided in an embodiment of the present application;

[0028] Figure 4 A schematic block diagram of a speech synthesis device provided in an embodiment of the present application;

[0029] Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0030] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0033] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0034] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0035] With the development of speech synthesis models, the quality of synthesized speech is constantly improving, and the similarity of speakers is getting higher and higher. However, speech synthesis has a one-to-many problem in which one text can map multiple speech sounds. Simply synthesizing the target speaker's voice is no longer enough. We need more methods and means to regulate the entire model to make the synthesized speech more controllable.

[0036] In response to this, speech synthesis based on emotion transfer came into being. By inputting the form of speech, the model is forced to synthesize speech that conforms to the current emotional form rather than arbitrarily synthesizing a learned emotion. In the past, speech synthesis models for emotion transfer often used a single acoustic feature such as Mel spectrum to predict emotional information, but the input features of this method were not sufficient and did not take into account the speaker's emotions when speaking, resulting in the synthesized speech having no melody changes and low expressiveness.

[0037] To solve the above problems, the embodiments of the present application provide a speech synthesis method, device, equipment and storage medium. The speech synthesis method includes: obtaining a target text, a reference audio and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer and a decoder layer, wherein the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder and an embedding fusion module; performing speaker feature recognition on the reference audio through the speaker encoder to obtain a speaker embedding vector, and performing sentence-level emotion embedding recognition on the reference audio through the sentence-level audio encoder to obtain a first emotion vector; performing frame-level deep emotion embedding recognition on the reference audio through the frame-level audio encoder to obtain a second emotion vector; performing vector fusion on the speaker embedding vector, the first emotion vector and the second emotion vector through the embedding fusion module to obtain an emotion embedding vector; performing feature encoding on the target text through the text coding layer to obtain a text embedding vector, and performing speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector; performing speech output on the speech audio vector through the decoder layer to obtain a target speech corresponding to the target text.

[0038] Among them, the speech synthesis method can be applied to computer devices, and the computer devices can be electronic devices such as mobile phones, tablet computers, laptops, desktop computers, personal digital assistants and wearable devices.

[0039] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0040] Please refer to Figure 1 , Figure 1 A flowchart of a speech synthesis method provided in an embodiment of the present application.

[0041] like Figure 1 As shown, the speech synthesis method includes steps S101 to S106.

[0042] Step S101, obtaining a target text, a reference audio and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer and a decoder layer, and the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder and an embedded fusion module.

[0043] like Figure 2 As shown, the preset speech synthesis model 100 includes an audio coding layer 10, a text coding layer 20, a speech prediction layer 30 and a decoder layer 40. The audio coding layer 10 includes a speaker encoder 11, a sentence-level audio encoder 12, a frame-level audio encoder 13 and an embedding fusion module 14. The speech synthesis model is a neural network model. The type of the neural network model can be selected according to actual conditions. The embodiment of the present invention does not specifically limit this. For example, the neural network model includes but is not limited to a convolutional neural network model and a recurrent convolutional neural network model.

[0044] In some embodiments, Figure 3 As shown, the present invention also provides another speech synthesis method, which includes steps S201 to S203.

[0045] Step S201: Acquire a sample data set, where the sample data set includes a plurality of sample data, and the sample data includes a sample text, a sample reference audio, and a reference speech identified by the sample text.

[0046] The sample data set includes a plurality of sample data, including sample text, sample reference audio, and reference speech corresponding to the sample text. The sample data is pre-established based on reference audio, text, and reference speech, and the sample data is constructed by repeatedly executing the steps of collecting reference audio, text, and reference speech until a large amount of sample data is obtained to construct the sample data set.

[0047] Step S202: Select a sample data from the sample data set as target sample data, train a preset speech synthesis model according to the target sample data, and determine whether the preset speech synthesis model converges.

[0048] Among them, the method of selecting sample data can be set according to actual conditions, and the embodiment of the present invention does not make specific limitations on this. For example, a sample data can be randomly selected as the target sample data, or the target sample data can be selected in the order of the time when the sample data enters the data set.

[0049] In some embodiments, speaker feature recognition is performed on sample reference audio through a speaker encoder to obtain a predicted speaker embedding vector, and whole sentence level emotion embedding recognition is performed on the sample reference audio through a sentence-level audio encoder to obtain a predicted first emotion vector; frame-level deep emotion embedding recognition is performed on the sample reference audio through a frame-level audio encoder to obtain a predicted second emotion vector; the predicted speaker embedding vector, the predicted first emotion vector and the predicted second emotion vector are vector fused through an embedding fusion module to obtain a predicted emotion embedding vector; feature encoding is performed on the sample text through a text encoding layer to obtain a predicted text embedding vector, and speech audio synthesis is performed on the predicted text embedding vector and the predicted emotion embedding vector through a speech prediction layer to generate a predicted speech audio vector; speech output is performed on the predicted speech audio vector through a decoder layer to obtain a predicted speech corresponding to the sample text; and whether a preset speech synthesis model converges is determined based on the identified reference speech and the predicted speech.

[0050] In some embodiments, the method of performing sentence-level emotion embedding recognition on the sample reference audio by a sentence-level audio encoder to obtain the predicted first emotion vector can be: performing overall emotion embedding recognition on each sentence in the sample reference audio by the sentence-level audio encoder to obtain the predicted first emotion vector. By recognizing the sample reference audio by a sentence-level audio encoder, the predicted first emotion vector of the entire sentence of the sample reference audio can be accurately obtained.

[0051] In some embodiments, the method of performing frame-level deep emotion embedding recognition on the sample reference audio through a frame-level audio encoder to obtain a predicted second emotion vector can be: segmenting the sample reference audio at a preset frame granularity to obtain multiple sample reference sub-audios; performing emotion embedding recognition on each sample reference sub-audio through a frame-level audio encoder to obtain multiple predicted sub-emotion vectors; fusing the multiple predicted sub-emotion vectors to generate a sample second emotion vector, and the predicted sub-emotion vectors in the sample second emotion vector are arranged along the time axis.

[0052] In some embodiments, the predicted speaker embedding vector, the predicted first emotion vector and the predicted second emotion vector are vector-fused by an embedding fusion module to obtain the predicted emotion embedding vector in the following manner: the embedding fusion module inserts each predicted sub-emotion vector in the predicted second emotion vector into the corresponding time position in the predicted first emotion vector according to the time axis to adjust the emotion embedding information represented by the predicted first emotion vector to obtain the predicted third emotion vector; the predicted speaker embedding vector and the predicted third emotion vector are vector-fused to obtain the predicted emotion embedding vector.

[0053] In some embodiments, speech audio synthesis is performed on the predicted text embedding vector and the predicted emotion embedding vector through a speech prediction layer, and the method for generating the predicted speech audio vector can be: speech generation is performed on the predicted text embedding vector through the prediction layer to obtain a predicted candidate speech audio vector; the phonemes represented by the predicted candidate speech audio vector are adjusted according to the predicted emotion embedding vector to generate a predicted speech audio vector.

[0054] In some embodiments, the method for determining whether the preset speech synthesis model has converged based on the identified reference speech and the predicted speech can be: determining the loss value of the preset speech synthesis model based on the identified reference speech and the predicted speech; if the loss value is greater than or equal to the preset loss value, determining that the preset speech synthesis model has not converged; if the loss value is less than the preset loss value, determining that the preset speech synthesis model has converged. Among them, the preset loss value can be set according to actual conditions, and the embodiment of the present invention does not make specific limitations on this. For example, the preset loss value can be set to 0.002. By calculating the loss value of the speech synthesis model, it is possible to accurately determine whether the model has converged, which greatly improves the stability of the trained model.

[0055] In some embodiments, the loss value of the preset speech synthesis model can be determined based on the identified reference speech and the predicted speech by calculating the Euclidean distance between the identified reference speech and the predicted speech, and determining the value obtained by subtracting the Euclidean distance from unit 1 as the loss value of the preset speech synthesis model. Of course, the cosine similarity between the identified reference speech and the predicted speech can also be calculated, and the value obtained by subtracting the cosine similarity from unit 1 is determined as the loss value of the preset speech synthesis model.

[0056] Step S203: If the preset speech synthesis model has not converged, adjust the model parameters of the preset speech synthesis model, and loop through the step of selecting a sample data from the sample data set as the target sample data until the preset speech synthesis model converges to obtain a speech synthesis model.

[0057] If the preset speech synthesis model has not converged, adjust the model parameters of the preset speech synthesis model, and cyclically select a sample data from the sample data set as the target sample data, train the preset speech synthesis model according to the target sample data, and determine whether the preset speech synthesis model has converged; if the preset speech synthesis model has not converged, adjust the model parameters of the preset speech synthesis model until the preset speech synthesis model converges to obtain the speech synthesis model.

[0058] Step S102: performing speaker feature recognition on the reference audio through the speaker encoder to obtain a speaker embedding vector, and performing whole sentence level emotion embedding recognition on the reference audio through the sentence-level audio encoder to obtain a first emotion vector.

[0059] In some embodiments, speaker feature recognition is performed on the reference audio through a speaker encoder to obtain a speaker embedding vector. By performing speaker feature recognition on the reference audio through a speaker encoder, the speaker embedding vector of the speaker can be accurately obtained, which greatly improves the efficiency and accuracy of speech synthesis. The speaker features include but are not limited to the speaker's name and gender, etc.

[0060] In some embodiments, the sentence-level audio encoder performs overall emotion embedding recognition on each sentence in the reference audio to obtain a first emotion vector. By recognizing the reference audio through the sentence-level audio encoder, the first emotion vector of the entire reference audio sentence can be accurately obtained, which greatly improves the efficiency and accuracy of speech synthesis.

[0061] Step S103: Perform frame-level deep emotion embedding recognition on the reference audio through the frame-level audio encoder to obtain a second emotion vector.

[0062] In some embodiments, the reference audio is segmented at a preset frame granularity to obtain multiple reference sub-audios; emotion embedding recognition is performed on each reference sub-audio through a frame-level audio encoder to obtain multiple sub-emotion vectors; the multiple sub-emotion vectors are fused to generate a second emotion vector, and each sub-emotion vector in the second emotion vector is arranged on a time axis. Among them, the preset frame granularity can be set according to actual conditions, and the embodiment of the present invention does not make specific limitations on this. For example, the preset frame granularity can be set to 0.05 seconds. By segmenting the reference audio at a preset frame granularity, multiple reference sub-audios can be accurately obtained, and by performing emotion embedding recognition on each reference sub-audio through a frame-level audio encoder, multiple sub-emotion vectors can be accurately obtained.

[0063] It should be noted that by performing frame-level deep emotion embedding recognition on the reference audio through a frame-level audio encoder, emotional information at the reference audio frame level can be obtained, thereby effectively improving the authenticity and accuracy of the synthesized speech emotion.

[0064] Step S104: performing vector fusion on the speaker embedding vector, the first emotion vector and the second emotion vector through the embedding fusion module to obtain an emotion embedding vector.

[0065] In some embodiments, the embedding fusion module inserts each sub-emotion vector in the second emotion vector into the corresponding time position in the first emotion vector according to the time axis to adjust the emotion embedding information represented by the first emotion vector to obtain a third emotion vector; and performs vector fusion on the speaker embedding vector and the third emotion vector to obtain an emotion embedding vector. By inserting each sub-emotion vector in the second emotion vector into the corresponding time position in the first emotion vector to adjust the emotion embedding information represented by the first emotion vector, the emotion embedding vector corresponding to the reference audio is made more accurate and real, thereby effectively improving the efficiency and accuracy of speech synthesis.

[0066] Step S105: feature encode the target text through the text encoding layer to obtain a text embedding vector, and perform speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector.

[0067] In some embodiments, the target text is feature encoded by the text encoding layer to obtain a text embedding vector. The text embedding vector can be accurately obtained by feature encoding the target text by the text encoding layer, which greatly improves the efficiency of speech synthesis.

[0068] In some embodiments, the prediction layer generates speech from the text embedding vector to obtain a candidate speech audio vector; the phonemes represented by the candidate speech audio vector are adjusted according to the emotion embedding vector to generate a speech audio vector. The prediction layer can accurately synthesize the candidate speech audio vector corresponding to the text embedding vector, and then adjust the phonemes represented by the candidate speech audio vector based on the emotion embedding vector, so as to accurately obtain the speech audio vector with the required emotional rhythm.

[0069] Step S106: perform speech output on the speech audio vector through the decoder layer to obtain a target speech corresponding to the target text.

[0070] The decoder layer outputs the speech audio vector to obtain the target speech corresponding to the target text. The decoder layer can accurately output the target speech corresponding to the target text, greatly improving the accuracy of speech synthesis.

[0071] Exemplarily, in the process of financial product sales, the target text that the salesperson needs to narrate, the salesperson's reference audio and a preset speech synthesis model are obtained, the speaker feature recognition is performed on the reference audio through a speaker encoder to obtain a speaker embedding vector, and the whole sentence level emotion embedding recognition is performed on the reference audio through a sentence-level audio encoder to obtain a first emotion vector; the reference audio is subjected to frame-level deep emotion embedding recognition through a frame-level audio encoder to obtain a second emotion vector; the speaker embedding vector, the first emotion vector and the second emotion vector are vector-fused through an embedding fusion module to accurately obtain the salesperson's emotion embedding vector; the target text is feature-encoded through a text encoding layer to obtain a text embedding vector, and the text embedding vector and the emotion embedding vector are subjected to speech audio synthesis through the speech prediction layer to generate a speech audio vector; the speech audio vector is voice-outputted through a decoder layer to obtain a target speech corresponding to the target text that the salesperson needs to describe, thereby greatly improving the efficiency and accuracy of customers in obtaining sales information.

[0072] Exemplarily, during the diagnosis process of seeing a doctor, the target text that the doctor needs to narrate, the doctor's reference audio and a preset speech synthesis model are obtained, the speaker feature recognition is performed on the reference audio through a speaker encoder to obtain a speaker embedding vector, and the whole sentence level emotion embedding recognition is performed on the reference audio through a sentence-level audio encoder to obtain a first emotion vector; the reference audio is subjected to frame-level deep emotion embedding recognition through a frame-level audio encoder to obtain a second emotion vector; the speaker embedding vector, the first emotion vector and the second emotion vector are vector-fused through an embedding fusion module to accurately obtain the doctor's emotion embedding vector; the target text is feature-encoded through a text encoding layer to obtain a text embedding vector, and the text embedding vector and the emotion embedding vector are subjected to speech audio synthesis through a speech prediction layer to generate a speech audio vector; the speech audio vector is speech-outputted through a decoder layer to obtain the target speech corresponding to the target text that the doctor needs to describe, thereby greatly improving the efficiency and accuracy of patients in obtaining diagnosis and treatment information.

[0073] The speech synthesis method provided in the above embodiment obtains a target text, a reference audio and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer and a decoder layer, wherein the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder and an embedding fusion module; the speaker feature recognition is performed on the reference audio by the speaker encoder to obtain a speaker embedding vector, and the whole sentence-level emotion embedding recognition is performed on the reference audio by the sentence-level audio encoder to obtain a first emotion vector; the frame-level deep emotion embedding recognition is performed on the reference audio by the frame-level audio encoder to obtain a second emotion vector; the speaker embedding vector, the first emotion vector and the second emotion vector are vector-fused by the embedding fusion module to obtain an emotion embedding vector; the target text is feature-encoded by the text coding layer to obtain a text embedding vector, and the text embedding vector and the emotion embedding vector are subjected to speech audio synthesis by the speech prediction layer to generate a speech audio vector; the speech audio vector is speech-outputted by the decoder layer to obtain a target speech corresponding to the target text. In this application, speech synthesis is performed by performing sentence-level emotion embedding and frame-level deep emotion embedding on the target text and reference audio through various levels in the preset speech synthesis model, so that the synthesized speech has better sound quality, naturalness and emotion, greatly improving the accuracy of speech synthesis.

[0074] See also Figure 4 , Figure 4 A schematic block diagram of a speech synthesis device provided in an embodiment of the present application.

[0075] like Figure 4 As shown, the speech synthesis device 300 includes an acquisition module 310 and a generation module 320, wherein:

[0076] The acquisition module 310 is used to acquire the target text, the reference audio and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer and a decoder layer, and the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder and an embedding fusion module;

[0077] The generating module 320 is used for:

[0078] Perform speaker feature recognition on the reference audio through a speaker encoder to obtain a speaker embedding vector, and perform sentence-level emotion embedding recognition on the reference audio through a sentence-level audio encoder to obtain a first emotion vector;

[0079] Performing frame-level deep emotion embedding recognition on the reference audio through a frame-level audio encoder to obtain a second emotion vector;

[0080] Performing vector fusion on the speaker embedding vector, the first emotion vector and the second emotion vector through the embedding fusion module to obtain an emotion embedding vector;

[0081] Performing feature encoding on the target text through the text encoding layer to obtain a text embedding vector, and performing speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector;

[0082] The speech audio vector is outputted via the decoder layer to obtain a target speech corresponding to the target text.

[0083] In some embodiments, the generating module 320 is further configured to:

[0084] Segmenting the reference audio at a preset frame granularity to obtain a plurality of reference sub-audios;

[0085] Performing emotion embedding recognition on each of the reference sub-audios through the frame-level audio encoder to obtain a plurality of sub-emotion vectors;

[0086] The multiple sub-emotion vectors are merged to generate the second emotion vector, and the sub-emotion vectors in the second emotion vector are arranged on a time axis.

[0087] In some embodiments, the generating module 320 is further configured to:

[0088] The embedding fusion module inserts each sub-emotion vector in the second emotion vector into a corresponding time position in the first emotion vector according to the time axis, so as to adjust the emotion embedding information represented by the first emotion vector to obtain a third emotion vector;

[0089] The speaker embedding vector and the third emotion vector are fused to obtain the emotion embedding vector.

[0090] In some embodiments, the generating module 320 is further configured to:

[0091] Performing speech generation on the text embedding vector through the prediction layer to obtain a candidate speech audio vector;

[0092] The phonemes represented by the candidate speech audio vector are adjusted according to the emotion embedding vector to generate the speech audio vector.

[0093] In some embodiments, the speech synthesis device 300 is further used for:

[0094] Acquire a sample data set, the sample data set comprising a plurality of sample data, the sample data comprising a sample text, a sample reference audio, and a reference speech marked corresponding to the sample text;

[0095] Selecting a sample data from the sample data set as target sample data, training a preset speech synthesis model according to the target sample data, and determining whether the preset speech synthesis model converges;

[0096] If the preset speech synthesis model has not converged, adjust the model parameters of the preset speech synthesis model, and repeat the step of selecting a sample data from the sample data set as the target sample data until the preset speech synthesis model converges to obtain the speech synthesis model.

[0097] In some embodiments, the speech synthesis device 300 is further used for:

[0098] Performing speaker feature recognition on the sample reference audio through the speaker encoder to obtain a predicted speaker embedding vector, and performing whole sentence level emotion embedding recognition on the sample reference audio through the sentence level audio encoder to obtain a predicted first emotion vector;

[0099] Performing frame-level deep emotion embedding recognition on the sample reference audio by the frame-level audio encoder to obtain a predicted second emotion vector;

[0100] Performing vector fusion on the predicted speaker embedding vector, the predicted first emotion vector and the predicted second emotion vector through the embedding fusion module to obtain a predicted emotion embedding vector;

[0101] Performing feature encoding on the sample text through the text encoding layer to obtain a predicted text embedding vector, and performing speech audio synthesis on the predicted text embedding vector and the predicted emotion embedding vector through the speech prediction layer to generate a predicted speech audio vector;

[0102] Performing speech output on the predicted speech audio vector through the decoder layer to obtain the predicted speech corresponding to the sample text;

[0103] Determine whether the preset speech synthesis model converges according to the identified reference speech and the predicted speech.

[0104] In some embodiments, the speech synthesis device 300 is further used for:

[0105] Determining a loss value of the preset speech synthesis model according to the identified reference speech and the predicted speech;

[0106] If the loss value is greater than or equal to a preset loss value, determining that the preset speech synthesis model has not converged;

[0107] If the loss value is less than a preset loss value, it is determined that the preset speech synthesis model has converged.

[0108] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-mentioned speech synthesis device can refer to the corresponding process in the aforementioned speech synthesis method embodiment, and will not be repeated here.

[0109] See also Figure 5 , Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0110] like Figure 5 As shown, the computer device 400 includes a processor 402 and a memory 403 connected via a system bus 401 , wherein the memory 403 may include a storage medium and an internal memory.

[0111] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any speech synthesis method.

[0112] The processor 402 is used to provide computing and control capabilities and support the operation of the entire computer device.

[0113] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any speech synthesis method.

[0114] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0115] It should be understood that the processor 402 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0116] In one embodiment, the processor 402 is used to run a computer program stored in the memory to implement the following steps:

[0117] Acquire a target text, a reference audio, and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer, and a decoder layer, wherein the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder, and an embedding fusion module;

[0118] Perform speaker feature recognition on the reference audio by the speaker encoder to obtain a speaker embedding vector, and perform sentence-level emotion embedding recognition on the reference audio by the sentence-level audio encoder to obtain a first emotion vector;

[0119] Performing frame-level deep emotion embedding recognition on the reference audio by the frame-level audio encoder to obtain a second emotion vector;

[0120] Performing vector fusion on the speaker embedding vector, the first emotion vector and the second emotion vector through the embedding fusion module to obtain an emotion embedding vector;

[0121] Performing feature encoding on the target text through the text encoding layer to obtain a text embedding vector, and performing speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector;

[0122] The speech audio vector is outputted via the decoder layer to obtain a target speech corresponding to the target text.

[0123] In one embodiment, when the processor 402 performs frame-level deep emotion embedding recognition on the reference audio through the frame-level audio encoder to obtain the second emotion vector, it is configured to implement:

[0124] Segmenting the reference audio at a preset frame granularity to obtain a plurality of reference sub-audios;

[0125] Performing emotion embedding recognition on each of the reference sub-audios through the frame-level audio encoder to obtain a plurality of sub-emotion vectors;

[0126] The multiple sub-emotion vectors are merged to generate the second emotion vector, and the sub-emotion vectors in the second emotion vector are arranged on a time axis.

[0127] In one embodiment, when the processor 402 implements the vector fusion of the speaker embedding vector, the first emotion vector and the second emotion vector by the embedding fusion module to obtain the emotion embedding vector, it is used to implement:

[0128] The embedding fusion module inserts each sub-emotion vector in the second emotion vector into a corresponding time position in the first emotion vector according to the time axis, so as to adjust the emotion embedding information represented by the first emotion vector to obtain a third emotion vector;

[0129] The speaker embedding vector and the third emotion vector are fused to obtain the emotion embedding vector.

[0130] In one embodiment, when the processor 402 performs speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector, it is used to implement:

[0131] Performing speech generation on the text embedding vector through the prediction layer to obtain a candidate speech audio vector;

[0132] The phonemes represented by the candidate speech audio vector are adjusted according to the emotion embedding vector to generate the speech audio vector.

[0133] In one embodiment, before acquiring the target text, the reference audio and the preset speech synthesis model, the processor 402 is further configured to implement:

[0134] Acquire a sample data set, the sample data set comprising a plurality of sample data, the sample data comprising a sample text, a sample reference audio, and a reference speech marked corresponding to the sample text;

[0135] Selecting a sample data from the sample data set as target sample data, training a preset speech synthesis model according to the target sample data, and determining whether the preset speech synthesis model converges;

[0136] If the preset speech synthesis model has not converged, adjust the model parameters of the preset speech synthesis model, and repeat the step of selecting a sample data from the sample data set as the target sample data until the preset speech synthesis model converges to obtain the speech synthesis model.

[0137] In one embodiment, the processor 402, when implementing the training of the preset speech synthesis model according to the target sample data and determining whether the preset speech synthesis model converges, is used to implement:

[0138] Performing speaker feature recognition on the sample reference audio through the speaker encoder to obtain a predicted speaker embedding vector, and performing whole sentence level emotion embedding recognition on the sample reference audio through the sentence level audio encoder to obtain a predicted first emotion vector;

[0139] Performing frame-level deep emotion embedding recognition on the sample reference audio by the frame-level audio encoder to obtain a predicted second emotion vector;

[0140] Performing vector fusion on the predicted speaker embedding vector, the predicted first emotion vector and the predicted second emotion vector through the embedding fusion module to obtain a predicted emotion embedding vector;

[0141] Performing feature encoding on the sample text through the text encoding layer to obtain a predicted text embedding vector, and performing speech audio synthesis on the predicted text embedding vector and the predicted emotion embedding vector through the speech prediction layer to generate a predicted speech audio vector;

[0142] Performing speech output on the predicted speech audio vector through the decoder layer to obtain the predicted speech corresponding to the sample text;

[0143] Determine whether the preset speech synthesis model converges according to the identified reference speech and the predicted speech.

[0144] In one embodiment, when the processor 402 determines whether the preset speech synthesis model converges according to the identified reference speech and the predicted speech, it is configured to implement:

[0145] Determining a loss value of the preset speech synthesis model according to the identified reference speech and the predicted speech;

[0146] If the loss value is greater than or equal to a preset loss value, determining that the preset speech synthesis model has not converged;

[0147] If the loss value is less than a preset loss value, it is determined that the preset speech synthesis model has converged.

[0148] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the computer device described above can refer to the corresponding process in the aforementioned speech synthesis method embodiment, and will not be repeated here.

[0149] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the speech synthesis method of the present application.

[0150] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may be non-volatile or volatile. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0151] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0152] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0153] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0154] It should also be understood that the term "and / or" used in the present specification refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0155] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application.

Claims

1. A speech synthesis method, characterized in that: include: Acquire a target text, a reference audio, and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer, and a decoder layer, wherein the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder, and an embedding fusion module; Perform speaker feature recognition on the reference audio by the speaker encoder to obtain a speaker embedding vector, and perform sentence-level emotion embedding recognition on the reference audio by the sentence-level audio encoder to obtain a first emotion vector; Performing frame-level deep emotion embedding recognition on the reference audio by the frame-level audio encoder to obtain a second emotion vector; Performing vector fusion on the speaker embedding vector, the first emotion vector and the second emotion vector through the embedding fusion module to obtain an emotion embedding vector; Performing feature encoding on the target text through the text encoding layer to obtain a text embedding vector, and performing speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector; The speech audio vector is outputted via the decoder layer to obtain a target speech corresponding to the target text.

2. The speech synthesis method according to claim 1, characterized in that: The step of performing frame-level deep emotion embedding recognition on the reference audio by the frame-level audio encoder to obtain the second emotion vector includes: Segmenting the reference audio at a preset frame granularity to obtain a plurality of reference sub-audios; Performing emotion embedding recognition on each of the reference sub-audios through the frame-level audio encoder to obtain a plurality of sub-emotion vectors; The multiple sub-emotion vectors are merged to generate the second emotion vector, and the sub-emotion vectors in the second emotion vector are arranged on a time axis.

3. The speech synthesis method according to claim 2, characterized in that: The step of fusing the speaker embedding vector, the first emotion vector, and the second emotion vector by the embedding fusion module to obtain the emotion embedding vector includes: The embedding fusion module inserts each sub-emotion vector in the second emotion vector into a corresponding time position in the first emotion vector according to the time axis, so as to adjust the emotion embedding information represented by the first emotion vector to obtain a third emotion vector; The speaker embedding vector and the third emotion vector are fused to obtain the emotion embedding vector.

4. The speech synthesis method according to claim 1, wherein: The performing speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector includes: Performing speech generation on the text embedding vector through the prediction layer to obtain a candidate speech audio vector; The phonemes represented by the candidate speech audio vector are adjusted according to the emotion embedding vector to generate the speech audio vector.

5. The speech synthesis method according to claim 1, wherein: Before obtaining the target text, reference audio and preset speech synthesis model, the method further includes: Acquire a sample data set, the sample data set comprising a plurality of sample data, the sample data comprising a sample text, a sample reference audio, and a reference speech marked corresponding to the sample text; Selecting a sample data from the sample data set as target sample data, training a preset speech synthesis model according to the target sample data, and determining whether the preset speech synthesis model converges; If the preset speech synthesis model has not converged, adjust the model parameters of the preset speech synthesis model, and repeat the step of selecting a sample data from the sample data set as the target sample data until the preset speech synthesis model converges to obtain the speech synthesis model.

6. The speech synthesis method according to claim 5, characterized in that: The step of training a preset speech synthesis model according to the target sample data and determining whether the preset speech synthesis model converges comprises: Performing speaker feature recognition on the sample reference audio through the speaker encoder to obtain a predicted speaker embedding vector, and performing whole sentence level emotion embedding recognition on the sample reference audio through the sentence level audio encoder to obtain a predicted first emotion vector; Performing frame-level deep emotion embedding recognition on the sample reference audio by the frame-level audio encoder to obtain a predicted second emotion vector; Performing vector fusion on the predicted speaker embedding vector, the predicted first emotion vector and the predicted second emotion vector through the embedding fusion module to obtain a predicted emotion embedding vector; Performing feature encoding on the sample text through the text encoding layer to obtain a predicted text embedding vector, and performing speech audio synthesis on the predicted text embedding vector and the predicted emotion embedding vector through the speech prediction layer to generate a predicted speech audio vector; Performing speech output on the predicted speech audio vector through the decoder layer to obtain the predicted speech corresponding to the sample text; Determine whether the preset speech synthesis model converges according to the identified reference speech and the predicted speech.

7. The speech synthesis method according to claim 6, characterized in that: The step of determining whether the preset speech synthesis model converges according to the identified reference speech and the predicted speech includes: Determining a loss value of the preset speech synthesis model according to the identified reference speech and the predicted speech; If the loss value is greater than or equal to a preset loss value, determining that the preset speech synthesis model has not converged; If the loss value is less than a preset loss value, it is determined that the preset speech synthesis model has converged.

8. A speech synthesis device, characterized in that: The speech synthesis device comprises an acquisition module and a generation module, wherein: The acquisition module is used to acquire the target text, the reference audio and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer and a decoder layer, and the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder and an embedding fusion module; The generating module is used for: Perform speaker feature recognition on the reference audio through a speaker encoder to obtain a speaker embedding vector, and perform sentence-level emotion embedding recognition on the reference audio through a sentence-level audio encoder to obtain a first emotion vector; Performing frame-level deep emotion embedding recognition on the reference audio through a frame-level audio encoder to obtain a second emotion vector; Performing vector fusion on the speaker embedding vector, the first emotion vector and the second emotion vector through the embedding fusion module to obtain an emotion embedding vector; Performing feature encoding on the target text through the text encoding layer to obtain a text embedding vector, and performing speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector; The speech audio vector is outputted via the decoder layer to obtain a target speech corresponding to the target text.

9. A computer device, characterized in that: The computer device comprises a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the speech synthesis method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the speech synthesis method according to any one of claims 1 to 7 are implemented.

Images