Speech synthesis method, device, equipment and storage medium

Abstract

The present application provides a speech synthesis method, apparatus, device, and storage medium, belonging to the field of speech synthesis. The method comprises: obtaining a target text, reference audio, and a preset speech synthesis model; and performing speech synthesis on the target text and reference audio based on the preset speech synthesis model to obtain a target speech corresponding to the target text. By performing speech synthesis on the target text and reference audio using sentence-level emotion embedding and frame-level deep emotion embedding at various levels in the preset speech synthesis model, the synthesized speech has better sound quality, naturalness, and emotion, greatly improving the accuracy of speech synthesis. The present application also relates to blockchain technology, and the preset speech synthesis model can be stored in a blockchain.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and in particular to a speech synthesis method, apparatus, device and storage medium. Background Art

[0002] With the development of speech synthesis models, the quality of synthesized speech is constantly improving, and the similarity of speakers is getting higher and higher. However, speech synthesis has a one-to-many problem where text can map to multiple speech sounds. Simply synthesizing the voice of the target speaker is no longer enough. We need more methods and means to regulate the entire model to make the synthesized speech more controllable.

[0003] In response to this, speech synthesis based on emotion transfer came into being. By inputting speech, the model is forced to synthesize speech that conforms to the current emotional form rather than arbitrarily synthesizing a learned emotion. In the past, emotion transfer speech synthesis models often used a single acoustic feature such as Mel spectrum to predict emotional information. However, this method does not provide sufficient input features and does not take into account the speaker's emotions when speaking, resulting in the synthesized speech having no rhythm changes and low expressiveness.

[0004] Therefore, how to synthesize speech with emotion embedding is an urgent problem to be solved. Summary of the Invention

[0005] The main purpose of this application is to provide a speech synthesis method, apparatus, device and storage medium, aiming to improve the emotional naturalness and accuracy of synthesized speech.

[0006] In a first aspect, the present application provides a speech synthesis method, the speech synthesis method comprising the following steps:

[0007] Obtaining target text, reference audio, and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer, and a decoder layer, wherein the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder, and an embedding fusion module;

[0008] Performing speaker feature recognition on the reference audio through the speaker encoder to obtain a speaker embedding vector, and performing whole sentence level emotion embedding recognition on the reference audio through the sentence-level audio encoder to obtain a first emotion vector;

[0009] Performing frame-level deep emotion embedding recognition on the reference audio by the frame-level audio encoder to obtain a second emotion vector;

[0010] Performing vector fusion on the speaker embedding vector, the first emotion vector, and the second emotion vector by the embedding fusion module to obtain an emotion embedding vector;

[0011] Performing feature encoding on the target text through the text encoding layer to obtain a text embedding vector, and performing speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector;

[0012] The speech audio vector is outputted via the decoder layer to obtain a target speech corresponding to the target text.

[0013] In a second aspect, the present application further provides a speech synthesis device, the speech synthesis device comprising an acquisition module and a generation module, wherein:

[0014] The acquisition module is used to acquire the target text, reference audio and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer and a decoder layer, and the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder and an embedding fusion module;

[0015] The generating module is used to:

[0016] Performing speaker feature recognition on the reference audio through a speaker encoder to obtain a speaker embedding vector, and performing sentence-level emotion embedding recognition on the reference audio through a sentence-level audio encoder to obtain a first emotion vector;

[0017] Performing frame-level deep emotion embedding recognition on the reference audio through a frame-level audio encoder to obtain a second emotion vector;

[0018] Performing vector fusion on the speaker embedding vector, the first emotion vector, and the second emotion vector by the embedding fusion module to obtain an emotion embedding vector;

[0019] Performing feature encoding on the target text through the text encoding layer to obtain a text embedding vector, and performing speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector;

[0020] The speech audio vector is outputted via the decoder layer to obtain a target speech corresponding to the target text.

[0021] In a third aspect, the present application also provides a computer device comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the speech synthesis method as described above are implemented.

[0022] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the speech synthesis method as described above are implemented.

[0023] The present application provides a speech synthesis method, apparatus, device and storage medium. The present application obtains a target text, a reference audio and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer and a decoder layer, and the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder and an embedding fusion module; the speaker feature recognition is performed on the reference audio by the speaker encoder to obtain a speaker embedding vector, and the whole sentence-level emotion embedding recognition is performed on the reference audio by the sentence-level audio encoder to obtain a first emotion vector; the frame-level audio encoder is performed on the reference audio to perform frame-level deep emotion embedding recognition to obtain a second emotion vector; the speaker embedding vector, the first emotion vector and the second emotion vector are vector-fused by the embedding fusion module to obtain an emotion embedding vector; the target text is feature-encoded by the text coding layer to obtain a text embedding vector, and the text embedding vector and the emotion embedding vector are subjected to speech audio synthesis by the speech prediction layer to generate a speech audio vector; the speech audio vector is output by the decoder layer to obtain a target speech corresponding to the target text. In this application, speech synthesis is performed by performing sentence-level emotion embedding and frame-level deep emotion embedding on the target text and reference audio at various levels in the preset speech synthesis model, so that the synthesized speech has better sound quality, naturalness and emotion, greatly improving the accuracy of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 A flowchart of a speech synthesis method provided in an embodiment of the present application;

[0026] Figure 2 A schematic block diagram of a speech synthesis model provided in an embodiment of the present application;

[0027] Figure 3 A flowchart of another speech synthesis method provided in an embodiment of the present application;

[0028] Figure 4 A schematic block diagram of a speech synthesis device provided in an embodiment of the present application;

[0029] Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0030] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0032] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0033] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0034] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0035] With the development of speech synthesis models, the quality of synthesized speech is constantly improving, and the similarity of speakers is getting higher and higher. However, speech synthesis has a one-to-many problem where one text can be mapped to multiple speech sounds. Simply synthesizing the voice of the target speaker is no longer enough. We need more methods and means to regulate the entire model to make the synthesized speech more controllable.

[0036] In response to this, speech synthesis based on emotion transfer came into being. By inputting speech, the model is forced to synthesize speech that conforms to the current emotional form rather than arbitrarily synthesizing a learned emotion. In the past, emotion transfer speech synthesis models often used a single acoustic feature such as Mel spectrum to predict emotional information. However, this method does not provide sufficient input features and does not take into account the speaker's emotions when speaking, resulting in the synthesized speech having no rhythm changes and low expressiveness.

[0037] To solve the above-mentioned problems, the embodiments of the present application provide a speech synthesis method, apparatus, device and storage medium. The speech synthesis method includes: obtaining a target text, a reference audio and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer and a decoder layer, wherein the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder and an embedding fusion module; performing speaker feature recognition on the reference audio through the speaker encoder to obtain a speaker embedding vector, and performing sentence-level emotion embedding recognition on the reference audio through the sentence-level audio encoder to obtain a first emotion vector; performing frame-level deep emotion embedding recognition on the reference audio through the frame-level audio encoder to obtain a second emotion vector; performing vector fusion on the speaker embedding vector, the first emotion vector and the second emotion vector through the embedding fusion module to obtain an emotion embedding vector; performing feature encoding on the target text through the text coding layer to obtain a text embedding vector, and performing speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector; performing speech output on the speech audio vector through the decoder layer to obtain a target speech corresponding to the target text.

[0038] The speech synthesis method can be applied to computer devices, which may be electronic devices such as mobile phones, tablet computers, laptop computers, desktop computers, personal digital assistants, and wearable devices.

[0039] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0040] Please refer to Figure 1 , Figure 1 A flowchart of a speech synthesis method provided in an embodiment of the present application.

[0041] like Figure 1 As shown, the speech synthesis method includes steps S101 to S106.

[0042] Step S101: Obtain target text, reference audio and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer and a decoder layer, and the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder and an embedded fusion module.

[0043] like Figure 2 As shown, the preset speech synthesis model 100 includes an audio encoding layer 10, a text encoding layer 20, a speech prediction layer 30, and a decoder layer 40. The audio encoding layer 10 includes a speaker encoder 11, a sentence-level audio encoder 12, a frame-level audio encoder 13, and an embedding fusion module 14. The speech synthesis model is a neural network model. The type of the neural network model can be selected based on actual conditions and is not specifically limited in this embodiment of the present invention. For example, the neural network model includes but is not limited to a convolutional neural network model and a recurrent convolutional neural network model.

[0044] In some embodiments, as Figure 3 As shown, the present invention also provides another speech synthesis method, which includes steps S201 to S203.

[0045] Step S201: Acquire a sample data set, where the sample data set includes a plurality of sample data, and the sample data includes sample text, sample reference audio, and reference speech corresponding to the sample text.

[0046] The sample data set includes a plurality of sample data, including sample text, sample reference audio, and reference speech corresponding to the sample text. The sample data is pre-established based on the reference audio, text, and reference speech. The sample data is constructed by repeatedly performing the steps of collecting reference audio, text, and reference speech until a large amount of sample data is obtained to construct the sample data set.

[0047] Step S202: Select a sample data from the sample data set as target sample data, train a preset speech synthesis model according to the target sample data, and determine whether the preset speech synthesis model converges.

[0048] Among them, the method of selecting sample data can be set according to actual conditions, and the embodiment of the present invention does not make specific limitations on this. For example, a sample data can be randomly selected as the target sample data, or the target sample data can be selected in the order of the time when the sample data enters the data set.

[0049] In some embodiments, speaker feature recognition is performed on the sample reference audio through a speaker encoder to obtain a predicted speaker embedding vector, and whole-sentence-level emotion embedding recognition is performed on the sample reference audio through a sentence-level audio encoder to obtain a predicted first emotion vector; frame-level deep emotion embedding recognition is performed on the sample reference audio through a frame-level audio encoder to obtain a predicted second emotion vector; the predicted speaker embedding vector, the predicted first emotion vector and the predicted second emotion vector are vector fused through an embedding fusion module to obtain a predicted emotion embedding vector; feature encoding is performed on the sample text through a text encoding layer to obtain a predicted text embedding vector, and speech audio synthesis is performed on the predicted text embedding vector and the predicted emotion embedding vector through a speech prediction layer to generate a predicted speech audio vector; speech output is performed on the predicted speech audio vector through a decoder layer to obtain a predicted speech corresponding to the sample text; and whether the preset speech synthesis model converges is determined based on the identified reference speech and the predicted speech.

[0050] In some embodiments, a method for performing sentence-level emotion embedding recognition on the sample reference audio by a sentence-level audio encoder to obtain the predicted first emotion vector may be: performing holistic emotion embedding recognition on each sentence in the sample reference audio by the sentence-level audio encoder to obtain the predicted first emotion vector. By recognizing the sample reference audio by the sentence-level audio encoder, the predicted first emotion vector of the entire sentence of the sample reference audio can be accurately obtained.

[0051] In some embodiments, the sample reference audio is subjected to frame-level deep emotion embedding recognition by a frame-level audio encoder to obtain a predicted second emotion vector by: segmenting the sample reference audio at a preset frame granularity to obtain multiple sample reference sub-audios; performing emotion embedding recognition on each sample reference sub-audio by a frame-level audio encoder to obtain multiple predicted sub-emotion vectors; and fusing the multiple predicted sub-emotion vectors to generate a sample second emotion vector, in which the predicted sub-emotion vectors are arranged along the time axis.

[0052] In some embodiments, the predicted speaker embedding vector, the predicted first emotion vector and the predicted second emotion vector are vector-fused by the embedding fusion module to obtain the predicted emotion embedding vector in the following manner: the embedding fusion module inserts each predicted sub-emotion vector in the predicted second emotion vector into the corresponding time position in the predicted first emotion vector according to the time axis to adjust the emotion embedding information represented by the predicted first emotion vector to obtain the predicted third emotion vector; and the predicted speaker embedding vector and the predicted third emotion vector are vector-fused to obtain the predicted emotion embedding vector.

[0053] In some embodiments, the predicted text embedding vector and the predicted emotion embedding vector are subjected to speech audio synthesis through a speech prediction layer, and the method for generating the predicted speech audio vector can be: performing speech generation on the predicted text embedding vector through the prediction layer to obtain a predicted candidate speech audio vector; and adjusting the phonemes represented by the predicted candidate speech audio vector according to the predicted emotion embedding vector to generate a predicted speech audio vector.

[0054] In some embodiments, the method for determining whether the preset speech synthesis model has converged based on the identified reference speech and the predicted speech can be: determining the loss value of the preset speech synthesis model based on the identified reference speech and the predicted speech; if the loss value is greater than or equal to the preset loss value, determining that the preset speech synthesis model has not converged; if the loss value is less than the preset loss value, determining that the preset speech synthesis model has converged. Among them, the preset loss value can be set according to actual conditions, and the embodiment of the present invention does not make specific limitations on this. For example, the preset loss value can be set to 0.002. By calculating the loss value of the speech synthesis model, it is possible to accurately determine whether the model has converged, which greatly improves the stability of the trained model.

[0055] In some embodiments, the loss value of the preset speech synthesis model may be determined based on the identified reference speech and the predicted speech by calculating the Euclidean distance between the identified reference speech and the predicted speech, and determining the loss value of the preset speech synthesis model by subtracting the Euclidean distance from the unit of one. Alternatively, the cosine similarity between the identified reference speech and the predicted speech may be calculated, and the loss value of the preset speech synthesis model may be determined by subtracting the cosine similarity from the unit of one.

[0056] Step S203: If the preset speech synthesis model has not converged, adjust the model parameters of the preset speech synthesis model, and loop through the step of selecting a sample data from the sample data set as the target sample data until the preset speech synthesis model converges to obtain a speech synthesis model.

[0057] If the preset speech synthesis model has not converged, adjust the model parameters of the preset speech synthesis model, and cyclically select a sample data from the sample data set as the target sample data, train the preset speech synthesis model according to the target sample data, and determine whether the preset speech synthesis model has converged; if the preset speech synthesis model has not converged, adjust the model parameters of the preset speech synthesis model until the preset speech synthesis model converges to obtain the speech synthesis model.

[0058] Step S102: Perform speaker feature recognition on the reference audio through the speaker encoder to obtain a speaker embedding vector, and perform whole-sentence-level emotion embedding recognition on the reference audio through the sentence-level audio encoder to obtain a first emotion vector.

[0059] In some embodiments, a speaker encoder is used to perform speaker feature recognition on reference audio to obtain a speaker embedding vector. This can accurately obtain the speaker embedding vector for the speaker, significantly improving the efficiency and accuracy of speech synthesis. The speaker features include, but are not limited to, the speaker's name and gender.

[0060] In some embodiments, the sentence-level audio encoder performs holistic emotion embedding recognition on each sentence in the reference audio to obtain a first emotion vector. Using the sentence-level audio encoder to recognize the reference audio accurately obtains the first emotion vector for the entire reference audio sentence, significantly improving the efficiency and accuracy of speech synthesis.

[0061] Step S103: Perform frame-level deep emotion embedding recognition on the reference audio through the frame-level audio encoder to obtain a second emotion vector.

[0062] In some embodiments, the reference audio is segmented at a preset frame granularity to obtain a plurality of reference sub-audios; emotion embedding recognition is performed on each reference sub-audio through a frame-level audio encoder to obtain a plurality of sub-emotion vectors; the plurality of sub-emotion vectors are fused to generate a second emotion vector, and each sub-emotion vector in the second emotion vector is arranged along a time axis. The preset frame granularity can be set according to actual conditions, and the embodiment of the present invention does not make specific limitations on this. For example, the preset frame granularity can be set to 0.05 seconds. By segmenting the reference audio at the preset frame granularity, a plurality of reference sub-audios can be accurately obtained, and by performing emotion embedding recognition on each reference sub-audio through a frame-level audio encoder, a plurality of sub-emotion vectors can be accurately obtained.

[0063] It should be noted that by performing frame-level deep emotion embedding recognition on the reference audio through a frame-level audio encoder, emotional information at the reference audio frame level can be obtained, thereby effectively improving the authenticity and accuracy of the synthesized speech emotion.

[0064] Step S104: Perform vector fusion on the speaker embedding vector, the first emotion vector, and the second emotion vector through the embedding fusion module to obtain an emotion embedding vector.

[0065] In some embodiments, the embedding fusion module inserts each sub-emotion vector in the second emotion vector into the corresponding time position in the first emotion vector according to the time axis to adjust the emotion embedding information represented by the first emotion vector to obtain a third emotion vector; and performs vector fusion on the speaker embedding vector and the third emotion vector to obtain an emotion embedding vector. By inserting each sub-emotion vector in the second emotion vector into the corresponding time position in the first emotion vector to adjust the emotion embedding information represented by the first emotion vector, the emotion embedding vector corresponding to the reference audio is made more accurate and realistic, thereby effectively improving the efficiency and accuracy of speech synthesis.

[0066] Step S105: feature encode the target text through the text encoding layer to obtain a text embedding vector, and perform speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector.

[0067] In some embodiments, the target text is feature-encoded by the text encoding layer to obtain a text embedding vector. This feature encoding of the target text by the text encoding layer can accurately obtain the text embedding vector, greatly improving the efficiency of speech synthesis.

[0068] In some embodiments, a prediction layer generates speech from the text embedding vector to obtain a candidate speech audio vector; the phonemes represented by the candidate speech audio vector are adjusted based on the emotion embedding vector to generate a speech audio vector. The prediction layer can accurately synthesize the candidate speech audio vector corresponding to the text embedding vector, and then adjust the phonemes represented by the candidate speech audio vector based on the emotion embedding vector to accurately obtain the desired emotional prosody speech audio vector.

[0069] Step S106: Output the speech audio vector through the decoder layer to obtain the target speech corresponding to the target text.

[0070] The decoder layer outputs the speech audio vector to generate the target speech corresponding to the target text. The decoder layer can accurately output the target speech corresponding to the target text, greatly improving the accuracy of speech synthesis.

[0071] Exemplarily, in the process of selling financial products, the target text that the salesperson needs to narrate, the salesperson's reference audio and a preset speech synthesis model are obtained, the speaker feature recognition is performed on the reference audio through a speaker encoder to obtain a speaker embedding vector, and the whole sentence level emotion embedding recognition is performed on the reference audio through a sentence-level audio encoder to obtain a first emotion vector; the frame-level audio encoder is used to perform frame-level deep emotion embedding recognition on the reference audio to obtain a second emotion vector; the speaker embedding vector, the first emotion vector and the second emotion vector are vector-fused through an embedding fusion module to accurately obtain the salesperson's emotion embedding vector; the target text is feature-encoded through a text encoding layer to obtain a text embedding vector, and the text embedding vector and the emotion embedding vector are subjected to speech audio synthesis through the speech prediction layer to generate a speech audio vector; the speech audio vector is voice-outputted through a decoder layer to obtain the target speech corresponding to the salesperson's need to describe the target text, thereby greatly improving the efficiency and accuracy of customers in obtaining sales information.

[0072] For example, during the diagnosis process of seeing a doctor, the target text that the doctor needs to say, the doctor's reference audio and a preset speech synthesis model are obtained, the speaker feature recognition of the reference audio is performed through the speaker encoder to obtain the speaker embedding vector, and the whole sentence level emotion embedding recognition of the reference audio is performed through the sentence-level audio encoder to obtain the first emotion vector; the frame-level audio encoder performs frame-level deep emotion embedding recognition on the reference audio to obtain the second emotion vector; the speaker embedding vector, the first emotion vector and the second emotion vector are vector fused through the embedding fusion module to accurately obtain the doctor's emotion embedding vector; the target text is feature-encoded through the text encoding layer to obtain a text embedding vector, and the text embedding vector and the emotion embedding vector are subjected to speech audio synthesis through the speech prediction layer to generate a speech audio vector; the speech audio vector is output through the decoder layer to obtain the target speech corresponding to the target text that the doctor needs to describe, which greatly improves the efficiency and accuracy of patients in obtaining diagnosis and treatment information.

[0073] The speech synthesis method provided in the above embodiment obtains target text, reference audio and a preset speech synthesis model, the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer and a decoder layer, the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder and an embedding fusion module; the speaker feature recognition is performed on the reference audio by the speaker encoder to obtain a speaker embedding vector, and the whole sentence-level emotion embedding recognition is performed on the reference audio by the sentence-level audio encoder to obtain a first emotion vector; the frame-level audio encoder is performed on the reference audio at the frame level to obtain a second emotion vector; the speaker embedding vector, the first emotion vector and the second emotion vector are vector-fused by the embedding fusion module to obtain an emotion embedding vector; the target text is feature-encoded by the text coding layer to obtain a text embedding vector, and the text embedding vector and the emotion embedding vector are subjected to speech audio synthesis by the speech prediction layer to generate a speech audio vector; the speech audio vector is speech-outputted by the decoder layer to obtain a target speech corresponding to the target text. In this application, speech synthesis is performed by performing sentence-level emotion embedding and frame-level deep emotion embedding on the target text and reference audio at various levels in the preset speech synthesis model, so that the synthesized speech has better sound quality, naturalness and emotion, greatly improving the accuracy of speech synthesis.

[0074] See also Figure 4 , Figure 4 A schematic block diagram of a speech synthesis device provided in an embodiment of the present application.

[0075] like Figure 4 As shown, the speech synthesis device 300 includes an acquisition module 310 and a generation module 320, wherein:

[0076] The acquisition module 310 is used to acquire target text, reference audio, and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer, and a decoder layer, and the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder, and an embedding fusion module;

[0077] The generating module 320 is configured to:

[0078] Performing speaker feature recognition on the reference audio through a speaker encoder to obtain a speaker embedding vector, and performing sentence-level emotion embedding recognition on the reference audio through a sentence-level audio encoder to obtain a first emotion vector;

[0079] Performing frame-level deep emotion embedding recognition on the reference audio through a frame-level audio encoder to obtain a second emotion vector;

[0080] Performing vector fusion on the speaker embedding vector, the first emotion vector, and the second emotion vector by the embedding fusion module to obtain an emotion embedding vector;

[0081] Performing feature encoding on the target text through the text encoding layer to obtain a text embedding vector, and performing speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector;

[0082] The speech audio vector is outputted via the decoder layer to obtain a target speech corresponding to the target text.

[0083] In some embodiments, the generating module 320 is further configured to:

[0084] Segmenting the reference audio at a preset frame granularity to obtain a plurality of reference sub-audios;

[0085] Performing emotion embedding recognition on each of the reference sub-audios using the frame-level audio encoder to obtain a plurality of sub-emotion vectors;

[0086] The multiple sub-emotion vectors are fused to generate the second emotion vector, and the sub-emotion vectors in the second emotion vector are arranged along a time axis.

[0087] In some embodiments, the generating module 320 is further configured to:

[0088] The embedding fusion module inserts each sub-emotion vector in the second emotion vector into the corresponding time position in the first emotion vector according to the time axis, so as to adjust the emotion embedding information represented by the first emotion vector to obtain a third emotion vector;

[0089] Perform vector fusion on the speaker embedding vector and the third emotion vector to obtain the emotion embedding vector.

[0090] In some embodiments, the generating module 320 is further configured to:

[0091] Performing speech generation on the text embedding vector through the prediction layer to obtain a candidate speech audio vector;

[0092] The phonemes represented by the candidate speech audio vector are adjusted according to the emotion embedding vector to generate the speech audio vector.

[0093] In some embodiments, the speech synthesis apparatus 300 is further configured to:

[0094] Acquire a sample data set, the sample data set including a plurality of sample data, the sample data including a sample text, a sample reference audio, and a reference speech identified by the sample text;

[0095] Selecting a sample data from the sample data set as target sample data, training a preset speech synthesis model according to the target sample data, and determining whether the preset speech synthesis model converges;

[0096] If the preset speech synthesis model has not converged, adjust the model parameters of the preset speech synthesis model, and loop through the step of selecting a sample data from the sample data set as the target sample data until the preset speech synthesis model converges to obtain a speech synthesis model.

[0097] In some embodiments, the speech synthesis apparatus 300 is further configured to:

[0098] Performing speaker feature recognition on the sample reference audio through the speaker encoder to obtain a predicted speaker embedding vector, and performing sentence-level emotion embedding recognition on the sample reference audio through the sentence-level audio encoder to obtain a predicted first emotion vector;

[0099] Performing frame-level deep emotion embedding recognition on the sample reference audio by the frame-level audio encoder to obtain a predicted second emotion vector;

[0100] Performing vector fusion on the predicted speaker embedding vector, the predicted first emotion vector, and the predicted second emotion vector through the embedding fusion module to obtain a predicted emotion embedding vector;

[0101] Performing feature encoding on the sample text through the text encoding layer to obtain a predicted text embedding vector, and performing speech audio synthesis on the predicted text embedding vector and the predicted emotion embedding vector through the speech prediction layer to generate a predicted speech audio vector;

[0102] Performing speech output on the predicted speech audio vector through the decoder layer to obtain the predicted speech corresponding to the sample text;

[0103] Determine whether the preset speech synthesis model converges based on the identified reference speech and the predicted speech.

[0104] In some embodiments, the speech synthesis apparatus 300 is further configured to:

[0105] Determining a loss value of the preset speech synthesis model based on the identified reference speech and the predicted speech;

[0106] If the loss value is greater than or equal to a preset loss value, determining that the preset speech synthesis model has not converged;

[0107] If the loss value is less than a preset loss value, it is determined that the preset speech synthesis model has converged.

[0108] It should be noted that those skilled in the art will clearly understand that for the sake of convenience and brevity of description, the specific working process of the above-mentioned speech synthesis device can refer to the corresponding process in the aforementioned speech synthesis method embodiment, and will not be repeated here.

[0109] See also Figure 5 , Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0110] like Figure 5 As shown, the computer device 400 includes a processor 402 and a memory 403 connected via a system bus 401 , wherein the memory 403 may include a storage medium and an internal memory.

[0111] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any speech synthesis method.

[0112] The processor 402 is used to provide computing and control capabilities and support the operation of the entire computer device.

[0113] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any speech synthesis method.

[0114] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0115] It should be understood that the processor 402 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0116] In one embodiment, the processor 402 is configured to execute a computer program stored in a memory to implement the following steps:

[0117] Obtaining target text, reference audio, and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer, and a decoder layer, wherein the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder, and an embedding fusion module;

[0118] Performing speaker feature recognition on the reference audio through the speaker encoder to obtain a speaker embedding vector, and performing whole sentence level emotion embedding recognition on the reference audio through the sentence-level audio encoder to obtain a first emotion vector;

[0119] Performing frame-level deep emotion embedding recognition on the reference audio by the frame-level audio encoder to obtain a second emotion vector;

[0120] Performing vector fusion on the speaker embedding vector, the first emotion vector, and the second emotion vector by the embedding fusion module to obtain an emotion embedding vector;

[0121] Performing feature encoding on the target text through the text encoding layer to obtain a text embedding vector, and performing speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector;

[0122] The speech audio vector is outputted via the decoder layer to obtain a target speech corresponding to the target text.

[0123] In one embodiment, when implementing the performing of frame-level deep emotion embedding recognition on the reference audio by the frame-level audio encoder to obtain the second emotion vector, the processor 402 is configured to implement:

[0124] Segmenting the reference audio at a preset frame granularity to obtain a plurality of reference sub-audios;

[0125] Performing emotion embedding recognition on each of the reference sub-audios using the frame-level audio encoder to obtain a plurality of sub-emotion vectors;

[0126] The multiple sub-emotion vectors are fused to generate the second emotion vector, and the sub-emotion vectors in the second emotion vector are arranged along a time axis.

[0127] In one embodiment, when implementing the vector fusion of the speaker embedding vector, the first emotion vector, and the second emotion vector by the embedding fusion module to obtain the emotion embedding vector, the processor 402 is configured to implement:

[0128] The embedding fusion module inserts each sub-emotion vector in the second emotion vector into the corresponding time position in the first emotion vector according to the time axis, so as to adjust the emotion embedding information represented by the first emotion vector to obtain a third emotion vector;

[0129] Perform vector fusion on the speaker embedding vector and the third emotion vector to obtain the emotion embedding vector.

[0130] In one embodiment, when implementing the speech audio synthesis of the text embedding vector and the emotion embedding vector through the speech prediction layer to generate the speech audio vector, the processor 402 is configured to implement:

[0131] Performing speech generation on the text embedding vector through the prediction layer to obtain a candidate speech audio vector;

[0132] The phonemes represented by the candidate speech audio vector are adjusted according to the emotion embedding vector to generate the speech audio vector.

[0133] In one embodiment, before obtaining the target text, reference audio, and preset speech synthesis model, the processor 402 is further configured to:

[0134] Acquire a sample data set, the sample data set including a plurality of sample data, the sample data including a sample text, a sample reference audio, and a reference speech identified by the sample text;

[0135] Selecting a sample data from the sample data set as target sample data, training a preset speech synthesis model according to the target sample data, and determining whether the preset speech synthesis model converges;

[0136] If the preset speech synthesis model has not converged, adjust the model parameters of the preset speech synthesis model, and loop through the step of selecting a sample data from the sample data set as the target sample data until the preset speech synthesis model converges to obtain a speech synthesis model.

[0137] In one embodiment, the processor 402, when implementing the training of the preset speech synthesis model according to the target sample data and determining whether the preset speech synthesis model has converged, is configured to implement:

[0138] Performing speaker feature recognition on the sample reference audio through the speaker encoder to obtain a predicted speaker embedding vector, and performing sentence-level emotion embedding recognition on the sample reference audio through the sentence-level audio encoder to obtain a predicted first emotion vector;

[0139] Performing frame-level deep emotion embedding recognition on the sample reference audio by the frame-level audio encoder to obtain a predicted second emotion vector;

[0140] Performing vector fusion on the predicted speaker embedding vector, the predicted first emotion vector, and the predicted second emotion vector through the embedding fusion module to obtain a predicted emotion embedding vector;

[0141] Performing feature encoding on the sample text through the text encoding layer to obtain a predicted text embedding vector, and performing speech audio synthesis on the predicted text embedding vector and the predicted emotion embedding vector through the speech prediction layer to generate a predicted speech audio vector;

[0142] Performing speech output on the predicted speech audio vector through the decoder layer to obtain the predicted speech corresponding to the sample text;

[0143] Determine whether the preset speech synthesis model converges based on the identified reference speech and the predicted speech.

[0144] In one embodiment, when determining whether the preset speech synthesis model has converged based on the identified reference speech and the predicted speech, the processor 402 is configured to implement:

[0145] Determining a loss value of the preset speech synthesis model based on the identified reference speech and the predicted speech;

[0146] If the loss value is greater than or equal to a preset loss value, determining that the preset speech synthesis model has not converged;

[0147] If the loss value is less than a preset loss value, it is determined that the preset speech synthesis model has converged.

[0148] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the computer device described above can refer to the corresponding process in the aforementioned speech synthesis method embodiment, and will not be repeated here.

[0149] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the speech synthesis method of the present application.

[0150] The computer-readable storage medium may be an internal storage unit of the computer device described in the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may be non-volatile or volatile. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash memory card, etc., equipped on the computer device.

[0151] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0152] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0153] It should be understood that the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0154] It should also be understood that the term "and / or" used in this specification refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system that includes the element.

[0155] The serial numbers of the embodiments of the present application are for descriptive purposes only and do not represent the merits of the embodiments. The above description is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.

Claims

1. A speech synthesis method, characterized in that: include: Obtaining target text, reference audio, and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer, and a decoder layer, wherein the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder, and an embedding fusion module; Performing speaker feature recognition on the reference audio through the speaker encoder to obtain a speaker embedding vector, and performing whole sentence level emotion embedding recognition on the reference audio through the sentence-level audio encoder to obtain a first emotion vector; Performing frame-level deep emotion embedding recognition on the reference audio by the frame-level audio encoder to obtain a second emotion vector, where the second emotion vector includes multiple sub-emotion vectors; Performing vector fusion on the speaker embedding vector, the first emotion vector, and the second emotion vector by the embedding fusion module to obtain an emotion embedding vector; Performing feature encoding on the target text through the text encoding layer to obtain a text embedding vector, and performing speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector; Performing speech output on the speech audio vector through the decoder layer to obtain a target speech corresponding to the target text; The step of performing vector fusion on the speaker embedding vector, the first emotion vector, and the second emotion vector by the embedding fusion module to obtain the emotion embedding vector includes: The embedding fusion module inserts each sub-emotion vector in the second emotion vector into the corresponding time position in the first emotion vector according to the time axis, so as to adjust the emotion embedding information represented by the first emotion vector to obtain a third emotion vector; Perform vector fusion on the speaker embedding vector and the third emotion vector to obtain the emotion embedding vector.

2. The speech synthesis method according to claim 1, wherein: The step of performing frame-level deep emotion embedding recognition on the reference audio by the frame-level audio encoder to obtain a second emotion vector includes: Segmenting the reference audio at a preset frame granularity to obtain a plurality of reference sub-audios; Performing emotion embedding recognition on each of the reference sub-audios using the frame-level audio encoder to obtain a plurality of sub-emotion vectors; The multiple sub-emotion vectors are fused to generate the second emotion vector, and the sub-emotion vectors in the second emotion vector are arranged along a time axis.

3. The speech synthesis method according to claim 1, wherein: The performing speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector includes: Performing speech generation on the text embedding vector through the prediction layer to obtain a candidate speech audio vector; The phonemes represented by the candidate speech audio vector are adjusted according to the emotion embedding vector to generate the speech audio vector.

4. The speech synthesis method according to claim 1, wherein: Before obtaining the target text, reference audio, and preset speech synthesis model, the method further includes: Acquire a sample data set, the sample data set including a plurality of sample data, the sample data including a sample text, a sample reference audio, and a reference speech identified by the sample text; Selecting a sample data from the sample data set as target sample data, training a preset speech synthesis model according to the target sample data, and determining whether the preset speech synthesis model converges; If the preset speech synthesis model has not converged, adjust the model parameters of the preset speech synthesis model, and loop through the step of selecting a sample data from the sample data set as the target sample data until the preset speech synthesis model converges to obtain a speech synthesis model.

5. The speech synthesis method according to claim 4, wherein: The step of training a preset speech synthesis model according to the target sample data and determining whether the preset speech synthesis model has converged comprises: Performing speaker feature recognition on the sample reference audio through the speaker encoder to obtain a predicted speaker embedding vector, and performing sentence-level emotion embedding recognition on the sample reference audio through the sentence-level audio encoder to obtain a predicted first emotion vector; Performing frame-level deep emotion embedding recognition on the sample reference audio by the frame-level audio encoder to obtain a predicted second emotion vector; Performing vector fusion on the predicted speaker embedding vector, the predicted first emotion vector, and the predicted second emotion vector through the embedding fusion module to obtain a predicted emotion embedding vector; Performing feature encoding on the sample text through the text encoding layer to obtain a predicted text embedding vector, and performing speech audio synthesis on the predicted text embedding vector and the predicted emotion embedding vector through the speech prediction layer to generate a predicted speech audio vector; Performing speech output on the predicted speech audio vector through the decoder layer to obtain the predicted speech corresponding to the sample text; Determine whether the preset speech synthesis model converges based on the identified reference speech and the predicted speech.

6. The speech synthesis method according to claim 5, wherein: The determining, based on the identified reference speech and the predicted speech, whether the preset speech synthesis model converges includes: Determining a loss value of the preset speech synthesis model based on the identified reference speech and the predicted speech; If the loss value is greater than or equal to a preset loss value, determining that the preset speech synthesis model has not converged; If the loss value is less than a preset loss value, it is determined that the preset speech synthesis model has converged.

7. A speech synthesis device, characterized in that: The speech synthesis device includes an acquisition module and a generation module, wherein: The acquisition module is used to acquire the target text, reference audio and a preset speech synthesis model, wherein the preset speech synthesis model includes an audio coding layer, a text coding layer, a speech prediction layer and a decoder layer, and the audio coding layer includes a speaker encoder, a sentence-level audio encoder, a frame-level audio encoder and an embedding fusion module; The generating module is configured to: perform speaker feature recognition on the reference audio through a speaker encoder to obtain a speaker embedding vector, and perform sentence-level emotion embedding recognition on the reference audio through a sentence-level audio encoder to obtain a first emotion vector; Performing frame-level deep emotion embedding recognition on the reference audio through a frame-level audio encoder to obtain a second emotion vector, where the second emotion vector includes multiple sub-emotion vectors; Inserting each sub-emotion vector in the second emotion vector into a corresponding time position in the first emotion vector according to the time axis by the embedding fusion module to adjust the emotion embedding information represented by the first emotion vector to obtain a third emotion vector; Performing vector fusion on the speaker embedding vector and the third emotion vector to obtain the emotion embedding vector; Performing feature encoding on the target text through the text encoding layer to obtain a text embedding vector, and performing speech audio synthesis on the text embedding vector and the emotion embedding vector through the speech prediction layer to generate a speech audio vector; The speech audio vector is outputted via the decoder layer to obtain a target speech corresponding to the target text.

8. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the speech synthesis method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the speech synthesis method according to any one of claims 1 to 6 are implemented.

Images