Speech synthesis method, device, electronic device and storage medium

Abstract

The present disclosure provides a speech synthesis method, device, electronic device and storage medium. The speech synthesis method of an embodiment of the present disclosure may include: obtaining information of a first text and a first speaker, the first text carries a first tone label; generating a first tone acoustic feature corresponding to the first speaker and the first text through a pre-trained acoustic model, the first tone acoustic feature includes acoustic features corresponding to the first speaker and the first text and tone features corresponding to the first tone label; synthesizing the first tone acoustic feature into a first speech with tone, the tone of the first speech is consistent with the first tone label and the first speech has the pronunciation features of the first speaker; wherein the acoustic model is trained based on the corpus of M second speakers, the information of M second speakers and the information of the first speaker, and M is an integer greater than or equal to 1. The present disclosure adopts feature transfer technology to realize speech synthesis with tone, which can improve the naturalness and expressiveness of speech without reducing the quality of speech synthesis.

Description

Technical Field

[0001] The present disclosure relates to a speech synthesis method, apparatus, electronic device and storage medium. Background Art

[0002] Currently, end-to-end speech synthesis systems have achieved synthesized speech that is close to human speech in terms of sound quality, accuracy, and fluency. However, speech synthesized by these systems still lacks expressiveness, has a flat intonation, lacks cadence, and is less natural. Consequently, a significant amount of research is currently underway to improve the expressiveness of speech synthesis.

[0003] Currently, the primary approach to improving the naturalness and expressiveness of synthesized speech is to add paralinguistic features, such as accent, emotion, and tone, to synthesized speech. This approach also aims to make the synthesized speech more controllable. This not only enriches the emotional expression of speech and better emphasizes the speaker's perspective, but also significantly enhances the naturalness and expressiveness of speech synthesis.

[0004] Currently, most styled speech synthesis (i.e., speech synthesis with added tone features) requires a large amount of corpus annotated with style data. However, this is very time-consuming and labor-intensive, and speakers often lack the ability to accurately grasp tone features, resulting in poor corpus quality. Consequently, styled speech synthesis often yields poor results and even degrades the quality of the synthesized speech. However, synthesizing speech with tone is still desirable. Summary of the Invention

[0005] In order to solve at least one of the above technical problems, the present disclosure provides a speech synthesis method, apparatus, electronic device and storage medium.

[0006] A first aspect of the present disclosure provides a speech synthesis method, comprising:

[0007] Acquire a first text and information about a first speaker, wherein the first text carries a first tone tag;

[0008] Generating a first tone acoustic feature corresponding to the first speaker and the first text using a pre-trained acoustic model, wherein the first tone acoustic feature includes an acoustic feature corresponding to the first speaker and the first text and a tone feature corresponding to the first tone label;

[0009] synthesizing a first speech with tone using the first tone acoustic feature, wherein the tone of the first speech is consistent with the first tone label and the first speech has pronunciation characteristics of the first speaker;

[0010] The acoustic model is trained based on corpora of M second speakers, information of the M second speakers, and information of the first speaker, where M is an integer greater than or equal to 1.

[0011] In some possible implementations of the present disclosure, the corpus of the M second speakers includes: a second text; N tone tags, each tone tag represents a tone, and N is an integer greater than 1; N types of real speech whose content is consistent with the second text, each type of real speech has a tone and contains at least one real speech, and each of the real speech corresponds to one second speaker.

[0012] In some possible implementations of the present disclosure, the tone tag is text information indicating one of the following tones: interrogative tone; declarative tone; exclamatory tone.

[0013] In some possible implementations of the present disclosure, the acoustic model includes a first encoder, a second encoder and a decoder; the acoustic model is trained in the following manner: the first encoder is synchronously processed on the second text and N mood tags to obtain N mood text features, and the mood text features include the text features of the mood tags and the text features of the second text; the second encoder is synchronously processed on the information of the M second speakers and the information of the first speaker to obtain the features of the M second speakers and the features of the first speaker; the encoder is synchronously processed on the features of the M second speakers, the features of the first speaker and the N mood text features to obtain multiple second mood acoustic features; the parameters of the first encoder, the second encoder and the decoder are updated according to the multiple second mood acoustic features and the acoustic features of the corresponding real speech in the corpus.

[0014] In some possible implementations of the present disclosure, the decoder includes a preprocessing network, an attention network, a two-layer long short-term memory network, a linear projection layer, a post-processing network, and a decoding output network; during the execution of the training, the input data of the two-layer long short-term memory network, the linear projection layer, and the decoding output network all contain the features of the M second speakers and the features of the first speaker.

[0015] In some possible implementations of the present disclosure, synthesizing the first tone acoustic feature into the first speech with tone includes: synthesizing the first tone acoustic feature into the first speech with tone through a pre-trained vocoder; wherein the vocoder is trained based on the real voice of the first speaker.

[0016] A second aspect of the present disclosure provides a speech synthesis device, comprising:

[0017] an acquiring unit, configured to acquire a first text and information about a first speaker, wherein the first text carries a first tone tag;

[0018] a feature extraction unit, configured to generate a first tone acoustic feature corresponding to the first speaker and the first text using a pre-trained acoustic model, wherein the first tone acoustic feature includes an acoustic feature corresponding to the first speaker and the first text and a tone feature corresponding to the first tone label;

[0019] a speech generation unit, configured to synthesize a first speech with a tone from the first tone acoustic feature, wherein the tone of the first speech is consistent with the first tone label and the first speech has a pronunciation feature of the first speaker;

[0020] The acoustic model is trained based on the corpus of M second speakers, the information of the M second speakers, and the information of the first speaker.

[0021] In some possible implementations of the present disclosure, the corpus of the M second speakers includes: a second text; N tone tags, each tone tag represents a tone, and N is an integer greater than 1; N types of real speech whose content is consistent with the second text, each type of real speech has a tone and contains at least one real speech, and each of the real speech corresponds to one second speaker.

[0022] A third aspect of the present disclosure provides an electronic device, including:

[0023] a memory storing execution instructions; and

[0024] A processor executes the execution instructions stored in the memory, so that the processor executes the above-mentioned speech synthesis method.

[0025] A fourth aspect of the present disclosure provides a readable storage medium, wherein the readable storage medium stores execution instructions, and the execution instructions are used for the above-mentioned speech synthesis method when executed by a processor.

[0026] In the embodiment of the present disclosure, an acoustic model trained with the corpus of a second speaker is used to generate the tone acoustic features of the first speaker and speech with tone is synthesized based on the tone acoustic features. Different tone features learned based on the corpus of multiple second speakers are transferred to the speech synthesis of the first speaker. The speech synthesis with tone of the first speaker can be achieved without pre-labeling the speech data with tone of the first speaker. This not only effectively reduces the complexity of model training, reduces the amount of data during model training, saves manpower, improves model training efficiency, reduces model training costs and improves the generalization ability of the model, but also improves the naturalness and expressiveness of the synthesized speech without reducing the quality of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0028] Figure 1 It is a flowchart of a speech synthesis method according to some embodiments of the present disclosure.

[0029] Figure 2 It is a schematic block diagram of the structure of a speech synthesis device using a hardware implementation of a processing system according to an embodiment of the present disclosure.

[0030] Description of Reference Numerals

[0031] 200 Speech Synthesis Model

[0032] 300 bus

[0033] 400 processors

[0034] 500 memory

[0035] 600 Various other circuits DETAILED DESCRIPTION

[0036] The present disclosure will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the relevant content and are not intended to limit the present disclosure. It should also be noted that, for ease of description, only the portions relevant to the present disclosure are shown in the accompanying drawings.

[0037] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0038] Unless otherwise stated, the exemplary embodiments / examples shown are to be understood as providing exemplary features of various details of some ways in which the technical concepts of the present disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of the various embodiments / examples may be further combined, separated, interchanged, and / or rearranged without departing from the technical concepts of the present disclosure.

[0039] The use of cross hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise indicated, the presence or absence of cross hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the order described. In addition, the same figure numbers represent the same components.

[0040] When a component is referred to as being “on,” “over,” “connected to,” or “coupled to” another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being “directly on,” “directly connected to,” or “directly coupled to” another component, there are no intervening components present. For this purpose, the term “connected” may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0041] The terms used herein are for the purpose of describing specific embodiments and are not intended to be restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, the features, integral bodies, steps, operations, parts, assemblies and / or their groups stated are indicated, but the presence or addition of one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups is not excluded. It should also be noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values and / or values provided that will be recognized by those of ordinary skill in the art.

[0042] Explanation of terms in this article:

[0043] Text-to-speech (TTS): A technology that converts computer-generated or externally input text information into understandable and fluent human speech and outputs it.

[0044] Feature transfer: Transfer the knowledge learned in task A to task B to improve the generalization performance on task B.

[0045] Tone: As a crucial form of expression in speech, tone provides crucial emotional color. Tone intuitively reveals the speaker's subjective attitude and represents heartfelt behavior and emotion. Commonly used tones in Chinese can be categorized into three main types: declarative, interrogative, and exclamatory. Changes in tone are reflected in the pitch, strength, duration, and speed of the voice, with variations in intonation being particularly significant. Therefore, tone is closely related to rhythm. Speech prosodic characteristics primarily include fundamental frequency, duration, and intensity, with different tones manifesting themselves in distinct rhythmic characteristics.

[0046] Neural Network Vocoder (LPCNet): A neural network-based vocoder that can synthesize acoustic features such as mel-spectrograms into audio.

[0047] Text encoder: This consists of a sequentially connected Character Embedding convolutional neural network and a bidirectional long short-term memory (bi-directional LSTM) network. The convolutional neural network can include three sequentially connected convolutional layers. This encoder can be used to encode text into vectors.

[0048] The decoder consists of a pre-processing network (Pre-Net), an attention network (Stepwise Attention), two stacked LSTM layers (two LSTM layers), a linear projection layer, a linear projection + sigmoid activation layer, a post-processing network (Post-Net), and a decoder output network. The outputs of the pre-processing network and the attention network are fed into the two-layer LSTM layer, which in turn feeds into the linear projection layer and the activation linear projection layer. The output of the linear projection layer feeds into the post-processing network, which in turn feeds into the decoder output network. The output of the decoder output network serves as the decoder output, i.e., acoustic features such as mel-spectrograms. For example, the post-processing network (Post-Net) can consist of five convolutional layers, which can modify the spectral frames output by the linear projection layer. The decoder is typically implemented as an autoregressive recurrent neural network.

[0049] Herein, the first text refers to the text that needs to be synthesized into speech, and the second text refers to the text used to train the acoustic model. The content of the first text and / or the second text may include but is not limited to characters, words, sentences, paragraphs, etc.

[0050] Figure 1 A flowchart of a speech synthesis method according to some embodiments of the present disclosure is shown.

[0051] like Figure 1 As shown, the speech synthesis method of the embodiment of the present disclosure may include:

[0052] Step S12, obtaining information of a first text and a first speaker, wherein the first text carries a first tone tag;

[0053] The first tone label refers to a tone label corresponding to a certain tone category selected by the user, which corresponds to the first text.

[0054] Step S14: generating a first tone acoustic feature corresponding to the first speaker and the first text using a pre-trained acoustic model, wherein the first tone acoustic feature includes an acoustic feature corresponding to the first speaker and the first text and a tone feature corresponding to the first tone label;

[0055] The acoustic model is trained based on the corpus of M second speakers, the information of M second speakers, and the information of the first speaker, where M is an integer greater than or equal to 1.

[0056] In some implementations, the speaker information may be, but is not limited to, the speaker's number or other identification information, as long as the information can uniquely represent a speaker. The present disclosure does not limit the specific form of the speaker information.

[0057] In some implementations, the corpus of the M second speakers may include the following three items:

[0058] 1) Second text;

[0059] 2) N tone labels, each tone label represents a tone, and N is an integer greater than 1;

[0060] 3) N types of real speech whose content is consistent with the second text, each type of real speech has a tone and includes at least one real speech, and each real speech corresponds to a second speaker.

[0061] In the disclosed embodiments, the tone tag can be, but is not limited to, text information indicating a tone category. This allows for simultaneous character embedding and other processing of the tone tag and text, without requiring modifications to the network structure of the text encoding portion of the existing acoustic model. This allows for easy implementation, low cost, and ease of use.

[0062] Tone categories can be flexibly set based on the needs of the scenario or actual application. Taking Chinese as an example, tone categories can include but are not limited to interrogative tone, declarative tone, and exclamatory tone. Correspondingly, tone tags can include but are not limited to interrogative tone tags, declarative tone tags, and exclamatory tone tags.

[0063] For example, a mood tag can be represented as textual information such as punctuation marks. Still using Chinese as an example, an interrogative mood tag can be represented by a question mark "?", a declarative mood tag by a period ".", and an exclamatory mood tag by an exclamation mark "!". Thus, when synthesizing speech with a mood, one simply adds the punctuation mark corresponding to the mood to the end of the text.

[0064] It should be noted that, among the corpus of the M second speakers, as long as the corpus of the main second speaker (for example, any one of the second speakers) is complete, it is not necessary for the corpus of all the second speakers to be complete. The completeness here means that in addition to containing the second text and all N tone tags, the corpus also contains N real voices of the main second speaker, the content of the N real voices is the same as the content of the second text and the tone of the N real voices corresponds one to one with the tone category represented by the N tone tags. In specific applications, the main second speaker can be selected as needed. The main second speaker can be one speaker or two or more speakers. The speaker whose tone features of the real voice are accurate and of higher quality is preferably used as the main second speaker.

[0065] For the other second speakers among the M second speakers except the main second speaker, their corpus can only include the real tone of the corresponding part of the tone category, the corresponding tone label and the second text. Preferably, the corpus has good real speech quality and accurate tone expression.

[0066] It can be seen that the embodiment of the present disclosure can effectively transfer the tone characteristics of the second speaker to the first speaker without tone annotations through the parallel corpus of the second speaker and the information of the first speaker without tone annotations, so that only less corpus is needed to achieve speech synthesis with tone for the unannotated speaker.

[0067] In some implementations, step S14 may include:

[0068] Step a1: obtaining a first tone text feature using the first text and the first tone tag. The first tone text feature may include a tone feature corresponding to the first tone tag and a text feature corresponding to the first text.

[0069] In some implementations, step a1 may include: synchronously and sequentially performing character embedding processing, convolution processing, and bidirectional LSTM processing on the first text and the first mood tag, thereby obtaining a vector serving as a feature of the first mood text.

[0070] In specific applications, during the training phase, a first encoder can be set in the acoustic model and trained with the corpus of the second speaker. In this step a1, a vector representing the first tone text feature can be obtained by running the first encoder or executing the processing of the first encoder. Exemplarily, the model structure of the first encoder can be the same as the text encoder of the previous speech synthesis model Tacotron2. Of course, the model architecture of the first encoder is not limited to this, and can be replaced with other similar machine learning models as needed in specific applications.

[0071] Step a2, obtaining the first speaker's characteristics using the first speaker's information;

[0072] In some implementations, step a2 may include: performing an embedding process, etc., on the information of the first speaker to obtain a vector serving as a feature of the first speaker.

[0073] In specific applications, in the training phase, a second encoder can be set in the acoustic model and trained by the information of the second speaker, the information of the first speaker, etc. In this step a2, a vector representing the first speaker's features can be obtained by running the second encoder or executing the processing of the second encoder. Exemplarily, the second encoder can be implemented by a machine learning model such as a character embedding network or other similar functions. Similarly, the model structure of the second encoder, etc., is not limited in the disclosed embodiment.

[0074] Step a3: Obtain a first tone acoustic feature based on the first tone text feature and the first speaker feature.

[0075] In specific applications, it is possible to obtain the first tone acoustic feature by setting a decoder in the acoustic model and training it through the corpus of the second speaker, the information of the second speaker, and the information of the first speaker during the training phase. In this step a3, a vector or tensor representing the first tone acoustic feature can be obtained by running the decoder or executing the processing of the decoder. Exemplarily, the model structure of the decoder can be the same as the decoder (decoder) in the speech synthesis model Tacotron2 described above. Similarly, the model structure of the decoder can also be changed or replaced according to actual needs.

[0076] In some implementations, step a3 may include:

[0077] Step a31, generating a first context feature based on the first tone text feature, where the first context feature represents a context feature of the first sentence;

[0078] For example, the association between the intrinsic elements in the first tone text feature can be extracted through the attention network (Stepwise Attention) in the decoder, thereby obtaining the first context feature corresponding to the first tone text feature.

[0079] Step a32, generating a current acoustic spectrum frame according to the first context feature, the preprocessed previous acoustic spectrum frame, and the first speaker feature;

[0080] For example, the previous acoustic spectrum frame may be preprocessed by the preprocessing network in the decoder, and the first context feature, the preprocessed previous acoustic spectrum frame, and the first speaker feature may be processed by the double-layer LSTM in the decoder to generate the current acoustic spectrum frame.

[0081] Step a33, performing linear projection processing and post-processing on the first speaker feature and the current acoustic spectrum frame to obtain a fine-tuned current acoustic spectrum frame;

[0082] For example, a linear projection layer may be used to perform linear projection processing on the first speaker feature and the current acoustic spectrum frame, and then a post-processing network may be used to post-process the result of the linear projection processing to obtain a fine-tuned current acoustic spectrum frame.

[0083] In step a34, the first speaker feature, the result of the linear projection processing, and the fine-tuned current acoustic spectrum frame are superimposed to obtain a mel spectrogram, which is the first tone acoustic feature.

[0084] For example, the first speaker feature, the result of the linear projection processing, and the fine-tuned current acoustic spectrum frame can be superimposed through the decoding output network.

[0085] As can be seen above, while generating a tone acoustic feature that includes both tone and acoustic features, the first speaker's characteristics are incorporated into the tone acoustic feature at multiple stages of the generation process. This allows for better integration of the first speaker's characteristics into the tone acoustic feature, resulting in the final first tone acoustic feature incorporating the speaker's characteristics, tone characteristics, and text characteristics. This allows for the synthesis of high-quality speech that accurately expresses tone and conforms to the speaker's pronunciation characteristics.

[0086] In some implementations, the acoustic model includes a first encoder, a second encoder, and a decoder. The acoustic model can be trained by the following steps:

[0087] Step b1, synchronously performing processing of the first encoder on the second text and the N mood tags to obtain N mood text features, where the mood text features include text features of the mood tags and text features of the second text;

[0088] Step b2, synchronously performing a second encoder process on the information of the M second speakers and the information of the first speaker to obtain features of the M second speakers and features of the first speaker;

[0089] Step b3, synchronously performing encoder processing on the M second speaker features, the first speaker features, and the N tone text features to obtain a plurality of second tone acoustic features;

[0090] Each second tone acoustic feature corresponds to the second text, a speaker (the speaker may be a second speaker or a first speaker) and a tone label.

[0091] Step b4: updating the parameters of the first encoder, the second encoder, and the decoder according to the plurality of second tone acoustic features and the acoustic features of the corresponding real speech in the corpus.

[0092] Specifically, an algorithm such as gradient descent can be used to update the parameters using a predetermined loss function. For example, the loss function of the speech synthesis model Tacotron2 can be used.

[0093] Therefore, by using the text and the tone label as the input of the first encoder at the same time, the first encoder of the acoustic module can learn the characteristic differences between different tones. By using the parallel corpus of the second speaker and the information of the first speaker for joint training, the tone characteristics of the second speaker can be effectively transferred to the first speaker, so that the acoustic model has the ability to generate the tone acoustic characteristics of any speaker without labeled data (i.e., corpus).

[0094] In some implementation modes, the decoder may include a pre-processing network, an attention network, a double-layer long short-term memory network, a linear projection layer, a post-processing network, and a decoded network. In the training process of the acoustic model, that is, in step b3, the input data of the double-layer long short-term memory network, the linear projection layer, and the decoded output network all contain the features of M second speakers and the features of the first speaker. As can be seen, by modeling the speaker information and splicing the speaker information to the multi-layer network, the cross-layer splicing of the speaker information is achieved, and the acoustic model can be allowed to better learn the pronunciation differences between different speakers, thereby more effectively migrating the tone features to the first speaker without tone annotation.

[0095] In some embodiments, the decoder can be implemented as a self-circulating neural network, which can also include an activation function linear projection layer, which can be used to generate a stop token using the acoustic spectrum frame output by the double-layer LSTM so that the decoder can terminate processing based on the stop token.

[0096] Step S16: synthesize the first tone acoustic feature into a first speech with tone, where the tone of the first speech is consistent with the first tone label and the first speech has the pronunciation characteristics of the first speaker.

[0097] In some implementations, the first tone acoustic feature can be synthesized into the first speech with tone using a pre-trained vocoder. The vocoder can be trained based on the real speech of the first speaker. For example, the vocoder can be, but is not limited to, the LPCNet described above.

[0098] The disclosed embodiment uses an acoustic model trained with corpora from other speakers and a vocoder trained with the current speaker's real speech to achieve speech synthesis with intonation for the current speaker. This eliminates the need to pre-label a large amount of corpora for the current speaker. By using feature transfer technology, speech synthesis with intonation for any unlabeled speaker is achieved. The model training complexity is low, and the amount of data required for training the model is small. This not only significantly reduces production and labor costs, but also provides a strong generalization capability and a wide range of applications. Furthermore, it can improve the naturalness and expressiveness of synthesized speech without compromising speech synthesis quality.

[0099] Figure 2 1 is a schematic block diagram of the structure of a speech synthesis apparatus 200 implemented in hardware using a processing system according to an embodiment of the present disclosure.

[0100] The device may include corresponding modules for executing each or several steps in the above flowchart. Therefore, each step or several steps in the above flowchart may be executed by a corresponding module, and the device may include one or more of these modules. The module may be one or more hardware modules specifically configured to execute the corresponding steps, or implemented by a processor configured to execute the corresponding steps, or stored in a computer-readable medium for execution by a processor, or implemented by some combination thereof.

[0101] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 300 connects various circuits including one or more processors 400, memory 500, and / or hardware modules. The bus 300 can also connect various other circuits 600 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0102] Bus 300 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component Architecture (EISA) bus. Buses can be classified as address buses, data buses, control buses, and the like. For ease of illustration, this figure shows only one connecting line, but this does not imply that there is only one bus or only one type of bus.

[0103] Any process or method description in the flowchart or otherwise described herein can be understood to represent a module, fragment or portion of code including one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong. The processor performs the various methods and processes described above. For example, the method embodiments of the present disclosure can be implemented as a software program that is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0104] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0105] For the purposes of this specification, a "readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples (a non-exhaustive list) of readable storage media include the following: an electrical connection having one or more wires (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium can even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a memory.

[0106] It should be understood that various parts of the present disclosure can be implemented using hardware, software, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement the hardware: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0107] Those skilled in the art will understand that all or part of the steps of the above-mentioned implementation method can be accomplished by instructing related hardware through a program, and the program can be stored in a readable storage medium. When the program is executed, it includes one or a combination of the steps of the method implementation method.

[0108] Furthermore, the functional units in the various embodiments of the present disclosure may be integrated into a single processing module, each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules. If the integrated modules are implemented as software functional modules and sold or used as independent products, they may also be stored in a readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0109] Figure 2 2 is a schematic structural diagram of a speech synthesis device 200 according to an embodiment of the present disclosure.

[0110] like Figure 2 As shown, the speech synthesis device 200 may include:

[0111] An acquiring unit 202 is configured to acquire a first text and information about a first speaker, wherein the first text carries a first tone tag;

[0112] A feature extraction unit 204 is configured to generate a first tone acoustic feature corresponding to the first speaker and the first text using a pre-trained acoustic model, wherein the first tone acoustic feature includes an acoustic feature corresponding to the first speaker and the first text and a tone feature corresponding to the first tone label;

[0113] The speech generation unit 206 is configured to synthesize a first speech with a tone from the first tone acoustic feature, wherein the tone of the first speech is consistent with the first tone label and the first speech has a pronunciation feature of the first speaker;

[0114] The acoustic model is trained based on the corpus of M second speakers, the information of the M second speakers, and the information of the first speaker.

[0115] In some implementations, the speech generation unit 206 may be specifically configured to synthesize the first tone acoustic feature into the first speech with tone using a pre-trained vocoder; wherein the vocoder is trained based on the real speech of the first speaker.

[0116] The present disclosure also provides an electronic device, including: a memory, the memory storing execution instructions; and a processor or other hardware module, the processor or other hardware module executing the execution instructions stored in the memory, so that the processor or other hardware module performs the above-mentioned speech synthesis method.

[0117] The present disclosure also provides a readable storage medium, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the above-mentioned speech synthesis method.

[0118] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms are not necessarily the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in a suitable manner in any one or more embodiments / methods or examples. In addition, those skilled in the art may combine and combine different embodiments / methods or examples described in this specification and the features of different embodiments / methods or examples, unless they are mutually inconsistent.

[0119] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0120] Those skilled in the art will appreciate that the above embodiments are merely intended to clearly illustrate the present disclosure and are not intended to limit the scope of the present disclosure. Other changes or modifications may be made based on the above disclosure, and such changes or modifications are still within the scope of the present disclosure.

Claims

1. A speech synthesis method, characterized in that: include: Acquire a first text and information about a first speaker, wherein the first text carries a first tone tag; Generating a first tone acoustic feature corresponding to the first speaker and the first text using a pre-trained acoustic model, wherein the first tone acoustic feature includes an acoustic feature corresponding to the first speaker and the first text and a tone feature corresponding to the first tone label; synthesizing a first speech with tone using the first tone acoustic feature, wherein the tone of the first speech is consistent with the first tone label and the first speech has pronunciation characteristics of the first speaker; The acoustic model is trained based on corpora of M second speakers, information of the M second speakers, and information of the first speaker, where M is an integer greater than or equal to 1. The corpus of the M second speakers includes: Second text; N tone tags, each tone tag represents a tone, and N is an integer greater than 1; N types of real speech with content consistent with the second text, each type of real speech having a tone and including at least one real speech, each real speech corresponding to one second speaker; The acoustic model includes a first encoder, a second encoder and a decoder; The acoustic model is trained in the following way: Synchronously performing processing of the first encoder on the second text and the N mood tags to obtain N mood text features, where the mood text features include text features of the mood tags and text features of the second text; Synchronously performing a second encoder process on the information of the M second speakers and the information of the first speaker to obtain features of the M second speakers and features of the first speaker; Synchronously performing the encoder processing on the M second speaker features, the first speaker features, and the N tone text features to obtain a plurality of second tone acoustic features; The parameters of the first encoder, the second encoder, and the decoder are updated according to the multiple second tone acoustic features and the acoustic features of the corresponding real speech in the corpus.

2. The speech synthesis method according to claim 1, wherein: The tone tag is text information indicating one of the following tones: interrogative tone; Indicative mood; Exclamatory tone.

3. The speech synthesis method according to claim 1, wherein: The decoder includes a preprocessing network, an attention network, a two-layer long short-term memory network, a linear projection layer, a post-processing network and a decoding output network; during the execution of the training, the input data of the two-layer long short-term memory network, the linear projection layer and the decoding output network all contain the features of the M second speakers and the features of the first speaker.

4. The speech synthesis method according to claim 1, wherein: The method of synthesizing the first tone acoustic feature into the first speech with tone includes: synthesizing the first tone acoustic feature into the first speech with tone through a pre-trained vocoder; wherein the vocoder is trained based on the real speech of the first speaker.

5. A speech synthesis device, characterized in that: include: an acquiring unit, configured to acquire a first text and information about a first speaker, wherein the first text carries a first tone tag; a feature extraction unit, configured to generate a first tone acoustic feature corresponding to the first speaker and the first text using a pre-trained acoustic model, wherein the first tone acoustic feature includes an acoustic feature corresponding to the first speaker and the first text and a tone feature corresponding to the first tone label; a speech generation unit, configured to synthesize a first speech with a tone from the first tone acoustic feature, wherein the tone of the first speech is consistent with the first tone label and the first speech has a pronunciation feature of the first speaker; The acoustic model is trained based on the corpus of M second speakers, the information of the M second speakers, and the information of the first speaker; The corpus of the M second speakers includes: Second text; N tone tags, each tone tag represents a tone, and N is an integer greater than 1; N types of real speech with content consistent with the second text, each type of real speech having a tone and including at least one real speech, each real speech corresponding to one second speaker; The acoustic model includes a first encoder, a second encoder and a decoder; The acoustic model is trained in the following way: Synchronously performing processing of the first encoder on the second text and the N mood tags to obtain N mood text features, where the mood text features include text features of the mood tags and text features of the second text; Synchronously performing a second encoder process on the information of the M second speakers and the information of the first speaker to obtain features of the M second speakers and features of the first speaker; Synchronously performing the encoder processing on the M second speaker features, the first speaker features, and the N tone text features to obtain a plurality of second tone acoustic features; The parameters of the first encoder, the second encoder, and the decoder are updated according to the multiple second tone acoustic features and the acoustic features of the corresponding real speech in the corpus.

6. The speech synthesis device according to claim 5, characterized in that The speech generation unit is specifically configured to synthesize the first tone acoustic feature into the first speech with tone through a pre-trained vocoder; wherein the vocoder is trained based on the real speech of the first speaker.

7. An electronic device, characterized in that: include: a memory storing execution instructions; as well as A processor, wherein the processor executes the execution instruction stored in the memory, so that the processor executes the speech synthesis method according to any one of claims 1 to 4.

8. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which are used to implement the speech synthesis method according to any one of claims 1 to 4 when executed by a processor.

Images