Self-adaptive emotion-driven timbre clone character-to-speech method and device

Abstract

The invention relates to an adaptive emotion-driven timbre clone character-to-speech method and device. Segmenting the target text according to sentences, extracting semantic features, and splitting each sentence into a target phoneme sequence; the method comprises the following steps: splitting a reference text into reference phoneme sequences, splicing the reference phoneme sequences with each target phoneme sequence to obtain a combined phoneme sequence, processing a reference audio to obtain a reference voice feature and a frequency spectrum, processing based on a semantic feature corresponding to a sentence, the combined phoneme sequence and the reference voice feature to obtain a corresponding voice feature, and extracting a corresponding emotional feature from each sentence semantic feature, and processing based on the speech feature, the emotional feature and the target phoneme sequence corresponding to each sentence and the frequency spectrum to obtain a speech signal corresponding to the target text. And extracting emotional features from the semantic features of the target text, enabling the emotional colors of the voice signals to fit the text, and automatically adjusting the emotional colors of the voice signals based on the mapping relationship between the preset output features and the emotional features.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and more specifically, to an adaptive emotion-driven timbre cloning text-to-speech method and device. Background Art

[0002] Although the current voice cloning TTS (Text To Speech) technology on the market has made some progress in achieving personalized speech synthesis, it still faces several key challenges and shortcomings.

[0003] On the one hand, there are limitations to emotional expression. Although some existing technologies attempt to incorporate emotional elements to improve the realism and expressiveness of synthesized speech, these methods generally have the problem of single or unnatural emotional expression. The sound emotions generated by most systems do not match the emotions of the input text content, resulting in the synthesized speech sounding monotonous and lacking in vividness. For those systems that support emotion-driven, users or developers are often required to manually specify corresponding emotional labels or parameters for each sentence of text. This process is cumbersome and inefficient, which seriously limits the smoothness of the user experience and the ability to interact naturally.

[0004] On the other hand, the model training data requirements are high. Traditional voice cloning methods usually require obtaining a large number of audio samples of specific individuals as training data. This means that in order to provide customized voice services for each user, an independent model needs to be built. This not only greatly increases the consumption of computing resources, but also prolongs the time cycle for service deployment. The ability to quickly respond to users' personalized needs is limited, and it is impossible to efficiently meet the market's high requirements for immediacy and flexibility. Summary of the invention

[0005] In order to solve at least one of the above-mentioned defects, the present invention application proposes an adaptive emotion-driven timbre cloning text-to-speech method and device for solving the problem of mismatch between speech emotional color and text.

[0006] According to a first aspect of the present application, an adaptive emotion-driven timbre cloning text-to-speech method is provided, comprising:

[0007] Divide the target text into at least one sentence, extract the semantic features of each sentence, and split each sentence into a target phoneme sequence;

[0008] Splitting the reference text into a reference phoneme sequence, and combining the reference phoneme sequence with each of the target phoneme sequences to obtain a combined phoneme sequence for each sentence;

[0009] Performing feature extraction and spectrum extraction on the reference audio corresponding to the reference text to obtain reference speech features and reference audio spectrum;

[0010] Processing and decoding based on the semantic features and combined phoneme sequences corresponding to each sentence and the reference speech features to obtain speech features of each sentence;

[0011] Extracting the sentiment feature of each sentence from the semantic feature of each sentence;

[0012] Based on the speech features, emotional features and target phoneme sequence corresponding to each sentence and the reference audio spectrum, feature audio conversion is performed to obtain a speech signal corresponding to the target text.

[0013] Preferably, the sentiment feature of each sentence is extracted from the semantic feature of each sentence, specifically including:

[0014] Adjusting the feature dimension of the semantic feature of each sentence, and extracting corresponding basic features from the semantic features with the adjusted feature dimension;

[0015] Performing residual connection processing on the basic features, adding the processed basic features to the basic features before the processing, and performing dimensionality reduction processing on the added result to obtain intermediate output features;

[0016] Performing residual connection processing on the intermediate output features, adding the intermediate output features after processing to the intermediate output features before processing, and performing dimensionality reduction processing on the addition result to obtain the output features of each sentence;

[0017] Based on the mapping relationship between the preset output features and the sentiment features, sentiment classification is performed on the output features of each sentence to obtain the sentiment features of each sentence;

[0018] Preferably, the adjustment of the feature dimension of the semantic features of each sentence and the extraction of the corresponding basic features from the semantic features with adjusted feature dimensions are achieved through an input processing layer, and the input processing layer is composed of a convolutional layer, a batch normalization layer and a ReLU activation function layer connected in sequence.

[0019] Preferably, the performing residual connection processing on the basic features, adding the processed basic features to the basic features before the processing, and performing dimensionality reduction processing on the added results to obtain intermediate output features is achieved by a first residual connection layer and a first feature dimensionality reduction layer connected to the first residual connection layer;

[0020] and / or,

[0021] The performing residual connection processing on the intermediate output features, adding the intermediate output features after the processing to the intermediate output features before the processing, and performing dimensionality reduction processing on the addition result to obtain the output features of each sentence is achieved by a second residual connection layer and a second feature dimensionality reduction layer connected to the second residual connection layer;

[0022] The first residual connection layer and the second residual connection layer are each composed of a convolution layer, a batch normalization layer, a ReLU activation function layer, a convolution layer, a batch normalization layer, and a ReLU activation function layer connected in sequence;

[0023] The first feature dimension reduction layer and the second feature dimension reduction layer are both composed of a convolution layer, a batch normalization layer and a ReLU activation function layer connected in sequence.

[0024] Preferably, the sentiment classification of the output features of each sentence based on the mapping relationship between the preset output features and the sentiment features to obtain the sentiment features of each sentence is achieved through a fully connected classification layer.

[0025] Preferably, the step of dividing the target text into at least one sentence, extracting the semantic features of each sentence, and splitting each sentence into a target phoneme sequence specifically includes:

[0026] The target text is divided into at least one sentence, and each sentence is input into the text processing network to extract deep semantic information and obtain the semantic features of each sentence;

[0027] Split each sentence to obtain the target phoneme sequence for each sentence;

[0028] Splitting the reference text into a reference phoneme sequence, and combining the reference phoneme sequence with each of the target phoneme sequences to obtain a combined phoneme sequence for each sentence;

[0029] and / or,

[0030] The feature extraction and spectrum extraction of the reference audio corresponding to the reference text to obtain the reference speech feature and the reference audio spectrum specifically includes:

[0031] An audio processing network is used to extract features of a reference audio corresponding to the reference text to obtain reference speech features, and a spectrum is extracted from the reference audio to obtain a reference audio spectrum.

[0032] Preferably, the processing and decoding based on the semantic features and combined phoneme sequences corresponding to each sentence and the reference speech features to obtain the speech features of each sentence are specifically:

[0033] Converting the combined phoneme sequence of each sentence into corresponding phoneme features in a high-dimensional vector space;

[0034] Performing dimension transformation on the semantic features corresponding to each of the sentences respectively to obtain transformed semantic features with the same dimension as the phoneme features;

[0035] The transformed semantic features corresponding to each sentence are respectively fused with the corresponding phoneme features to obtain at least one first fused feature;

[0036] Performing position encoding on the at least one first fused feature respectively to obtain at least one second fused feature;

[0037] Acquire the reference speech feature as the first speech feature, and predefine a stop feature EOS, wherein the stop feature EOS is used as a determination condition for stopping the loop;

[0038] Multiple decoding is performed through loop iteration. The specific process of each decoding loop includes:

[0039] Embedding and position encoding the first speech feature to generate a coded speech feature;

[0040] Combining the at least one second fusion feature with the coded speech feature to generate at least one combined feature;

[0041] Constructing an attention mask according to the length of the stitched feature, and decoding the corresponding stitched feature using the attention mask to generate a decoding output;

[0042] Performing full connection processing on the decoded output to obtain the intermediate speech features of this cycle;

[0043] Combining the intermediate speech feature with the first speech feature to generate a second speech feature;

[0044] Among them, if the second speech feature does not contain the stop feature EOS, the second speech feature is used as the first speech feature and the next decoding cycle is performed; if the second speech feature contains the stop feature EOS, the cycle is stopped and the second speech feature is used as the speech feature of the corresponding sentence.

[0045] Preferably, the feature audio conversion process is performed based on the speech features, emotional features and target phoneme sequence corresponding to each sentence, and the reference audio spectrum to obtain a speech signal corresponding to the target text, specifically including:

[0046] Combine the speech features and emotional features corresponding to each sentence to obtain the speech emotional features of each sentence;

[0047] Quantifying the speech emotion features to obtain the quantized features of each sentence;

[0048] Processing the reference audio spectrum to obtain embedded features of the reference audio;

[0049] Encoding the quantized features and target phoneme sequence corresponding to each sentence, as well as the embedded features, to generate a mean, variance, and mask of the latent variables as an intermediate representation for speech synthesis;

[0050] The intermediate representation is decoded to generate a speech signal corresponding to the target text.

[0051] According to a second aspect of the present application, there is provided an adaptive emotion-driven timbre cloning text-to-speech device, comprising:

[0052] A target text processing module is used to segment the target text into at least one sentence, extract the semantic features of each sentence, and split each sentence into a target phoneme sequence;

[0053] A reference text processing module, used for splitting the reference text into a reference phoneme sequence, and combining the reference phoneme sequence with the target phoneme sequence of each sentence to obtain a combined phoneme sequence of each sentence;

[0054] A reference audio processing module is used to perform feature extraction and spectrum extraction on the reference audio corresponding to the reference text to obtain reference speech features and reference audio spectrum;

[0055] A speech feature acquisition module, used for processing and decoding based on the semantic features and combined phoneme sequences corresponding to each sentence and the reference speech features to obtain the speech features of each sentence;

[0056] The sentiment feature acquisition module is used to extract the sentiment feature of each sentence from the semantic features of each sentence;

[0057] The feature audio conversion module is used to perform feature audio conversion based on the speech features, emotional features and target phoneme sequence corresponding to each sentence, as well as the reference audio spectrum, to obtain a speech signal corresponding to the target text.

[0058] According to a third aspect of the present application, an electronic device is provided, including:

[0059] a memory for storing one or more computer programs;

[0060] The processor, when the one or more computer programs are executed by the processor, implements the adaptive emotion-driven timbre cloning text-to-speech method of the first aspect mentioned above.

[0061] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement an adaptive emotion-driven timbre cloning text-to-speech method according to the first aspect when executed.

[0062] Based on any of the above aspects, an adaptive emotion-driven timbre cloning text-to-speech method and device provided in an embodiment of the present application divides a target text into multiple sentences, extracts semantic features and target phoneme sequences of each sentence, extracts reference phoneme sequences of reference texts, combines the target phoneme sequences and reference phoneme sequences to obtain a combined phoneme sequence, extracts speech features and audio spectra of reference audio corresponding to the reference text, decodes to obtain speech features, extracts emotional features of the semantic features of each sentence, performs feature audio conversion based on the speech features, emotional features and target phoneme sequences corresponding to each sentence, and the reference audio spectrum, to obtain a speech signal corresponding to the target text; by extracting emotional features from the semantic features of the target text, the emotional color of the speech signal is made to fit the text more closely, and fast and high-quality synthesis of speech with unique timbre features is achieved, thereby improving the convenience and personalization of the service; based on the mapping relationship between preset output features and emotional features, the emotional color of the speech signal is adaptively adjusted, and diversified emotions are incorporated into the speech signal, significantly enhancing the realism and expressiveness of the speech, and making communication more vivid and infectious. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 A schematic application scenario diagram of an adaptive emotion-driven timbre cloning text-to-speech method provided in this embodiment.

[0065] Figure 2 A flowchart of an adaptive emotion-driven timbre cloning text-to-speech method provided in this embodiment.

[0066] Figure 3 A schematic diagram of the functional modules of an adaptive emotion-driven timbre cloning text-to-speech device provided in this embodiment.

[0067] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this embodiment.

[0068] Icons: 100, server; 200, terminal; 201, preprocessing module; 202, speech feature acquisition module; 203, emotion feature acquisition module; 204, feature audio conversion module; 710, electronic device; 711, memory; 712, processor; 713, communication module; 714, input / output interface; 715, bus. DETAILED DESCRIPTION

[0069] The drawings of this application are only used for illustrative purposes and should not be construed as limiting the present application. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0070] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0071] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0072] At present, the traditional voice cloning speech generation method has limited emotional expression and cumbersome process. It has high demand for model training data and cannot efficiently meet the market's high requirements for immediacy and flexibility, which limits the smoothness of user experience and the ability of natural interaction.

[0073] For example, a schematic diagram of an application scenario of an adaptive emotion-driven voice cloning text-to-speech method provided in an embodiment of the present application is provided. Figure 1 As shown, the application scenario at least includes a server 100 and a terminal 200 that can communicate with the server 100. The server 100 has the function of receiving text information from the terminal device 200, and processing the text information through some steps of the adaptive emotion-driven timbre cloning text-to-speech method provided in an embodiment of the present application to generate a corresponding voice signal; the terminal device 200 has the function of supporting the input of text information, sending the text information to the server 100, and receiving and playing the voice signal from the server 100.

[0074] It is understandable that the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smart phone terminal, a personal computer, a tablet computer, a car terminal, etc., but is not limited thereto.

[0075] In an operative manner, the server 100 and the terminal 200 may respectively execute the adaptive emotion-driven timbre cloning text-to-speech method provided in the embodiment of the present application, or, optionally, the adaptive emotion-driven timbre cloning text-to-speech method provided in the embodiment of the present application is partially executed in the server 100 and partially executed in the terminal 200.

[0076] On the one hand, the embodiment of the present application provides an adaptive emotion-driven voice cloning text-to-speech method, such as Figure 2 As shown, the method at least includes:

[0077] S1. Divide the target text into at least one sentence, extract the semantic features of each sentence, and split each sentence into a target phoneme sequence;

[0078] S2, splitting the reference text into a reference phoneme sequence, and combining the reference phoneme sequence with each of the target phoneme sequences to obtain a combined phoneme sequence for each sentence;

[0079] S3, performing feature extraction and spectrum extraction on the reference audio corresponding to the reference text to obtain reference speech features and reference audio spectrum;

[0080] S4, processing and decoding based on the semantic features and combined phoneme sequences corresponding to each sentence and the reference speech features to obtain speech features of each sentence;

[0081] S5, extracting the sentiment feature of each sentence from the semantic feature of each sentence;

[0082] S6. Perform feature audio conversion based on the speech features, emotional features and target phoneme sequence corresponding to each sentence, as well as the reference audio spectrum, to obtain a speech signal corresponding to the target text.

[0083] The technical solution provided in this application is described in detail below in conjunction with specific embodiments.

[0084] In this embodiment, a text processing network may be used to extract text semantic features when executing step S1, and the text processing network may be a BERT network.

[0085] Specifically, the target text is first divided into at least one sentence according to punctuation marks, and the at least one sentence is input into a text processing network to extract deep semantic information of the sentence, thereby obtaining semantic features of each sentence, thereby ensuring that the subsequently generated speech signal can accurately restore the semantics of the target text.

[0086] At the same time, the at least one sentence is split into target phoneme sequences.

[0087] Exemplarily, when the input target text is in Chinese, the target text is first segmented into at least one sentence according to punctuation marks, and then each of the segmented sentences is converted into pinyin, and the pinyin is converted into a corresponding phoneme number to obtain at least one target phoneme sequence;

[0088] Similarly, in step S2, the same splitting method as step S1 can be used to split the reference text into a reference phoneme sequence.

[0089] Finally, the reference phoneme sequence is combined with the target phoneme sequence of each sentence to obtain a combined phoneme sequence of each sentence;

[0090] Optionally, step S3 uses an audio processing network to perform feature extraction and spectrum extraction on the reference audio to obtain reference speech features and a reference audio spectrum. The reference speech features may include timbre features for making the speech signal generated in subsequent steps have more realistic timbre features, and the reference audio spectrum is used for subsequent feature audio conversion.

[0091] In this embodiment, the audio processing network adopts the HuBERT network, and other audio processing networks with feature extraction and spectrum extraction functions may also be adopted.

[0092] In this embodiment, the specific process of S4 includes:

[0093] Inputting the combined phoneme sequence of each sentence into the phoneme feature embedding layer to convert it into corresponding phoneme features in a high-dimensional vector space;

[0094] Inputting the semantic features of each sentence into the semantic feature dimension transformation layer for dimension transformation, increasing the dimension of the semantic features, aligning the dimension with the dimension of the corresponding phoneme features, obtaining transformed semantic features, and adding the corresponding transformed semantic features to the phoneme features of each sentence to ensure that the pronunciation of the text is not only accurate but also contains the deep meaning of the target text;

[0095] The fusion feature position encoding layer fuses the transformed semantic feature corresponding to each sentence with the corresponding phoneme feature to obtain at least one first fusion feature, and also performs position encoding on the at least one first fusion feature to obtain at least one second fusion feature, wherein the position encoding process can be expressed as:

[0096]

[0097] Among them, x (p,i) represents the i-th element in the fusion feature, Represents the encoded element, and represents position encoding, and dmode represents the total dimension of fused features;

[0098] In this embodiment, the core of step S4 is to use an autoregressive decoding layer for decoding. The autoregressive decoding layer is a decoder composed of a speech feature embedding layer, a position embedding layer and multiple Transformer self-attention layers. The specific decoding process includes:

[0099] Acquire the reference speech feature as the first speech feature, and predefine a stop feature EOS, wherein the stop feature EOS is used as a determination condition for stopping the loop;

[0100] The speech feature embedding layer embeds the first speech feature, and the processed result is position-encoded by the position embedding layer to generate a coded speech feature;

[0101] Combining the at least one second fusion feature with the coded speech feature to generate at least one combined feature;

[0102] Constructing an attention mask according to the length of the spliced ​​feature, wherein the attention mask is used to ensure that the decoding process follows the autoregressive principle;

[0103] The multiple Transformer self-attention layers decode the corresponding stitched features based on the attention mask to generate a decoded output;

[0104] Performing full connection processing on the decoded output to obtain the intermediate speech features of this cycle;

[0105] Combining the intermediate speech feature with the first speech feature to generate a second speech feature;

[0106] determining whether the second speech feature contains a predefined stop feature EOS,

[0107] If not, the second speech feature is used as the first speech feature to perform the next decoding cycle;

[0108] If it is contained, the loop is stopped and the second speech feature is used as the speech feature of the corresponding sentence.

[0109] By performing feature interaction and decoding in a loop iteration mode, each decoding cycle generates a new voice feature based on the first voice feature input. The step S4 of the embodiment of the present invention only needs a small amount of reference voice as input to realize the instant voice cloning function without complicated training process. The cloning process is fast and efficient, eliminating the time-consuming step of building a model for each user separately. This means that users can immediately experience the text content read aloud with their unique voice, which greatly improves the convenience and personalization of the service.

[0110] In order to execute the step S5, a one-dimensional convolutional network composed of multiple one-dimensional convolutional layers connected in series is constructed. In this embodiment, the specific architecture of the one-dimensional convolutional network includes but is not limited to an input processing layer, a first residual connection layer, a first feature dimension reduction layer, a second residual connection layer, a second feature dimension reduction layer, and a fully connected classification layer connected in sequence. By refining information step by step, a transition is gradually made from basic feature extraction to recognition and classification of high-level emotional features.

[0111] Specifically,

[0112] The input processing layer is used to adjust the feature dimension of the semantic features of each input sentence, and extract the corresponding basic features from the semantic features with adjusted feature dimensions to lay the foundation for subsequent processing;

[0113] The first residual connection layer processes the basic features, adds the processed basic features to the basic features, and performs dimension reduction processing on the addition result through the first feature dimension reduction layer to obtain an intermediate output feature;

[0114] The second residual connection layer processes the intermediate output feature, adds the processed intermediate output feature to the intermediate output feature, and performs dimensionality reduction processing on the addition result through the second feature dimensionality reduction layer to obtain the output feature of the sentence.

[0115] Exemplarily, the first residual connection layer and the second residual connection layer are each composed of a convolutional layer, a batch normalization layer, a ReLU activation function layer, a convolutional layer, a batch normalization layer, and a ReLU activation function layer connected in sequence. The first residual connection layer and the second residual connection layer provide a shortcut path connecting input and output, enhance the network's learning of complex emotional features, while maintaining the coherence of information flow, and effectively avoid the common gradient vanishing problem in deep networks.

[0116] The first feature dimensionality reduction layer and the second feature dimensionality reduction layer are both composed of a convolutional layer, a batch normalization layer and a ReLU activation function layer connected in sequence. The convolutional layers of the first feature dimensionality reduction layer and the second feature dimensionality reduction layer will reduce the feature length and increase the number of feature channels, allowing the network to further focus on more critical emotional patterns, thereby extracting more abstract and expressive emotional features.

[0117] A fully connected classification layer is configured at the end of the one-dimensional convolutional network to perform sentiment classification on the output features of the sentence according to the mapping relationship between the preset output features and the sentiment features, and obtain the sentiment features of each sentence.

[0118] Among them, the mapping relationship between the preset output features and emotional features is obtained by pre-training the one-dimensional convolutional network. In actual application, the fully connected classification layer does not participate in the calculation, but directly obtains the output features with rich emotions output by the last feature reduction layer to provide emotional features for subsequent speech signal synthesis.

[0119] In an optional implementation, when training the one-dimensional convolutional network, a Softmax function and a cross entropy loss function Ce may be used, which may be:

[0120]

[0121] Among them, e i is the sentiment feature of the ith sentence classified by the fully connected classification layer, p i is the predicted value of the sentiment feature, y i is the true value of the sentiment feature.

[0122] Through the one-dimensional convolutional network processing in step S5, the present invention not only efficiently extracts emotional features from the target text, but also optimizes the performance in practical applications while ensuring the training effect, bringing more delicate and realistic emotional expression capabilities to speech synthesis technology.

[0123] In addition to the architectural methods listed in this embodiment, other architectural methods can also be used to achieve the same technical effects as this embodiment.

[0124] Optionally, in order to obtain an emotionally charged speech signal that fits the text semantics, a decoder is required to comprehensively decode speech features, emotional features, text phoneme sequences, and spectrograms of reference audio into a high-quality speech signal.

[0125] In this embodiment, a feature audio conversion module is used to execute the step S6. The feature audio conversion module is developed based on the decoder structure in the VITS variational autoencoder network and is specially designed for feature audio conversion. Its core function is to comprehensively decode the preprocessed speech features, emotional features, text phoneme sequences and reference audio spectrograms into high-quality speech signals. By integrating emotional information and reference audio style, a high degree of unity of content, emotion and personalization in speech synthesis is achieved.

[0126] The working process of the characteristic audio conversion module is specifically as follows:

[0127] Combine the speech features and emotional features corresponding to each sentence to obtain the speech emotional features of each sentence;

[0128] Quantifying the speech emotion features to obtain the quantized features of each sentence;

[0129] Using an encoder to process the reference audio spectrum to obtain embedded features of the reference audio;

[0130] Encode the quantized features and the target phoneme sequence corresponding to each sentence and the embedded features using an encoder to generate a mean, a variance, and a mask of latent variables as an intermediate representation of speech synthesis;

[0131] Finally, the decoder is used to decode the intermediate representation generated by the encoder to generate the speech signal corresponding to the target text.

[0132] The feature audio conversion module follows the efficient decoding logic of the VITS network, integrates the influence of emotional features and reference audio, realizes the precise conversion from highly abstract feature representation to natural speech waveform, and improves the naturalness and expressiveness of synthesized speech.

[0133] The present invention integrates advanced natural language processing and sentiment analysis technologies, so that the TTS system based on the method of the present invention can automatically parse the implicit emotional information in the input text. Through an in-depth understanding of the context, the system can adaptively adjust the emotional color of the voice, and naturally integrate various emotions such as joy, sadness, and surprise into the output voice without human intervention, thereby significantly enhancing the realism and expressiveness of the synthesized voice and making communication more vivid and infectious.

[0134] On the other hand, this embodiment provides an adaptive emotion-driven voice cloning text-to-speech device, such as Figure 3 As shown, including,

[0135] The target text processing module 201 is used to segment the target text into at least one sentence, input each sentence into a text processing network to extract deep semantic information, and obtain semantic features of each sentence;

[0136] The target text processing module 201 also splits each sentence to obtain a target phoneme sequence for each sentence;

[0137] A reference text processing module 202 is used to split the reference text into a reference phoneme sequence, and combine the reference phoneme sequence with each of the target phoneme sequences to obtain a combined phoneme sequence for each sentence;

[0138] The reference audio processing module 203 is used to extract features of the reference audio corresponding to the reference text using an audio processing network to obtain reference speech features, extract spectrum of the reference audio to obtain reference audio spectrum,

[0139] The specific working process of the speech feature acquisition module 204 is as follows:

[0140] Converting the combined phoneme sequence of each sentence into corresponding phoneme features in a high-dimensional vector space;

[0141] Performing dimension transformation on the semantic features corresponding to each of the sentences respectively to obtain transformed semantic features with the same dimension as the phoneme features;

[0142] The transformed semantic features corresponding to each sentence are respectively fused with the corresponding phoneme features to obtain at least one first fused feature;

[0143] Performing position encoding on the at least one first fused feature respectively to obtain at least one second fused feature;

[0144] Acquire the reference speech feature as the first speech feature, and predefine a stop feature EOS, wherein the stop feature EOS is used as a determination condition for stopping the loop;

[0145] Multiple decoding is performed through loop iteration. The specific process of each decoding loop includes:

[0146] Embedding and position encoding the first speech feature to generate a coded speech feature;

[0147] Combining the at least one second fusion feature with the coded speech feature to generate at least one combined feature;

[0148] Constructing an attention mask according to the length of the stitched feature, and decoding the corresponding stitched feature using the attention mask to generate a decoding output;

[0149] Performing full connection processing on the decoded output to obtain the intermediate speech features of this cycle;

[0150] Combining the intermediate speech feature with the first speech feature to generate a second speech feature;

[0151] Among them, if the second speech feature does not contain the stop feature EOS, the second speech feature is used as the first speech feature and the next decoding cycle is performed; if the second speech feature contains the stop feature EOS, the cycle is stopped and the second speech feature is used as the speech feature of the corresponding sentence.

[0152] The sentiment feature acquisition module 205 is used to extract the sentiment feature of each sentence from the semantic feature of each sentence, including an output processing layer, a first residual connection layer, a first feature dimension reduction layer, a second residual connection layer, a second feature dimension reduction layer, and a fully connected classification layer connected in sequence.

[0153] Among them, the first residual connection layer and the second residual connection layer are both composed of a convolution layer, a batch normalization layer, a ReLU activation function layer, a convolution layer, a batch normalization layer and a ReLU activation function layer connected in sequence; the first feature dimensionality reduction layer and the second feature dimensionality reduction layer are both composed of a convolution layer, a batch normalization layer and a ReLU activation function layer connected in sequence.

[0154] The specific working process of the emotion feature acquisition module 205 is as follows:

[0155] The input processing layer is used to adjust the feature dimension of the semantic features of each input sentence, and extract the corresponding basic features from the semantic features with adjusted feature dimensions to lay the foundation for subsequent processing;

[0156] The first residual connection layer processes the basic features, adds the processed basic features to the basic features, and performs dimension reduction processing on the addition result through the first feature dimension reduction layer to obtain an intermediate output feature;

[0157] The second residual connection layer processes the intermediate output feature, adds the processed intermediate output feature to the intermediate output feature, and performs dimensionality reduction processing on the addition result through the second feature dimensionality reduction layer to obtain the output feature of the sentence;

[0158] The fully connected classification layer performs sentiment classification on the output features of the sentence according to the mapping relationship between the preset output features and the sentiment features to obtain the sentiment features of each sentence.

[0159] The feature audio conversion module 206 is developed based on the decoder structure in the VITS variational autoencoder network, and is used to perform feature audio conversion based on the speech features, emotional features and target phoneme sequence corresponding to each sentence, as well as the reference audio spectrum, to obtain the speech signal corresponding to the target text. The specific process is as follows:

[0160] Combine the speech features and emotional features corresponding to each sentence to obtain the speech emotional features of each sentence;

[0161] Quantifying the speech emotion features to obtain the quantized features of each sentence;

[0162] Using an encoder to process the reference audio spectrum to obtain embedded features of the reference audio;

[0163] Encode the quantized features and the target phoneme sequence corresponding to each sentence and the embedded features using an encoder to generate a mean, a variance, and a mask of latent variables as an intermediate representation of speech synthesis;

[0164] Finally, the decoder is used to decode the intermediate representation generated by the encoder to generate the speech signal corresponding to the target text.

[0165] It can be understood that the above-mentioned device embodiments and the above-mentioned method embodiments can correspond to each other, and similar descriptions of the device embodiments can refer to the method embodiments. An adaptive emotion-driven timbre cloning text-to-speech device provided in an embodiment of the present application can execute an adaptive emotion-driven timbre cloning text-to-speech method provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method. The functional modules of the adaptive emotion-driven timbre cloning text-to-speech device can be implemented in the form of hardware, can be implemented by instructions in the form of software, or can be implemented by a combination of hardware and software modules.

[0166] Optionally, the software module may be located in a random access memory, a read-only memory, a programmable read-only memory, a flash memory, an electrically erasable programmable memory, a register, or other storage media. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in combination with its hardware.

[0167] The present application embodiment provides an electronic device 710, whose structure is as follows: Figure 3 The electronic device 710 may be Figure 1 The server 100 or the terminal 200 is shown.

[0168] like Figure 3 As shown, the electronic device 710 includes a memory 711, a processor 712, a communication module 713 and an input / output interface 714, etc. Optionally, the memory 711, the processor 712, the communication module 713 and the input / output interface 714 can be connected and communicated through a bus 715.

[0169] The memory 711 is used to store one or more computer programs and transmit the code of the computer program to the processor 712; when the one or more computer programs are executed by the processor 711, an adaptive emotion-driven timbre cloning text-to-speech method in an embodiment of the present application is implemented.

[0170] Optionally, the electronic device 710 can be connected to a network via a communication module 713 to communicate with other devices, such as a terminal or a server, through the network to achieve data interaction. The electronic device 710 can be various forms of digital computers, such as desktop computers, servers, workstations, mainframe computers or other types of computers. The electronic device 710 can also be various forms of mobile terminals, such as smart phones, tablet computers, wearable devices (such as helmets, glasses, watches, etc.) and other similar mobile terminals.

[0171] Optionally, the electronic device 710 can be connected to a required input / output device, such as a keyboard, a display device, etc., through the input / output interface 714. The electronic device 710 itself can have a display device, and can also be connected to other external display devices through the input / output interface 714. Optionally, a storage device, such as a hard disk, can be connected through the input / output interface 714, so that data in the electronic device 710 can be stored in the storage device, or data in the storage device can be read, and data in the storage device can also be stored in the memory 711.

[0172] It is understandable that the input / output interface 714 may be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 714 may be a component of the electronic device 710 or an external device connected to the electronic device 710 when needed.

[0173] Optionally, the memory 711 may be a volatile memory and / or a non-volatile memory, the volatile memory may be a random access memory, etc., and the non-volatile memory may be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory or a flash memory, etc.

[0174] Optionally, the computer program stored in the processor 711 may be divided into one or more modules, which are stored in the memory 711 and executed by the processor 712 to complete the method provided in the embodiment. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the computer program instruction segments are used to describe the execution process of the computer program in the electronic device 710.

[0175] Optionally, the processor 712 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 712 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special artificial intelligence computing chips, various processors running machine learning model algorithms, and may also be any appropriate controller, microcontroller, processor, etc. The processor 712 executes the various methods and processes of this embodiment, illustratively, such as an adaptive emotion-driven voice cloning text-to-speech method of an embodiment of the present application.

[0176] Optionally, the bus 715 may include a path for transmitting information. The bus 715 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. According to different functions, the bus 715 may be divided into an address bus, a data bus, a control bus, etc.

[0177] In an optional implementation, the embodiment of the present application further provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. Part or all of the computer program can be loaded and / or installed on the memory 711 of the electronic device 710. When the computer program is executed by the processor 712, one or more steps of an adaptive emotion-driven voice cloning text-to-speech method of the embodiment of the present application can be performed.

[0178] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or the like.

[0179] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. An adaptive emotion-driven timbre cloning text-to-speech method, characterized in that: include, Divide the target text into at least one sentence, extract the semantic features of each sentence, and split each sentence into a target phoneme sequence; Splitting the reference text into a reference phoneme sequence, and combining the reference phoneme sequence with each of the target phoneme sequences to obtain a combined phoneme sequence for each sentence; Performing feature extraction and spectrum extraction on the reference audio corresponding to the reference text to obtain reference speech features and reference audio spectrum; Processing and decoding based on the semantic features and combined phoneme sequences corresponding to each sentence and the reference speech features to obtain speech features of each sentence; Extracting the sentiment feature of each sentence from the semantic feature of each sentence; Based on the speech features, emotional features and target phoneme sequence corresponding to each sentence, and the reference audio spectrum, feature audio conversion processing is performed to obtain a speech signal corresponding to the target text.

2. The adaptive emotion-driven timbre cloning text-to-speech method according to claim 1, characterized in that: Extracting the sentiment feature of each sentence from the semantic feature of each sentence specifically includes: Adjusting the feature dimension of the semantic feature of each sentence, and extracting corresponding basic features from the semantic features with the adjusted feature dimension; Performing residual connection processing on the basic features, adding the processed basic features to the basic features before the processing, and performing dimensionality reduction processing on the added result to obtain intermediate output features; Performing residual connection processing on the intermediate output features, adding the intermediate output features after processing to the intermediate output features before processing, and performing dimensionality reduction processing on the addition result to obtain the output features of each sentence; Based on the mapping relationship between the preset output features and the sentiment features, sentiment classification is performed on the output features of each sentence to obtain the sentiment features of each sentence.

3. The adaptive emotion-driven timbre cloning text-to-speech method according to claim 2, characterized in that: The step of adjusting the feature dimension of the semantic features of each sentence and extracting the corresponding basic features from the semantic features with the adjusted feature dimension is implemented through an input processing layer, and the input processing layer is composed of a convolutional layer, a batch normalization layer and a ReLU activation function layer connected in sequence.

4. The adaptive emotion-driven timbre cloning text-to-speech method according to claim 2, characterized in that: The performing residual connection processing on the basic features, adding the processed basic features to the basic features before the processing, and performing dimensionality reduction processing on the added results to obtain intermediate output features is achieved by a first residual connection layer and a first feature dimensionality reduction layer connected to the first residual connection layer; and / or, The performing residual connection processing on the intermediate output features, adding the intermediate output features after the processing to the intermediate output features before the processing, and performing dimensionality reduction processing on the addition result to obtain the output features of each sentence is achieved by a second residual connection layer and a second feature dimensionality reduction layer connected to the second residual connection layer; The first residual connection layer and the second residual connection layer are each composed of a convolution layer, a batch normalization layer, a ReLU activation function layer, a convolution layer, a batch normalization layer, and a ReLU activation function layer connected in sequence; The first feature dimension reduction layer and the second feature dimension reduction layer are both composed of a convolution layer, a batch normalization layer and a ReLU activation function layer connected in sequence.

5. The adaptive emotion-driven timbre cloning text-to-speech method according to claim 2, characterized in that: The sentiment classification is performed on the output features of each sentence based on the mapping relationship between the preset output features and the sentiment features to obtain the sentiment features of each sentence, which is achieved through a fully connected classification layer.

6. The adaptive emotion-driven timbre cloning text-to-speech method according to claim 1, characterized in that: The step of dividing the target text into at least one sentence, extracting the semantic features of each sentence, and splitting each sentence into a target phoneme sequence specifically includes: The target text is divided into at least one sentence, and each sentence is input into the text processing network to extract deep semantic information and obtain the semantic features of each sentence; Split each sentence to obtain the target phoneme sequence for each sentence; Splitting the reference text into a reference phoneme sequence, and combining the reference phoneme sequence with each of the target phoneme sequences to obtain a combined phoneme sequence for each sentence; and / or, The feature extraction and spectrum extraction of the reference audio corresponding to the reference text to obtain the reference speech feature and the reference audio spectrum specifically includes: using an audio processing network to perform feature analysis on the reference audio corresponding to the reference text to obtain the reference speech feature; Perform spectrum extraction on the reference audio corresponding to the reference text to obtain a reference audio spectrum.

7. The adaptive emotion-driven timbre cloning text-to-speech method according to claim 1, characterized in that: The semantic features and combined phoneme sequences corresponding to each sentence and the reference speech features are processed and decoded to obtain the speech features of each sentence, specifically: Converting the combined phoneme sequence corresponding to each sentence into phoneme features in a high-dimensional vector space; Performing dimension transformation on the semantic features corresponding to each of the sentences respectively to obtain transformed semantic features with the same dimension as the phoneme features; The transformed semantic features corresponding to each sentence are respectively fused with the corresponding phoneme features to obtain at least one first fused feature; Performing position encoding on the at least one first fused feature respectively to obtain at least one second fused feature; Acquire the reference speech feature as the first speech feature, and predefine a stop feature EOS; Multiple decoding is performed through loop iteration. The specific process of each decoding loop includes: Embedding and position encoding the first speech feature to generate a coded speech feature; Combining the at least one second fusion feature with the coded speech feature to generate at least one combined feature; Constructing an attention mask according to the length of the stitched feature; Decoding the corresponding stitched features using the attention mask to generate a decoded output; Performing full connection processing on the decoded output to obtain the intermediate speech features of this cycle; Combining the intermediate speech feature with the first speech feature to generate a second speech feature; Among them, if the second speech feature does not contain the stop feature EOS, the second speech feature is used as the first speech feature and the next decoding cycle is performed; if the second speech feature contains the stop feature EOS, the cycle is stopped and the second speech feature is used as the speech feature of the corresponding sentence.

8. The adaptive emotion-driven timbre cloning text-to-speech method according to any one of claims 1 to 7, characterized in that: The feature audio conversion process is performed based on the speech features, emotional features and target phoneme sequence corresponding to each sentence, as well as the reference audio spectrum, to obtain a speech signal corresponding to the target text, specifically including: Combine the speech features and emotional features corresponding to each sentence to obtain the speech emotional features of each sentence; Quantifying the speech emotion features to obtain the quantized features of each sentence; Processing the reference audio spectrum to obtain embedded features of the reference audio; Encoding the quantized features and target phoneme sequence corresponding to each sentence, as well as the embedded features, to generate a mean, variance, and mask of the latent variables as an intermediate representation for speech synthesis; The intermediate representation is decoded to generate a speech signal corresponding to the target text.

9. An adaptive emotion-driven voice cloning text-to-speech device, characterized in that: include: A target text processing module, used to segment the target text into at least one sentence, extract the semantic features of each sentence, and split each sentence into a target phoneme sequence; A reference text processing module, used for splitting the reference text into a reference phoneme sequence, and combining the reference phoneme sequence with each of the target phoneme sequences to obtain a combined phoneme sequence for each sentence; A reference audio processing module is used to perform feature extraction and spectrum extraction on the reference audio corresponding to the reference text to obtain reference speech features and reference audio spectrum; A speech feature acquisition module, used for processing and decoding based on the semantic features and combined phoneme sequences corresponding to each sentence and the reference speech features to obtain the speech features of each sentence; The sentiment feature acquisition module is used to extract the sentiment feature of each sentence from the semantic features of each sentence; The feature audio conversion module is used to perform feature audio conversion processing based on the speech features, emotional features and target phoneme sequence corresponding to each sentence, as well as the reference audio spectrum, to obtain a speech signal corresponding to the target text.

10. An electronic device, comprising: a memory for storing one or more computer programs; The processor, when the one or more computer programs are executed by the processor, implements the adaptive emotion-driven voice cloning text-to-speech method of claims 1-8 above.

Images