Speech synthesis method and device

Abstract

The embodiment of the invention aims to provide a speech synthesis method, device and equipment, a computer program product and a computer program storage medium. Wherein the speech synthesis system obtains corresponding discrete semantic features according to an input text; mapping the discrete semantic features and audio features of the input text to the same feature space so as to align the discrete semantic features and the audio features; extracting pronunciation and rhythm features corresponding to the input text from the aligned features; converting the pronunciation and rhythm features and reference timbre features into Mel spectrogram features; and converting the Mel spectrogram features into output speech. According to the embodiments of the invention, in order to realize a high-reduction-degree speech synthesis scheme, intermediate acoustic features irrelevant to timbre are selected to decouple timbre learning and pronunciation and rhythm learning, and more complex-scene and high-confidence low-sampling-rate data are used for speech synthesis training by means of domain transfer learning.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of speech processing, and in particular to a technology for speech synthesis. Background Art

[0002] In previous speech synthesis scenarios, the relevant feature modules are often not decoupled within the system, the learning objectives of each model module are unclear, and each module contains timbre and pronunciation information, which greatly increases the difficulty of modeling the timbre dimension. Some solutions will filter the timbre information by adding training loss to the intermediate features, and some solutions will choose data augmentation to enhance the conditional timbre information, but the former often cannot effectively remove clean information, and the latter is not effective in solution processing and requires a better design for the data augmentation method.

[0003] On the other hand, modeling of timbre distribution requires reliance on a large amount of high-sampling rate audio data, some of which is obtained by purchasing data and some by data crawlers. The former requires high manpower and economic costs, while the latter often requires a lot of means to clean and screen the data to ensure data quality. Summary of the invention

[0004] The purpose of each embodiment of the present disclosure is to provide a method, an apparatus, a device, a computer program product, and a computer program storage medium for speech synthesis.

[0005] According to one aspect of the present disclosure, a speech synthesis method is provided, wherein the method comprises the following steps:

[0006] According to the input text, obtain its corresponding discrete semantic features;

[0007] Mapping the discrete semantic features and the audio features of the input text into the same feature space to align the two;

[0008] Extracting pronunciation and prosody features corresponding to the input text from the aligned features;

[0009] Converting the pronunciation and prosody features and reference timbre features into mel-spectrogram features;

[0010] The mel-spectrogram features are converted into output speech.

[0011] According to one aspect of the present disclosure, a speech synthesis device is also provided, wherein the device comprises:

[0012] Acoustic models for:

[0013] According to the input text, obtain its corresponding discrete semantic features;

[0014] Mapping the discrete semantic features and the audio features of the input text into the same feature space to align the two;

[0015] Extracting pronunciation and prosody features corresponding to the input text from the aligned features;

[0016] Vocoder for:

[0017] Converting the pronunciation and prosody features and reference timbre features into mel-spectrogram features;

[0018] The mel-spectrogram features are converted into output speech.

[0019] According to one aspect of the present disclosure, a speech synthesis device is further provided, wherein the device includes a memory and a processor, the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the device is configured to perform the following operations:

[0020] According to the input text, obtain its corresponding discrete semantic features;

[0021] Mapping the discrete semantic features and the audio features of the input text into the same feature space to align the two;

[0022] Extracting pronunciation and prosody features corresponding to the input text from the aligned features;

[0023] Converting the pronunciation and prosody features and reference timbre features into mel-spectrogram features;

[0024] The mel-spectrogram features are converted into output speech.

[0025] According to one aspect of the present disclosure, a computer program product is further provided, comprising computer program instructions, wherein when the computer program instructions are executed by a computer device, the computer device is configured to perform a speech synthesis method, wherein the method comprises the following steps:

[0026] According to the input text, obtain its corresponding discrete semantic features;

[0027] Mapping the discrete semantic features and the audio features of the input text into the same feature space to align the two;

[0028] Extracting pronunciation and prosody features corresponding to the input text from the aligned features;

[0029] Converting the pronunciation and prosody features and reference timbre features into mel-spectrogram features;

[0030] The mel-spectrogram features are converted into output speech.

[0031] According to one aspect of the present disclosure, there is also provided a computer program storage medium, in which computer executable instructions are stored. When the computer executable instructions are executed by a computer device, the computer device is configured to perform a speech synthesis method, wherein the method comprises the following steps:

[0032] According to the input text, obtain its corresponding discrete semantic features;

[0033] Mapping the discrete semantic features and the audio features of the input text into the same feature space to align the two;

[0034] Extracting pronunciation and prosody features corresponding to the input text from the aligned features;

[0035] Converting the pronunciation and prosody features and reference timbre features into mel-spectrogram features;

[0036] The mel-spectrogram features are converted into output speech.

[0037] In order to achieve a high-fidelity speech synthesis solution, the various embodiments of the present invention select intermediate acoustic features that are unrelated to timbre to decouple timbre learning from pronunciation and rhythm learning, and use domain transfer learning to train speech synthesis on more complex scenarios but high-confidence low-sampling rate data. On the one hand, the focus of model training and the corresponding data training efficiency are improved, and the learning method restores the complete features containing timbre based on the reference audio condition using the timbre-unrelated features. On the other hand, the low-sampling rate data of rich scenarios open sourced from academia and industry are greatly utilized to improve the generalization ability of the model, and ultimately a high-fidelity speech synthesis system is achieved.

[0038] The highly restored zero-sample vocoder provided by the embodiments of the present disclosure has an important role and significance. On the one hand, it can have a higher generalization ability with the help of massive data learning, and can restore the voice information of various complex scenes, such as highly expressive emotional speech, paralinguistic utterances such as laughter, etc. On the other hand, it can quickly support new timbre speech synthesis capabilities without training, and the model can be reused and efficiently deployed. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0040] Figure 1 A flow chart of a method for speech synthesis according to an embodiment of the present disclosure is shown;

[0041] Figure 2 A schematic diagram of a device for speech synthesis according to an embodiment of the present disclosure is shown.

[0042] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0044] In a typical configuration of the present disclosure, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0045] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0046] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0047] For example, a computer program for executing various functions and processes of multiple embodiments of the present disclosure is stored in the memory, and when the processor executes the corresponding computer program, multiple embodiments of the present disclosure are implemented.

[0048] See also Figure 1 , which shows a speech synthesis process according to an embodiment of the present disclosure.

[0049] like Figure 1As shown, in step S1, the computer device obtains the corresponding discrete semantic features according to the input text; in step S2, the computer device maps the discrete semantic features of the input text and the audio features of the input text to the same feature space to align the two; in step S3, the computer device extracts the pronunciation and rhythm features corresponding to the input text from the aligned features; in step S4, the computer device converts the pronunciation and rhythm features of the input text and the reference timbre features into mel-spectrogram features; in step S5, the computer device converts the mel-spectrogram features into output speech.

[0050] Here, "computer equipment" generally refers to various types of computing resources, such as user equipment, or equipment formed by integrating user equipment and network equipment through a network, or applications running on the above equipment. The user equipment includes but is not limited to various terminal devices such as computers, mobile phones, and tablet computers. The network equipment includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer collections, which can be used to implement some processing functions when setting alarms. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing (Cloud Computing), where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0051] These computer devices can be functionally referred to as "speech synthesis systems" and can be used equivalently herein unless otherwise specified. These computer devices can be configured to implement the speech synthesis functions required by the various embodiments of the present disclosure, including but not limited to data storage, processing, analysis, and application deployment. Through flexible configuration and optimization, these computer devices can support high-performance computing, real-time data processing, data security, and system scalability to meet the actual needs of the technical solutions of the present disclosure.

[0052] Related References Figure 1 and Figure 2 The speech synthesis system 20 includes an acoustic model 21 and a vocoder 22. Specifically, in steps S1-S3, the acoustic model 21 obtains the corresponding pronunciation and prosodic features according to the input text.

[0053] Pronunciation and prosody features focus on how text is presented in speech, involving elements such as phonemes, pinyin, pitch, stress marks, rhythm, and intonation. Phonemes are the basic units of speech and can clearly describe the pronunciation of each word. These features directly affect the naturalness and fluency of speech and are an indispensable part of the speech synthesis process.

[0054] By feature decoupling, the timbre features are decoupled from the pronunciation and prosody features, and the present disclosure focuses more on the pronunciation and prosody features. Feature decoupling is an important concept in machine learning and deep learning. Its core idea is to decompose the feature representation of data into multiple independent, meaningful, and interpretable sub-parts, each of which captures different aspects or attributes of the data and has as little interdependence or correlation as possible.

[0055] In some embodiments, the acoustic model 21 obtains the corresponding discrete semantic features from the input text. Subsequently, the obtained discrete semantic features and the audio features of the input text are mapped to the same feature space for alignment, and the pronunciation and prosody features corresponding to the input text are extracted from the aligned audio features.

[0056] In some other embodiments, the acoustic model 21 obtains the corresponding discrete semantic features from the input text. Subsequently, the obtained discrete semantic features and the discretized pronunciation and prosody features obtained from the corresponding audio signal of the input text are mapped to the same feature space for alignment, and the pronunciation and prosody features corresponding to the input text are extracted from the aligned discrete semantic features.

[0057] In some examples, the acoustic model 21 may use a Transformer model-based decoder to extract discrete semantic features from the input text.

[0058] Discrete semantic features refer to specific semantic units, usually including words, phrases or specific semantic categories, and also include features related to pronunciation and prosody. Such features clearly identify specific information in the text, such as nouns, verbs, sentiment categories, and phoneme information. Through these features, the speech synthesis system can understand the semantic details of the text and adjust pronunciation and stress during synthesis. For example, when emotional words or specific nouns appear in the text, the speech synthesis system can adjust the intonation and stress accordingly, thereby more realistically reproducing the emotion of the text.

[0059] The input text is first preprocessed and converted into a series of processable tokens. Next, the decoder based on the Transformer model processes these tokens. This model focuses on generation tasks and gradually generates discrete semantic features through autoregression.

[0060] During the training phase, the model input is a text sequence that has been segmented, and the goal is to predict the corresponding discrete semantic features. To this end, an end symbol is added to the end of the semantic feature for each sample of the training data. This allows the model to learn when to stop generating predictions and then generate a complete feature sequence through the mechanism of autoregression.

[0061] During the training process, the cross entropy loss function is used to measure the gap between the model output and the true label and optimize the model parameters. Through a large number of training iterations, the model gradually adjusts the weights to improve its prediction accuracy for discrete semantic features.

[0062] In the inference phase, the input text is fed into the trained model, and the model starts generating discrete semantic features. It does this by predicting features step by step, one token at a time, until the end symbol is predicted. This process ensures that the generated feature sequence is closely connected to the context of the input text, thereby effectively extracting relevant discrete semantic features.

[0063] In some examples, the wav2vec 2.0 model is used to extract audio features from the audio signal corresponding to the input text. The discrete semantic features and audio features of the input text are then mapped to the same feature space, and the wav2vec 2.0 model aligns the two and extracts the pronunciation and prosody features of the input text from the aligned audio features.

[0064] First, a text-to-speech (TTS) audio generation tool is used to generate an audio signal corresponding to the input text. The wav2vec2.0 model including CNN and Transformer structures extracts audio features from the audio signal.

[0065] Among them, CNN is used to process audio signals to extract low-level audio features. CNN can capture local audio information, such as short-term changes in the spectrum, and generate a series of low-dimensional feature representations.

[0066] The low-level audio features extracted by CNN are passed to the encoder of the Transformer structure. The encoder processes the input features in context through the self-attention mechanism and multi-layer stacked network structure to generate high-level feature representations. These high-level features can capture long-distance dependencies and global information in the audio signal.

[0067] In the decoder part of the Transformer, feature transformation is performed based on the high-level features generated by the encoder. After generating high-level features, Wav2Vec 2.0 uses a pre-trained codebook to discretize these continuous features, that is, to map the generated continuous features to a discrete feature space. This codebook is learned during the model training process, and each codebook entry represents a discrete feature vector. The discretization process is achieved by finding the closest entry in the codebook. Specifically, for a given continuous feature representation, the model calculates the distance between the feature and each entry in the codebook, and selects the entry with the smallest distance as the discretized output. In this way, audio features can be effectively discretized for better alignment and fusion with other features.

[0068] The discrete semantic features of the input text are also mapped to the shared feature space. Thus, in the shared feature space, the discrete semantic features of the input text and the audio features are aligned, and the pronunciation and rhythm features of the input text are extracted from the aligned audio features.

[0069] Here, the wav2vec 2.0 model aligns the text features and the audio features in a shared feature space. This step ensures that the two are as close as possible in space, indicating a semantic match between them. The goal of the alignment step is to refine and adjust the audio features so that they more accurately correspond to the text in terms of pronunciation and prosody. This may involve more complex acoustic features such as pitch, stress, speaking rate, etc. From the aligned audio features, the wav2vec 2.0 model extracts specific pronunciation and prosody features.

[0070] In some other examples, the wav2vec 2.0 model is used to extract discretized pronunciation and prosody features from the audio signal corresponding to the input text. The discrete semantic features of the input text and the discretized pronunciation and prosody features are then mapped to the same feature space, and the wav2vec 2.0 model aligns the two features and extracts the pronunciation and prosody features of the input text from the aligned discrete semantic features.

[0071] Here, the wav2vec 2.0 model is trained to extract discretized pronunciation and prosody features directly from the audio signal. The Wav2Vec 2.0 model is trained using self-supervised learning. The model learns the structure of the audio signal by predicting a portion of the input audio (the masked portion). Specifically, the model randomly masks a portion of the feature representation and trains the model to predict the masked portion.

[0072] The Wav2Vec 2.0 model uses a contrastive loss function to learn more effective audio representations by comparing correctly predicted features (positive samples) with irrelevant features (negative samples). This mechanism enables the Wav2Vec 2.0 model to learn from a large amount of unlabeled audio data without labels.

[0073] The wav2vec 2.0 model consists of two main parts: a convolutional neural network (CNN) and a Transformer structure.

[0074] First, the input of the wav2vec 2.0 model is the original audio signal. The CNN part is responsible for the preliminary processing of the audio signal, and extracts low-level audio features such as spectrograms and time series features through multiple convolutional layers. The hierarchical structure of CNN can capture local time-frequency features, laying the foundation for subsequent high-level representation learning.

[0075] The low-level audio features extracted by CNN are then passed to the Transformer part. The Transformer structure consists of an encoder and a decoder. The encoder receives the low-level audio features extracted by CNN and performs context processing. Through the self-attention mechanism, it generates high-level feature representations that can capture long-range dependencies in the audio signal. The decoder uses the high-level features generated by these encoders and combines the contextual information with the prediction of the target pronunciation and prosody features through the self-attention mechanism to generate the corresponding pronunciation and prosody features.

[0076] Using a shared codebook, the extracted high-dimensional continuous features are mapped to a set of predefined discrete feature representations, such as discretized pronunciation and prosodic feature representations. This mapping process usually involves calculating the distance between the feature vector and each discrete representation in the codebook (for example, using Euclidean distance or cosine similarity) and selecting the closest discrete representation as the output.

[0077] The discrete semantic features of the input text are also mapped to the shared feature space. Thus, in the shared feature space, the discrete semantic features of the input text are feature aligned with the discretized pronunciation and prosody features, and the pronunciation and prosody features of the input text are extracted from the aligned discrete semantic features.

[0078] Here, the wav2vec 2.0 model aligns the discrete semantic features with the discretized pronunciation and prosody features in a shared feature space. This step ensures that the two are as close as possible in space, indicating a semantic match between them. This may involve more complex acoustic features such as pitch, stress, speaking rate, etc. From the aligned discrete semantic features, the wav2vec 2.0 model extracts specific pronunciation and prosody features.

[0079] In some other alternative embodiments, the acoustic model 21 may use a decoder based on a Transformer model to extract pronunciation and prosodic features directly from the input text.

[0080] The input text is fed into the decoder after preprocessing, such as word segmentation and word embedding. The decoder can generate a sequence of phonemes word by word, for example predicting how each word should be pronounced, including features such as pitch and pronunciation duration. In this process, the decoder takes into account the previously generated phonemes and the text context to ensure the continuity and naturalness of pronunciation. Although the text may not contain prosodic information directly, it can be generated through prediction and inference. For example, the pause position and rhythm changes of a sentence can be identified. Intonation markers also need to be considered, which describe the rising or falling tone of a sentence, which is crucial for generating natural and fluent speech. Through the self-attention mechanism of the model, the decoder can predict the stress position, rhythm information (such as pauses) and intonation changes in the text based on the contextual information, and can also dynamically adjust the prosodic performance of each marker so that the generated prosodic features are consistent with the context. The prediction of these prosodic features is achieved through the understanding of the text sequence and the ability to generate sequences, thereby converting the implicit prosodic information in the text into explicit acoustic information. All extracted pronunciation and prosodic features are organized into a structured feature set. This feature set includes phoneme sequences, pinyin, tones, stress marks, rhythm information, etc., which will be used in the subsequent speech synthesis process.

[0081] Here, it should be noted that those skilled in the art can understand that other models / algorithms can also be used to extract pronunciation and prosodic features from input text. The extraction process can be roughly decomposed into pre-processing, pronunciation feature extraction and prosodic feature extraction. Some models / algorithms can be used to perform part or all of these operations / steps respectively, so that they can be combined to complete the extraction of pronunciation and prosodic features from input text. For example, word segmentation pre-processing can use word segmenters or special word segmentation tools in pre-trained language models such as BERT (Bidirectional Encoder Representations from Transformers, Transformers represented by bidirectional encoders), GPT (Generative Pre-trained Transformer, Generative Pre-trained Transformer), such as spaCy, NLTK (Natural Language Toolkit, Natural Language Toolkit). Pronunciation features can be extracted using existing phonetic symbol libraries or pronunciation dictionaries. Prosodic features can be extracted using rule-based models, using improved Transformer models or LSTM (Long Short-Term Memory, Long Short-Term Memory Network), etc. The present disclosure does not impose any limitation on the models / algorithms that can be used herein, and any models / algorithms that can be used herein are cited herein and are included within the scope of patent protection of the present disclosure.

[0082] In some other alternative embodiments, the acoustic model 21 can also use the wav2vec 2.0 model to directly extract pronunciation and prosody features from the audio signal of the input text. The pronunciation and prosody features thus extracted belong to acoustic features and can then be used by the vocoder 22 to synthesize the output speech.

[0083] Next, refer to Figure 1 and Figure 2 In step S4, the speech synthesis system 20, more specifically, the vocoder 22, converts the pronunciation and prosodic features of the input text and the reference timbre features into mel-spectrogram features. Then, in step S5, the vocoder 22 converts the mel-spectrogram features into output speech.

[0084] In some embodiments, the vocoder 22 further includes a front end and a decoder ( Figure 2 not shown).

[0085] The front end of the vocoder 22 converts the pronunciation and prosodic features of the input text and the reference timbre features into mel-spectrogram features. Subsequently, the decoder of the vocoder 22 converts the mel-spectrogram features into output speech.

[0086] Here, the Transformer model is used as the front end of the vocoder, and the input is the semantic features (pronunciation and rhythm features) that are unrelated to timbre and the reference timbre features. The reference timbre features can be consistent with the target timbre features, so that high-fidelity and personalized speech synthesis can be achieved while ensuring smooth and natural speech.

[0087] Mel-spectrogram is a commonly used audio feature representation. It is based on the human ear's perception of sound frequency and is based on the Mel scale. It can better simulate human auditory perception.

[0088] First, pronunciation features include phonemes, syllables and their corresponding pronunciation times, while prosodic features include information such as stress, intonation and rhythm. These features are combined to form a preliminary representation of the sound characteristics of the text. These features are then combined with reference timbre features, which usually contain frequency domain features of recorded natural speech data, which can guide the synthesis process to make the generated audio more natural and fit the target timbre.

[0089] Next, the time domain audio sequence is converted into frequency domain features through Fourier transform. The purpose of this transformation is to extract the spectral information of the audio signal and capture the frequency components of the audio. Fourier transform converts the audio signal from time domain representation to frequency domain representation, so that the various frequency components of the audio can be separated and analyzed.

[0090] After obtaining the frequency domain features, they are further processed using a filter bank. A filter bank is usually composed of a series of bandpass filters, each of which processes a specific frequency range. This method enables the model to extract important spectral information on the Mel scale, which is consistent with the auditory characteristics of the human ear, and can effectively describe the energy distribution of the audio signal.

[0091] Finally, after the above processing, the obtained Mel-spectrogram features can clearly reflect the pronunciation and rhythmic features of the input text and correspond to the reference timbre features. These Mel-spectrogram features can then be used as input to the vocoder decoder to generate synthesized audio, ensuring that the synthesized speech meets expectations in timbre, rhythm, and naturalness. In this way, the speech synthesis system can achieve the conversion from text to high-quality audio and improve the effect of speech synthesis.

[0092] In some embodiments, the scene generalization capability and timbre coverage of the model are improved by expanding the training database.

[0093] Here, the audio training database is expanded by domain knowledge transfer. Domain transfer learning is a machine learning technique that aims to transfer knowledge from one domain (source domain) to another domain (target domain) to improve the performance of the model in the target domain. This migration is usually because the data in the target domain is insufficient or the annotation cost is high, while the data in the source domain is relatively abundant and easy to obtain.

[0094] Since the mel-spectrogram features of different sampling rates have basically the same information in the low-frequency band, including pronunciation, rhythmic features and basic timbre features, high-sampling rate audio data will have more complex high-frequency band information than low-sampling rate audio data. Through the method of domain knowledge transfer, the model can integrate the knowledge of the two fields, enhance the model's ability to recover low-frequency band information by using the common points of different sampling rates, and learn to recover high-frequency band information by enhancing the prior knowledge of high-sampling rate data.

[0095] In some examples, the training database collects high-confidence speech data of different sampling rates, such as purchased audio data with high sampling rates, and high-confidence data with low sampling rates, such as open-source emotion recognition data from academia, (noisy) speech recognition data, and self-owned data. The corresponding mel-spectrogram features are generated based on these speech data.

[0096] During training, the model adopts zero-shot learning and uses conditional masking as a conditional injection method for zero-shot learning.

[0097] Zero-shot learning is a special learning paradigm that aims to solve the problem of classifying or predicting when a model is faced with completely unknown categories. In zero-shot learning, the model will not be exposed to any samples of the category to be recognized in the test phase during the training phase. The model needs to effectively extract useful features from the provided conditional reference information and match it with the feature patterns of the known categories during the training process to achieve accurate classification or prediction.

[0098] Specifically, part of the features of the Mel-spectrogram feature are masked, so that the unmasked feature part (features unrelated to timbre, such as pronunciation and rhythm features) is used as a reference feature condition, and the masked feature part is used as a target prediction feature to restore the complete features containing timbre.

[0099] Mask some Mel-spectrogram features and use them as the prediction target of the model to simulate the zero-shot learning scenario. This process allows the model to face some missing feature data during training, so that it can learn how to complete it by referring to feature conditions and other conditional features. In addition, the loss function uses the square error function to calculate the error between the unmasked Mel-spectrogram features and the completed Mel-spectrogram features.

[0100] Furthermore, the model is extended based on the reference timbre features by adding sampling rate features in addition to the original masked Mel-spectrogram features for enhancement.

[0101] In addition to the original Mel-spectrogram features, the sampling rate features can be concatenated with the Mel-spectrogram features during training to expand the model's input feature space. This helps the model understand the characteristics of the audio signal more comprehensively. These sampling rate features help the model distinguish and understand the audio features at different sampling rates, thereby effectively fusing the commonalities (such as low-frequency band information) and differences (such as high-frequency band information) of data at different sampling rates. The model's learning ability on audio data at different sampling rates is enhanced, and the performance of the model is improved by combining rich input features through feature fusion and parameter sharing.

[0102] In addition, considering the existence of prior knowledge between high-sampling rate Mel-spectrogram features and low-sampling rate Mel-spectrogram features, which can be converted into each other, the high-sampling rate data can be actively downsampled in addition to the original low-sampling rate data, so that the model can learn the differences in different fields through supervised training and enhance the sub-feature space mapping learning.

[0103] In some examples, the training database is further enhanced. In addition to collecting high-confidence speech data of different sampling rates, the high-sampling rate speech data can be downsampled to obtain a new batch of low-sampling rate speech data. The low-sampling rate speech data obtained by downsampling can also be used in the model training process according to the above method.

[0104] The trained model can be used to convert the pronunciation and rhythm features of the input text and the reference timbre features into mel-spectrogram features.

[0105] Subsequently, the decoder of the vocoder 22 converts the mel-spectrogram features into output speech. The vocoder decoder uses the mel-spectrogram features as input to complete the speech conversion, and the generative adversarial loss and the square error function can be used as optimization targets during training.

[0106] The zero-shot vocoder that combines feature decoupling (decoupling timbre features from pronunciation and rhythm features) and domain knowledge transfer can reduce the complexity of model learning through effective feature design. At the same time, the deep learning method combined with domain knowledge transfer can model the existing massive non-high sampling rate speech data, and finally obtain a vocoder with stronger zero-shot reasoning ability and higher recovery generalization.

[0107] The deep learning vocoder trained based on a large amount of data can combine the pronunciation and rhythmic features with the reference speech of the target timbre to restore the speech with high timbre similarity, meeting the needs of rapid customization of timbre in different speech synthesis scenarios.

[0108] The vocoder can achieve high-fidelity speech synthesis in different scenarios, such as high-expressive speech synthesis, or speaker voice cloning in complex scenarios with low signal-to-noise ratio, and support more interesting voice customization, thereby improving user stickiness on the product by enhancing the user's personality display.

[0109] The embodiments of the present disclosure construct a zero-sample vocoder model through feature decoupling and domain knowledge transfer, and redesign the speech synthesis system pipeline by using decoupled features, which can reduce the difficulty of training the model modeling timbre, thereby improving the prediction effect. In addition, the domain knowledge transfer method is used to use existing high-confidence low-sampling rate audio data to improve scene generalization (emotional speech synthesis, noisy speech synthesis) and timbre coverage, such as emotion recognition data and speaker recognition data.

[0110] Based on the same inventive concept, a speech synthesis device is also provided in an embodiment of the present disclosure, and the method corresponding to the speech synthesis device may be the speech synthesis method in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The speech synthesis device provided in an embodiment of the present disclosure includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the speech synthesis methods and / or technical solutions of the aforementioned multiple embodiments of the present disclosure.

[0111] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0112] In particular, the method and / or embodiment in the embodiment of the present disclosure can be implemented as a computer software program. For example, the embodiment disclosed in the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. When the computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present disclosure are executed.

[0113] Another embodiment of the present disclosure further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present disclosure.

[0114] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0115] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0116] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0117] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0120] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0121] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0122] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0123] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0124] It should be noted that the various embodiments of the present disclosure may be implemented in software and / or a combination of software and hardware, for example, may be implemented using an application specific integrated circuit (ASIC), a general purpose computer, or any other similar hardware device. In one embodiment, the software program of the various embodiments of the present disclosure may be executed by a processor to implement the steps or functions described above. Similarly, the software program of the various embodiments of the present disclosure (including related data structures) may be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive, or a floppy disk and the like. In addition, some steps or functions of the various embodiments of the present disclosure may be implemented using hardware, for example, as a circuit that cooperates with a processor to perform various steps or functions.

[0125] In addition, at least a portion of the embodiments of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computing device, can call or provide the method and / or technical solution according to the embodiments of the present disclosure through the operation of the computing device. The program instruction for calling / providing the method of the embodiments of the present disclosure may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in a working memory of a computing device that runs according to the program instruction.

[0126] For those skilled in the art, it is obvious that the embodiments of the present disclosure are not limited to the details of the above exemplary embodiments, and the embodiments of the present disclosure can be implemented in other specific forms without departing from the spirit or basic features of the embodiments of the present disclosure. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the embodiments of the present disclosure is limited by the attached claims rather than the above description, so it is intended to include all changes within the meaning and scope of the equivalent elements of the claims in the embodiments of the present disclosure. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "including" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

Claims

1. A speech synthesis method, wherein: The method comprises the following steps: According to the input text, obtain its corresponding discrete semantic features; Mapping the discrete semantic features and the audio features of the input text into the same feature space to align the two; Extracting pronunciation and prosody features corresponding to the input text from the aligned features; Converting the pronunciation and prosody features and reference timbre features into mel-spectrogram features; The mel-spectrogram features are converted into output speech.

2. The method according to claim 1, wherein: The audio features are discretized pronunciation and prosody features obtained based on the audio signal corresponding to the input text: Wherein, the pronunciation and prosodic features corresponding to the input text are extracted from the aligned discrete semantic features.

3. The method according to claim 1, wherein: The pronunciation and prosody-related features corresponding to the input text are extracted from the aligned audio features.

4. The method according to claim 1, wherein: The reference timbre feature is consistent with the target timbre feature.

5. The method according to claim 1, wherein: The step of converting the pronunciation and prosody features and the reference timbre features into mel-spectrogram features is performed by a trained Transformer model; The training data set of the Transformer model includes audio data of different sampling rates; When the Transformer model is trained based on the zero-shot learning method, the training is obtained by masking part of the Mel-spectrogram features of the audio data in the training data set and then predicting the complete Mel-spectrogram features.

6. The method according to claim 5, wherein: The training data set records the sampling rate of each audio data; The sampling rate feature of the audio data and its mel-spectrogram feature are concatenated and used to train the Transformer model.

7. The method according to claim 5 or 6, wherein: The training data set further includes corresponding audio data with a low sampling rate obtained by downsampling the audio data with a high sampling rate.

8. A speech synthesis device, wherein: The device includes: Acoustic models for: According to the input text, obtain its corresponding discrete semantic features; Mapping the discrete semantic features and the audio features of the input text into the same feature space to align the two; Extracting pronunciation and prosody features corresponding to the input text from the aligned features; Vocoder for: Converting the pronunciation and prosody features and reference timbre features into mel-spectrogram features; The mel-spectrogram features are converted into output speech.

9. A speech synthesis device, wherein: The device comprises a memory and a processor, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the device is configured to perform the following operations: According to the input text, obtain its corresponding discrete semantic features; Mapping the discrete semantic features and the audio features of the input text into the same feature space to align the two; Extracting pronunciation and prosody features corresponding to the input text from the aligned features; Converting the pronunciation and prosody features and reference timbre features into mel-spectrogram features; The mel-spectrogram features are converted into output speech.

10. A computer program product comprising computer program instructions, wherein: When the computer program instructions are executed by a computer device, the computer device is configured to perform the method according to any one of claims 1 to 7.

11. A computer program storage medium storing computer executable instructions, wherein when the computer executable instructions are executed by a computer device, the computer device is configured to perform the method according to any one of claims 1 to 7.

Images