TTS system, speech synthesis method and device, electronic equipment and storage medium

Abstract

The invention provides a TTS system, a speech synthesis method and device, electronic equipment and a storage medium. The speech synthesis method comprises the following steps: acquiring a phoneme sequence corresponding to a target text; the condition features of the reference speech are extracted, the reference speech is processed into discretized coded representation, and the condition features comprise acoustic features and / or semantic features; inputting the phoneme sequence, the condition feature and the coded representation into a pre-trained speech synthesis model, and obtaining target speech which is output by the speech synthesis model and matched with the target text; wherein the speech synthesis model comprises a neural coding and decoding language model and an audio decoder connected with the neural coding and decoding language model, and the neural coding and decoding language model comprises an autoregression module and does not comprise a non-autoregression module.

Description

Technical Field

[0001] The present invention relates to the field of AI (Artificial Intelligence), and in particular to a TTS system, a speech synthesis method, a device, an electronic device and a storage medium. Background Art

[0002] As an important branch of AI, TTS (Text-to-Speech) combines speech synthesis, artificial intelligence, natural language generation and other related technologies, aiming to convert plain text into natural speech recognizable by human ears. TTS has been widely used in intelligent voice broadcasting, navigation, smart speakers, voice assistants and other products.

[0003] At present, large model technology is increasingly being used in speech synthesis technology. Currently, conditional language models based on language models and regression algorithms are commonly used to implement TTS, such as the classic VALL-E (Voice Allocation Language and Expression) model and its variant VALL-E2 model (compared to the VALL-E model, this model uses repeated perceptual sampling and grouped code modeling technology), VALL-EX model (this model only needs a very short recording to achieve voice cloning), etc. This type of model usually includes AR (Auto regressive) modules and NAR (non-AR) modules, with a relatively complex structure. The large number of model parameters leads to a large amount of calculation, so the reasoning efficiency is low and needs to be improved.

[0004] Taking the classic VALL-E model as an example, the model proposed a speech synthesis method based on deep learning, which achieved better speech synthesis effect than traditional technology by assigning speech to language and emotion. However, the complex internal structure of the model still has defects such as large amount of calculation and low reasoning efficiency. Summary of the invention

[0005] In view of this, the present invention provides a TTS system, a speech synthesis method, an apparatus, an electronic device and a storage medium to solve the deficiencies in the related art.

[0006] Specifically, the present invention is achieved through the following technical solutions:

[0007] According to a first aspect of the present invention, there is provided a speech synthesis method, comprising:

[0008] Acquiring a phoneme sequence corresponding to the target text; and extracting conditional features of a reference speech and processing the reference speech into a discretized coding representation, wherein the conditional features include acoustic features and / or semantic features;

[0009] The phoneme sequence, the conditional features and the encoded representation are input into a pre-trained speech synthesis model, and a target speech output by the speech synthesis model that matches the target text is obtained; wherein the speech synthesis model includes a neural codec language model and an audio decoder connected thereto, and the neural codec language model includes an autoregressive AR module and does not include a non-autoregressive NAR module.

[0010] According to a second aspect of the present invention, there is provided a speech synthesis device, comprising:

[0011] A feature acquisition unit, configured to acquire a phoneme sequence corresponding to a target text; and extract conditional features of a reference speech and process the reference speech into a discretized coding representation, wherein the conditional features include acoustic features and / or semantic features;

[0012] A speech synthesis unit is used to input the phoneme sequence, the conditional features and the encoded representation into a pre-trained speech synthesis model, and obtain the target speech output by the speech synthesis model that matches the target text; wherein the speech synthesis model includes a neural codec language model and an audio decoder connected thereto, and the neural codec language model includes an autoregressive AR module and does not include a non-autoregressive NAR module.

[0013] According to a second aspect of the present invention, a TTS system is provided, comprising a speech synthesis model, wherein the speech synthesis model comprises a neural codec language model and an audio decoder connected thereto, wherein the neural codec language model comprises an autoregressive AR module and does not comprise a non-autoregressive NAR module; wherein:

[0014] The neural codec language model is used to receive a phoneme sequence corresponding to a target text, conditional features of a reference speech, and a discretized coded representation obtained by processing the reference speech, wherein the conditional features include acoustic features and / or semantic features; and send a low-dimensional discretized feature obtained by processing the phoneme sequence, the conditional features, and the coded representation to the audio decoder;

[0015] The audio decoder is used to generate and output a target speech matching the target text according to the low-dimensional discretization features.

[0016] According to a fourth aspect of the present invention, there is provided an electronic device, comprising:

[0017] a processor; a memory for storing instructions executable by the processor;

[0018] The processor implements the steps of the method described in the first aspect by running the executable instructions.

[0019] According to a fifth aspect of the present invention, there is provided a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0020] The present invention improves the conventional conditional language model and proposes an improved TTS system and a speech synthesis method implemented therein. In the embodiment of the present invention, the phoneme sequence corresponding to the target text is first obtained; and conditional features such as acoustic features and / or semantic features of the reference speech are extracted and the reference speech is processed into a discretized coding representation; further, the phoneme sequence, conditional features and coding representation are input into a pre-trained speech synthesis model and the target speech outputted by the pre-trained speech synthesis model is obtained to match the target text; wherein the speech synthesis model includes a neural codec language model and an audio decoder connected thereto, and the neural codec language model includes an AR module and does not include a NAR module.

[0021] Conventional conditional language models (such as the aforementioned classic VALL-E model) usually include both AR modules and NAR modules. During the reasoning process, only text features (i.e., phoneme sequences) and discretized features of the reference speech (i.e., the discretized encoding representation) are used as model inputs, and the NAR module extracts the conditional features of the reference speech (i.e., acoustic features and / or semantic features).

[0022] Unlike conventional conditional language models, this solution pre-extracts the conditional features of the reference speech and directly inputs them into the model as input parameters. This parameter input method allows the speech synthesis model to contain only valid AR modules (i.e., only AR modules participate in reasoning). It can be seen that this solution can greatly simplify the effective network structure inside the speech synthesis model without reducing the quality of speech synthesis by pre-placing the extraction of conditional features outside the speech synthesis model (for example, the NAR part in the model can be eliminated or frozen, and only the AR part can be retained as the effective network), thereby reducing the internal parameters and calculation amount of the model and significantly improving the reasoning efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a structural schematic diagram of a VALL-E model shown in an embodiment of the present invention;

[0025] Figure 2is a flow chart of a training process of a speech synthesis model and a neural vocoder shown in an embodiment of the present invention;

[0026] Figure 3 is a structural schematic diagram of a speech synthesis model to be trained shown in an embodiment of the present invention;

[0027] Figure 4 is a flow chart of a training method for a neural encoding and decoding language model shown in an embodiment of the present invention;

[0028] Figure 5 is a schematic diagram of the structure of a TTS system shown in an embodiment of the present invention;

[0029] Figure 6 is a flow chart of a speech synthesis method shown in an embodiment of the present invention;

[0030] Figure 7 is a schematic structural diagram of an electronic device shown in an embodiment of the present invention;

[0031] Figure 8 The block diagram of a speech synthesis device shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are only examples of devices and methods consistent with some aspects of the present invention.

[0033] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0034] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0035] As an important branch of AI, TTS combines speech synthesis, artificial intelligence, natural language generation and other related technologies, aiming to convert plain text into natural speech recognizable by human ears. TTS has been widely used in intelligent voice broadcasting, navigation, smart speakers, voice assistants and other products.

[0036] At present, large model technology is increasingly being used in speech synthesis technology. Currently, conditional language models based on language models and regression algorithms are often used to implement TTS. Such models usually include AR modules and NAR modules, with complex structures. The large number of model parameters leads to large amounts of calculation, and thus the reasoning efficiency is low, which needs to be improved.

[0037] Taking the typical VALL-E model as an example, the model proposes a speech synthesis method based on deep learning. By assigning speech to language and emotion, it achieves better speech synthesis effect than traditional technology. The VALL-E model actually proposes a TTS framework with contextual learning capabilities, which regards TTS as a conditional language model task and uses audio codec code as the intermediate representation, replacing the traditional Mel-spectrogram (Mel-spectrogram).

[0038] Figure 1 FIG. 1 is a schematic diagram of a typical VALL-E model shown in an embodiment of the present invention. Figure 1 As shown in the figure, the VALL-E model includes a neural codec language model and its connected phoneme converter, audio encoder and audio decoder. The neural codec language model consists of an AR module and a NAR module. The model is used to output a personalized voice whose timbre and intonation are close to the reference voice and match the input text.

[0039] However, since the input of the VALL-E model is text features (i.e., features output by the phoneme converter) and discretized features of the reference speech (i.e., features output by the audio encoder), and the neural encoding and decoding language model is built using a training method that combines AR and NAR, the model needs to extract conditional features of the audio by the NAR module. The AR module and the NAR module respectively extract low-dimensional discretized features and high-dimensional discretized features based on the aforementioned information input, and output them to a conventional audio decoder. For example, in the case where the VALL-E model includes eight quantizers, the first quantizer corresponding to the AR module outputs low-dimensional discretized features such as speaker features to the audio decoder, and the last seven quantizers corresponding to the NAR module output more detailed acoustic details to the audio decoder. Finally, the audio decoder decodes all the discretized features provided by the eight quantizers into audio and outputs them. It can be seen that the network architecture of the AR and NAR hierarchical feature extraction is relatively complex, and the large number of model parameters leads to low inference efficiency.

[0040] In view of the above technical problems existing in the conditional language model in the related art, the present invention proposes an improved TTS system (composed of a brand-new speech synthesis model proposed in this solution) and a speech synthesis method implemented by the system. The system effectively simplifies the effective network structure inside the neural encoding and decoding language model (for example, the NAR part in the model can be eliminated or frozen, and only the AR part is retained as the effective network), thereby reducing the network complexity and the number of model parameters, and significantly improving the reasoning efficiency.

[0041] The embodiments of the speech synthesis scheme of the present invention are described in detail below in conjunction with the accompanying drawings. In the following embodiments, the aforementioned typical VALL-E model is used as an example for comparison and explanation, but it does not mean that the present scheme is only improved for the typical VALL-E model. In fact, as mentioned above, the present scheme is an improvement made for any form of conditional language model (such as the aforementioned VALL-E2, VALL-EX, etc.) built based on a language model and a regression algorithm, and this is hereby explained.

[0042] The speech synthesis method described in the embodiment of the present invention needs to be implemented based on a new speech synthesis model. The training process of the model is first introduced as a whole. The speech synthesis model described in the embodiment of the present invention includes a neural codec language model and an audio decoder, wherein the audio decoder can be the same as the audio decoder included in the conventional VALL-E model, that is, the audio decoder in the VALL-E model can be directly used as the audio decoder in the speech synthesis model to reduce the workload of model modification.

[0043] In fact, the last step of the aforementioned VALL-E model directly uses a conventional audio decoder to decode the coded information output by the neural codec language model, which may cause the decoding result (i.e., the personalized voice output by the model at the end) to be distorted. In view of this, the audio decoder in the speech synthesis model to be trained may not use the conventional audio decoder in the VALL-E model, but use a neural vocoder built based on a neural network as a new audio decoder. In this case, in order to improve the degree of consistency between the neural vocoder and the AR module in the neural codec language model, in order to avoid distortion of the output result, the neural vocoder can be further trained based on the trained neural codec language model. At this point, the speech synthesis model to be trained may include a codec model to be trained and a vocoder to be trained, wherein the former also includes an AR module to be trained and a NAR module to be trained. Therefore, the training process for the speech synthesis model proposed in the embodiment of this specification can be divided into two stages: the first stage is used to train the codec model to be trained (this stage is essentially training the AR module to be trained) to obtain a neural codec language model, and the second stage is used to train the vocoder to be trained using the AR module trained in the first stage to obtain the neural vocoder.

[0044] like Figure 2 The training flow chart shown in the figure intuitively shows the data association between the various models involved in the above two stages.

[0045] Phase 1: For the codec model to be trained including the AR module to be trained and the NAR module to be trained, the model is trained using the first sample data to obtain an intermediate model including the AR module and the NAR module. Then, the NAR module is removed or frozen from the intermediate model to obtain a neural codec language model, in which only the AR module is in an available state, that is, only this module can participate in the calculation in the subsequent reasoning process.

[0046] The second stage: the second sample data is input into the neural encoding and decoding language model trained in the first stage to obtain the second sample low-dimensional discretization feature output by the model, and then the feature is used as the second sample data to train the vocoder to be trained to obtain the trained neural vocoder.

[0047] At this point, the neural encoding and decoding language model trained in the first stage and the neural vocoder trained in the second stage together constitute a speech synthesis model, which can be used for subsequent reasoning to generate a target speech that matches the target text. The specific forms of the various data involved in the above two stages can be found in the records of the embodiments below, which will not be repeated here. The two training stages are described in detail below in conjunction with the accompanying drawings.

[0048] Figure 3It is a structural diagram of a speech synthesis model to be trained shown in an embodiment of the present invention. The model has not yet started training at this time. It includes a codec model to be trained and a vocoder to be trained, wherein the codec model to be trained includes an AR module to be trained and a NAR module to be trained. It can be understood that the first stage is the process of training the codec model to be trained to obtain a neural codec language model, and the second stage is the process of training the vocoder to be trained to obtain a neural vocoder.

[0049] Corresponding to the first stage, Figure 4 It is a flowchart of a method for training a neural codec language model shown in an embodiment of the present invention, and the method includes the following steps 402-406. Among them, step 402 is used to prepare a training sample (i.e., first sample data), step 404 is used to perform model training using the training sample, and step 406 is used to post-process the training result to obtain the neural codec language model.

[0050] Step 402, obtaining a first sample phoneme sequence corresponding to the first sample text; and extracting a first sample conditional feature of a first sample reference speech and processing the first sample reference speech into a discretized first sample encoding representation, wherein the first sample conditional feature includes a first sample acoustic feature and / or a first sample semantic feature.

[0051] First of all, it should be noted that the training method of the speech synthesis model can be applied to a model training device, which can be a server, for example, a physical server including an independent host, or a virtual server, a cloud server, etc. carried by a host cluster. Of course, in the above training process, relevant technicians often need to interact with the server through a client running in the electronic device they use, such as uploading training samples, setting loss functions, adjusting model parameters, etc. The electronic device can be at least one of a PC (Personal Computer), a mobile phone, a tablet device, a laptop, a PDA (Personal Digital Assistants), a wearable device (such as smart glasses, smart watches, etc.), a VR (Virtual Reality) device, an AR (Augmented Reality) device, etc., and one or more embodiments of this specification do not limit this. The client can be started and run on the electronic device. The client can be a native application (Application, i.e., APP) installed on the electronic device, or it can also be a mini program (Mini Program), a quick application (Quick APP) or other similar forms. Of course, when using web technologies such as HTML5 or similar, relevant functions can be implemented through pages displayed by browsers. The browser here can be an independent browser application or a browser module embedded in certain applications.

[0052] Before model training, the first sample text and the first sample reference speech need to be determined. The first sample text is the text that needs to be matched by the first predicted speech to be generated by the speech synthesis model (that is, the first predicted speech matches the first sample text), and the first sample reference speech is the speech used to give the first predicted speech corresponding timbre, intonation and other personalized speech characteristics. Exemplarily, when the first sample text is the Chinese "rabbit brothers are pulling radishes in the field" and the first sample reference speech is the speech emitted by the natural person P1 when reading the text "consumers' requirements for miniaturization of smart phones are constantly increasing", the speech synthesis model to be trained during the training process is used to output the first predicted speech for reading the text "rabbit brothers are pulling radishes in the field", and the first predicted speech should have the personalized speech characteristics of the natural person P1 - the first predicted speech is as if it were spoken by the natural person P1.

[0053] In addition, the first sample text is text that can be read in natural language. The embodiment of the present invention does not limit the language to which the text belongs, the number of characters contained, etc., for example, it can be Chinese, English, Japanese, Korean, Russian, etc. The first sample reference speech is a speech corresponding to a natural language. The speech can be obtained by speaking and recording, or it can also be generated by other TTS technologies or related models. The embodiment of the present invention does not limit this. The reference speech can be a pure speech without noise (or a speech with noise but the noise ratio is lower than a preset value); of course, it can also be a speech containing a certain amount of noise, which is not limited by the embodiment of the present invention.

[0054] Before training the codec model to be trained, it is necessary to first obtain a first sample phoneme sequence corresponding to the first sample text, which is composed of the phonemes corresponding to the characters in the first sample text, arranged in sequence according to the order of the characters in the text. In one embodiment, the first sample phoneme sequence corresponding to the first sample text can be obtained according to the steps of vocabulary analysis, phoneme splitting and phoneme connection. Exemplarily, each character in the first sample text (such as Chinese characters, words, English words, English phrases, etc.) can be analyzed to determine the pronunciation of each character, and then the pronunciation of each character can be split into individual phonemes. Finally, according to the position and front-to-back order of each word in the first sample text, each phoneme is connected in sequence to obtain the first sample phoneme sequence corresponding to the first sample text.

[0055] Alternatively, the first sample text may be input into a pre-trained G2P (Grapheme to Phoneme) model, and a first sample phoneme sequence corresponding to the first sample text output by the G2P model may be received. G2P is a key step in speech synthesis, and its purpose is to convert written forms (letters, characters, etc.) into their corresponding pronunciation forms (phonemes). This process usually needs to consider the context, because the pronunciation of a character or letter may be different in different words or sentences. The G2P model can be pre-trained using a large-scale annotated corpus so that the model learns the mapping relationship between characters and phonemes, thereby determining the phonemes corresponding to each character in the first sample text according to the mapping relationship, and then generating the corresponding first sample phoneme sequence.

[0056] The first sample phoneme sequence may be embedded in the IPA (International Phonetic Alphabet) with a finer granularity to alleviate problems such as missing words and repeated readings.

[0057] In addition, for the aforementioned first sample reference speech, it is also necessary to extract the first sample conditional features of the speech and process the speech into a discretized first sample coding representation. The first sample conditional features include first sample acoustic features (such as pitch, rhythm, pronunciation pattern, etc.) and / or first sample semantic features (i.e., contextual semantics). It can be understood that this solution extracts the first sample acoustic features and / or first sample semantic features of the first sample text as input to the subsequent speech synthesis model, so that the model can ultimately generate a more natural and fluent first predicted speech based on the first sample low-dimensional discretized features of higher quality and carrying more comprehensive information.

[0058] In one embodiment, when extracting the above-mentioned first sample conditional features of the reference speech, the first sample reference speech can be input into a pre-trained Hubert extractor, and the first sample conditional features of the first sample reference speech outputted after extraction by the Hubert extractor are received. Among them, the Hubert (Hidden-Unit BERT) is a feature extraction technology based on self-supervised learning, which is used to learn rich speech representations from unlabeled speech data. It can capture various semantic and acoustic information in speech signals through a multi-layer structure, such as speech features, pronunciation patterns, and contextual semantic information. In this way, the conditional features of the reference speech can be automatically and efficiently extracted by the pre-trained Hubert extractor.

[0059] The Hubert extractor can be trained by a self-supervised learning method, that is, learning speech representation by predicting hidden signals or features. Unlike traditional supervised learning, self-supervised learning does not require a large amount of labeled data, but can directly use the original unlabeled speech signal for training, which reduces sample requirements and data costs, and helps to train the Hubert extractor at low cost. In addition, the conditional features output by the Hubert extractor may include multiple levels, such as low-level features including the acoustic features, high-level features including the semantic features, etc. Through the above-mentioned hierarchical feature representation, the Hubert extractor can generate rich speech features, which are suitable for a variety of speech tasks.

[0060] In addition to using the Hubert extractor, the first sample conditional feature can also be extracted according to the steps of pre-emphasis, framing, windowing and feature extraction. Exemplarily, the high-frequency signal in each speech frame can be pre-emphasized by a first-order high-pass filter to improve the resolution of this part of the signal. Then the continuous speech after pre-emphasis is divided into small segments of fixed length, called speech frames (such as 10 to 30ms per frame), and there can be partial overlap between frames (such as the overlap rate can be 50%), so as to avoid information omission and ensure a smooth transition between adjacent frames. Further, in view of the problem that there may be signal discontinuity at the two end edges of adjacent speech frames after framing, each continuous speech frame can be windowed to eliminate the above discontinuity. Specifically, window functions such as Hamming window, rectangular window and Hanning window can be used for windowing. After the above pre-processing is completed, the conditional feature can be extracted from the pre-processing result. For example, the processed time domain signal can be subjected to a Fast Fourier Transform (FFT) to convert the time domain signal of each frame into a frequency domain signal, and then the Mel Frequency Cepstral Coefficients (MFCC) or filter bank features (FBank) can be calculated as the first sample acoustic feature. In addition, the pre-processed speech signal can be converted into text using a speech recognition engine; the text can then be further analyzed using natural language processing technology (such as lexical analysis, syntactic analysis, and semantic understanding) to extract grammatical information such as grammatical structure, word meaning, and syntactic relationship, so as to understand the deep meaning of the speech; finally, the context and context are considered, that is, the semantic features can be more accurately determined by combining the previous and next relationships of the first sample benchmark speech and the scene of the conversation.

[0061] In one embodiment, when processing the first sample reference speech into a discretized first sample coding representation, the first sample reference speech can be first divided into a plurality of sample speech segments, and the residual between the original signal and the predicted signal of each sample speech segment can be calculated; then the residual of each sample speech segment can be vectorized to determine the matching code vector, and finally each code vector can be encoded to generate a discretized coding representation. Alternatively, the first sample reference speech can also be input into a pre-trained neural encoder for inference calculation, and the discretized first sample coding representation output by the neural encoder can be received. The neural encoder is used to compress the reference speech into a discretized coding representation through a residual vector quantization algorithm. In the above manner, the first sample reference speech can be compressed from an audio signal into a compact, low-bit rate representation, in which the key information of the speech signal is retained, thereby significantly reducing the amount of data while ensuring the sound quality, and reducing the calculation pressure of the subsequent neural coding and decoding language model to be predicted.

[0062] In one embodiment, for the sample reference text matching the first sample reference speech, a sample reference phoneme sequence corresponding to the sample reference text can also be obtained, and the sample reference phoneme sequence is associated with the first sample phoneme sequence corresponding to the first sample text and input into the speech synthesis model to be trained. Wherein, in the case where the first sample reference speech is annotated with a corresponding text, the text can be determined as the sample reference text matching the first sample reference speech; and in the case where the first sample reference speech is not annotated with a corresponding text, the text of the first sample reference speech can be identified by speech-to-text (STT) technology and used as the sample reference text. This method can introduce the sample reference phoneme sequence of the sample reference text corresponding to the first sample reference speech during the training process of the speech synthesis model, thereby helping the speech synthesis model to learn more comprehensive speech-related information and improve the quality of speech synthesis.

[0063] Still taking the aforementioned first sample text as the Chinese “rabbit brothers are pulling radishes in the field” and the first sample reference voice as the voice uttered by the natural person P1 when reading the text “consumers’ requirements for the miniaturization of smart phones are constantly increasing” (served as the first sample reference voice) as an example, at this time, the sample reference text corresponding to the first sample reference voice can be determined as “consumers’ requirements for the miniaturization of smart phones are constantly increasing” in the aforementioned manner, and then the phoneme sequence of the text can be extracted in the aforementioned manner as the sample reference phoneme sequence, and finally the sample reference phoneme sequence is concatenated with the aforementioned first sample phoneme sequence (the concatenation result can be the phoneme sequence corresponding to the text “rabbit brothers are pulling radishes in the field and consumers’ requirements for the miniaturization of smart phones are constantly increasing”) and input into the speech synthesis model to be trained.

[0064] Step 404: The codec model to be trained is trained using the first sample data consisting of the first sample phoneme sequence, the first sample conditional feature and the first sample coding representation. The codec model to be trained also includes an AR module to be trained and a NAR module to be trained. The difference between the first sample reference speech and the first predicted speech output by the model to be trained is used as the model loss of the codec model to be trained.

[0065] After obtaining information such as the first sample phoneme sequence, the first sample conditional features, and the first sample encoding representation in the aforementioned manner, the first sample data composed of these information can be used to train the neural codec language model, that is, the first sample data is input into the codec model to be trained for training, and a trained neural codec language model is obtained.

[0066] Wherein, the first sample data may include multiple data, each of which may include a first sample text and a first sample reference voice corresponding to a group of first sample phoneme sequences, the first sample conditional features and the first sample encoding representation and other information. In addition, before training, all the first sample data can be divided into a training set (Training Set), a test set (Test Set) and a validation set (Validation Set). Wherein, the test set is a data set used to train the machine learning model. During training, the model uses the samples in the training set to learn features and patterns in order to make predictions or classifications. The test set is a data set used to evaluate the performance of the trained model. After the model training is completed, the test set is used to verify the generalization ability of the model to unknown data. The performance of the model on the test set can help evaluate the accuracy and performance of the model. The validation set is a data set used to adjust the model hyperparameters and evaluate the performance of the model. During the training process, the validation set is used to adjust the parameters of the model to prevent the model from overfitting on the training set. In order to balance training efficiency and quality, an appropriate ratio can be set for the sizes of the training set, test set, and validation set. For example, the number of first sample data items contained in the three can be reduced successively. For example, 60% of all the first sample data can be used as the training set, and the remaining 20% ​​can be used as the test set and validation set respectively; or, when the total number of first sample data is large, 80% of all the first sample data can be used as the training set, and the remaining 10% can be used as the test set and validation set respectively. The details will not be repeated here.

[0067] It can be seen from the aforementioned embodiments that the input parameters of the neural codec language model to be trained described in this scheme include not only the first sample phoneme sequence corresponding to the first sample text and the discretized first sample coding representation obtained by processing the first sample reference speech, but also the first sample conditional feature of the first sample reference speech, that is, the process of obtaining the first sample conditional feature is pre-placed outside the speech synthesis model, which enables the AR module in the codec model to be trained to directly use the relevant information in the sample conditional feature to generate the first low-dimensional discretized feature and output it to the vocoder to be trained, without having to extract this information through the NAR module - this is why the NAR module can be directly removed or frozen from the intermediate model later (without affecting the normal operation of the speech synthesis model and achieving the expected function).

[0068] In one embodiment, when the speech synthesis model is trained using the aforementioned first sample data, the first sample phoneme sequence, the first sample conditional feature and the first sample encoding representation can be spliced ​​and input into the input end of the AR module to be trained, so that the module can send the processed first sample low-dimensional discretized features to the audio decoder to be trained; and receive the first predicted speech (such as the first sample text) generated and output by the audio decoder to be trained based on at least the first sample low-dimensional discretized features. Figure 4 Finally, according to the first predicted speech and the first sample reference speech (such as Figure 4 The parameters of the AR module to be trained are adjusted according to the difference between the first sample reference speech S1) shown in the figure and the training is continued until the preset termination condition is met to terminate the training, and the intermediate model can be obtained. Among them, the low-dimensional discretization feature can be regarded as the intermediate acoustic feature of the first sample reference speech, and the feature can be in the form of a feature vector or a Mel spectrum (its specific form needs to be able to be smoothly recognized and processed by the subsequent audio decoder). Exemplarily, the first sample conditional feature and the first sample coding representation can be in vector form (that is, the two are respectively corresponding feature vectors), and the above-mentioned multiple feature vectors can be spliced ​​in a vector cascade manner. In addition, the termination condition may be that the error of the test result and / or the verification result does not exceed the preset value, or the test set has been used up, etc., and the embodiments of the present invention are not limited to this.

[0069] Step 406, remove or freeze the NAR module from the trained intermediate model to obtain the speech synthesis model.

[0070] For the AR module and NAR module in the intermediate model trained in the above manner, the NAR module can be directly deleted to obtain a speech synthesis model containing only the AR module; or the NAR module in the intermediate model can be frozen to obtain a speech synthesis model containing both the AR module (in an available state) and the NAR module (in an unavailable state). Regardless of the processing method, only the AR module in the speech synthesis module is in an available state and can participate in the subsequent reasoning process.

[0071] Correspondingly, in the speech synthesis model to be trained, the codec model to be trained can be connected to the vocoder to be trained through at least one quantizer corresponding to the AR module to be trained. Taking a quantizer as an example, the AR module to be trained can send the first sample low-dimensional discretization feature extracted by itself to the vocoder to be trained through the quantizer. It can be understood that because the present scheme puts the conditional feature extraction process in front, that is, the first sample conditional feature is directly input as a parameter into the codec model to be trained, the first sample low-dimensional discretization feature output by the AR module to be trained may be different from the low-dimensional discretization feature output by the AR module in the VALL-E model. The specific difference is determined by the internal structure and specific parameters of the AR module, and the embodiment of the present invention is not limited to this.

[0072] In one embodiment, the audio decoder is a neural vocoder built and pre-trained based on a neural network. At this time, the neural vocoder can be trained in the following manner (i.e., the aforementioned second stage): first obtain a sample phoneme sequence corresponding to the second sample text; and extract the sample conditional features of the second sample reference speech and process the second sample reference speech into a discretized second sample encoding representation, wherein the second sample conditional features include second sample acoustic features and / or second sample semantic features; then input the second sample phoneme sequence, the second sample conditional features, and the second sample encoding into the neural encoding and decoding language model, and receive the second sample low-dimensional discretization features outputted by the model. Then, the second sample low-dimensional discretization features are inputted as the second sample data into the vocoder to be trained, and the second predicted speech outputted by the vocoder to be trained that matches the second sample text is received; finally, the parameters of the vocoder to be trained are adjusted according to the difference between the second predicted speech and the second sample reference speech, and the training is continued until the preset termination condition is met and the training is terminated to obtain the neural vocoder. Similar to the aforementioned low-dimensional discretization features of the first sample, the second low-dimensional discretization features can be sent to the neural vocoder by the AR module through the aforementioned first (or multiple) quantizers.

[0073] In this way, the neural vocoder can be further trained on the basis of the already trained neural codec language model (mainly the AR module therein), thereby ensuring that the later trained neural vocoder fully matches the output characteristics of the previously trained AR module, and can make the amount of calculation and the amount of parameters that need to be adjusted in each stage as small as possible, avoiding the problem of model non-convergence caused by excessive number of parameters or overly complex parameter correlation, and improving the overall training efficiency of the speech synthesis model.

[0074] Among them, the aforementioned first sample data can be directly used as the aforementioned second sample data. Alternatively, in order to ensure that the speech synthesis model has a high generalization ability, second sample data that is different from the first sample data can also be used. At this time, the various data in the second sample data can be obtained in the same manner as the manner in which the various data in the first sample data are obtained, such as using a G2P model to generate a second sample phoneme sequence corresponding to the second sample text, using a Hubert extractor to extract the second sample conditional features of the second sample reference speech, using a neural encoder to compress the second sample reference speech into a discretized second sample coding representation through a residual vector quantization algorithm, etc., which will not be repeated here.

[0075] In addition, the vocoder to be trained can adopt a Vocos vocoder to be trained based on a GAN (Generative Adversarial Networks, Generative Adversarial Network) network; correspondingly, the neural vocoder in the speech synthesis model after training is a Vocos vocoder. Among them, as the last link in the neural network included in the speech synthesis model, the Vocos vocoder is used to convert the low-dimensional discretized features (this feature is essentially also a discretized coding representation) generated by the AR module in the neural coding and decoding language model into a speech signal (i.e., a speech waveform audible to the human ear). The role of the Vocos vocoder is similar to that of the vocoder in the traditional TTS system (such as the audio decoder in the aforementioned VALL-E model), but because it is constructed and trained based on a neural network (the structure and parameters of the network are obtained by training through the aforementioned process based on a large amount of data), it is possible to efficiently capture the complex patterns in the reference speech and apply them to the generated target speech, so that the generated target speech is more natural and has a higher quality.

[0076] After the training process of the above two stages is completed, the training process of the speech synthesis model is completed. At this time, the neural codec language model (containing only the valid AR module) and the neural vocoder together constitute the speech synthesis model. This model is different from the VALL-E model in the related art in terms of internal structure and model input, which will not be repeated here.

[0077] In addition, the present invention also proposes a TTS system constructed based on the speech synthesis model trained by the above-mentioned embodiment, the system includes a speech synthesis model, the speech synthesis model includes a neural codec language model and a neural vocoder connected thereto, the neural codec language model includes an autoregressive AR module and does not include a non-autoregressive NAR module; wherein,

[0078] The neural codec language model is used to receive a phoneme sequence corresponding to a target text, conditional features of a reference speech, and a discretized coded representation obtained by processing the reference speech, wherein the conditional features include acoustic features and / or semantic features; and send a low-dimensional discretized feature obtained by processing the phoneme sequence, the conditional features, and the coded representation to the neural vocoder;

[0079] The neural vocoder is used to generate and output a target speech matching the target text according to the low-dimensional discretization features.

[0080] In one embodiment, the TTS system may further include at least one of the following:

[0081] A G2P model is used to generate a phoneme sequence corresponding to the target text and input the phoneme sequence into the neural encoding and decoding language model;

[0082] A Hubert extractor, configured to extract conditional features of the reference speech and input the conditional features into the neural codec language model;

[0083] A neural encoder is used to compress the reference speech into a discretized coding representation through a residual vector quantization algorithm.

[0084] Figure 5 FIG. 1 is a schematic diagram of the structure of a TTS system shown in an embodiment of the present invention. Figure 5 As shown, the system includes a speech synthesis model trained in the aforementioned manner as well as a G2P model, a Hubert extractor and a neural encoder.

[0085] The speech synthesis model includes a neural codec language model and a neural vocoder. As mentioned above, the neural codec language model may only include an AR module but not a NAR module ( Figure 5 That is, it is drawn according to this situation); or, it can also include both the AR module and the NAR module, but only the AR module is available and the NAR module is not available. In addition, as mentioned above, the audio decoder in the TTS system can be a traditional audio decoder (such as the audio decoder used by the VALL-E model), or it can be a neural vocoder ( Figure 5 That is, it is drawn according to this situation). Exemplarily, the neural vocoder can be a Vocos vocoder.

[0086] like Figure 5 As shown, the input parameters of the TTS system include the phoneme sequence corresponding to the target text output by the G2P model, the conditional features of the reference speech output by the Hubert extractor (specifically including the acoustic features and / or semantic features of the speech), and the discretized coding representation obtained and output by the neural encoder processing the reference speech. The AR module in the neural encoding and decoding language model can extract the corresponding low-dimensional discretized features based on the above-mentioned phoneme sequence, conditional features and coding representation, and output them to the neural vocoder (such as through the output of the first quantizer connected to it), and finally the neural vocoder decodes the low-dimensional discretized features to obtain the corresponding target speech.

[0087] For example, if the target text is the Chinese text "a group of lambs are frolicking on the grass" and the reference voice is the voice uttered by a natural person P2 when reading the text "the sun sets behind the mountains, the Yellow River flows into the sea", then through the processing of the above-mentioned TTS system, the target voice finally output by the neural vocoder is the voice uttered when reading the text "a group of lambs are frolicking on the grass", and the personalized indicators such as timbre and intonation of the voice are similar to or even very close to those of the natural person P2, making the listener feel as if the natural person P2 is reading the text "a group of lambs are frolicking on the grass".

[0088] It should also be noted that the AR module is built based on the automatic regression method. By receiving the phoneme sequence and coding representation, it can generate low-dimensional discretization features (i.e., low-dimensional discrete quantization coding of speech) for subsequent decoding of the neural vocoder supporting low-dimensional discrete features to generate the target speech. The output of the AR module at the current moment depends on the output at the previous moment, which means that the module will generate discrete quantization coding of speech step by step, and each step will be based on the previous generation result. Among them, the discrete quantization coding of speech includes speech coding corresponding to multiple continuous segments. When predicting the speech coding of the next speech segment, the AR module needs to be based on the previously generated speech segment and the multimodal information of the output (i.e., the conditional feature). The module continuously uses the current speech segment as input to predict the speech coding corresponding to the next speech segment, and finally iterates to generate a complete discrete quantization coding of speech. In addition, the AR module performs reasoning calculations based on the aforementioned phoneme sequence, coding representation, and conditional features. This combination of multi-source input enables the module to take into account contextual semantics and acoustic information when generating the target speech, thereby generating a more natural and fluent target speech.

[0089] Furthermore, the present invention proposes a speech generation method implemented based on the speech synthesis model trained in the aforementioned embodiment, that is, a method of using the speech synthesis model to complete reasoning to generate speech. Figure 6602-604 is a flowchart of the speech synthesis method according to an embodiment of the present invention. The method can be applied to the aforementioned TTS system and includes the following steps 602-604.

[0090] Step 602, obtaining a phoneme sequence corresponding to the target text; and extracting conditional features of a reference speech and processing the reference speech into a discretized coding representation, wherein the conditional features include acoustic features and / or semantic features.

[0091] Step 604: input the phoneme sequence, the conditional features and the encoded representation into a pre-trained speech synthesis model, and obtain the target speech output by the speech synthesis model that matches the target text; wherein the speech synthesis model includes a neural codec language model and an audio decoder connected thereto, and the neural codec language model includes an autoregressive AR module and does not include a non-autoregressive NAR module.

[0092] Similar to the aforementioned training process, the speech generation method (i.e., the reasoning process) can be executed by the operating device of the speech synthesis model (the speech synthesis model is deployed and runs in the device). The device can be a physical server containing an independent host, or a virtual server or cloud server carried by a host cluster. In addition, the caller of the speech synthesis model (such as a user who requests to generate the target speech) needs to interact with the model or related services, so it can run the corresponding client in the electronic device it uses. The electronic device can also be a PC, a mobile phone, a tablet computer, etc., and the client can also be in the form of an APP, a mini-program, an H5 web page, etc., which will not be repeated.

[0093] Before generating the target speech, it is necessary to first determine the target text and the reference speech. The target text is the text that the target speech to be generated by the speech synthesis model needs to match (that is, the target speech matches the target text), and the reference speech is the speech used to give the target speech corresponding timbre, intonation and other personalized speech characteristics. Similar to the aforementioned first sample text and first sample reference speech, and the second sample text and second sample reference speech, the embodiment of the present invention does not restrict the language to which the target text belongs and / or the number of characters contained, as well as the generation method of the reference speech, whether it contains noise, etc., and can be flexibly set according to actual needs or usage scenarios.

[0094] Before calling the speech synthesis model to generate the target speech, it is necessary to first obtain the phoneme sequence corresponding to the target text, which is composed of the phonemes corresponding to each character in the target text, arranged in sequence according to the order of the characters in the target text. In one embodiment, the phoneme sequence corresponding to the target text can be obtained according to the steps of vocabulary analysis, phoneme splitting and phoneme connection. Exemplarily, each character in the target text (such as Chinese characters, words, English words, English phrases, etc.) can be analyzed to determine the pronunciation of each character, and then the pronunciation of each character can be split into individual phonemes. Finally, according to the position and front-to-back order of each word in the target text, each phoneme is connected in sequence to obtain the phoneme sequence corresponding to the target text.

[0095] Alternatively, the target text may be input into the pre-trained G2P model, and the phoneme sequence corresponding to the target text output by the model may be received. G2P is a key step in speech synthesis, and its purpose is to convert written forms (letters, characters, etc.) into their corresponding pronunciation forms (phonemes). This process usually needs to consider the context, because the pronunciation of a character or letter in different words or sentences may be different. The G2P model can be pre-trained using a large-scale annotated corpus so that the model learns the mapping relationship between characters and phonemes, thereby determining the phonemes corresponding to each character in the target text according to the mapping relationship, and then generating the corresponding phoneme sequence.

[0096] The phoneme sequence may be embedded in IPA with finer granularity to alleviate problems such as missing words and repeated readings.

[0097] In addition, for the aforementioned reference speech, it is also necessary to extract the conditional features of the speech and process the speech into a discretized coded representation. The conditional features include acoustic features (such as pitch, rhythm, pronunciation pattern, etc.) and / or semantic features (i.e., contextual semantics). It can be understood that this solution extracts the acoustic features and / or semantic features of the target text as input to the subsequent speech synthesis model, so that the model can ultimately generate a more natural and fluent target speech based on high-quality, low-dimensional discretized features that carry more comprehensive information.

[0098] In one embodiment, when extracting the above-mentioned conditional features of the reference speech, the reference speech can be input into the aforementioned pre-trained Hubert extractor, and the conditional features of the reference speech output by the extractor after extraction are received. In this way, the conditional features of the reference speech can be automatically and efficiently extracted by the pre-trained Hubert extractor. In addition to using the Hubert extractor, the above-mentioned conditional features can also be extracted according to the steps of pre-emphasis, framing, windowing and feature extraction. The specific extraction method can refer to the records of the related embodiments of extracting the first sample conditional features in the same way in the previous text, which will not be repeated here.

[0099] In one embodiment, when processing the reference speech into a discrete coding representation, the reference speech can be first divided into multiple speech segments, and the residual between the original signal and the predicted signal of each speech segment can be calculated; then the residual of each speech segment can be vectorized to determine the matching code vector, and finally each code vector can be encoded to generate a discrete coding representation. Alternatively, the reference speech can also be input into the pre-trained neural encoder for inference calculation, and the discrete coding representation output by the neural encoder can be received. The neural encoder is used to compress the reference speech into a discrete coding representation through a residual vector quantization algorithm. In the above manner, the reference speech can be compressed from an audio signal into a compact, low-bit rate representation, which retains the key information of the speech signal, thereby significantly reducing the amount of data while ensuring the sound quality, and reducing the computational pressure of the subsequent neural encoding and decoding language model.

[0100] In one embodiment, if the first sample phoneme sequence of the sample benchmark text is used as the model input during the training of the speech synthesis model, then in the inference stage, for the benchmark target text matching the benchmark speech, the benchmark phoneme sequence corresponding to the benchmark target text can also be obtained, and the benchmark phoneme sequence and the phoneme sequence corresponding to the target text are associated and input into the speech synthesis model. Wherein, in the case where the benchmark speech is annotated with a corresponding text, the text can be determined as the benchmark text matching the benchmark speech; and in the case where the benchmark speech is not annotated with a corresponding text, the text of the benchmark speech can be identified by STT technology and used as the benchmark text. This method can introduce the benchmark phoneme sequence of the benchmark target text corresponding to the benchmark speech in the inference process of the speech synthesis model, thereby helping the speech synthesis model to learn more comprehensive speech-related information and improve the quality of speech synthesis.

[0101] Still taking the aforementioned target text as "a group of lambs frolicking on the grass" in Chinese, and the reference voice as the voice uttered by the natural person P2 when reading the text "the sun sets behind the mountains, and the Yellow River flows into the sea" (used as the reference voice) as an example, at this time, the reference text corresponding to the reference voice can be determined as "consumers' demand for miniaturization of smartphones continues to increase" in the aforementioned manner, and then the phoneme sequence of the text can be extracted in the aforementioned manner as the reference phoneme sequence, and finally the reference phoneme sequence is spliced ​​with the aforementioned phoneme sequence (the splicing result can be the phoneme sequence corresponding to the text "a group of lambs frolicking on the grass, the sun sets behind the mountains, and the Yellow River flows into the sea") and input into the speech synthesis model.

[0102] After the above-mentioned phoneme sequence, conditional features and coding representation are input into the speech synthesis model, the audio decoder can output the target speech matching the target text after being processed by the model. The specific processing process of the model can be referred to the detailed record of the above-mentioned embodiment, which will not be repeated here.

[0103] It can be seen from the above embodiments that, unlike conventional conditional language models, this solution pre-extracts the conditional features of the reference speech and directly inputs them into the model as input parameters. This parameter input method allows the speech synthesis model to contain only valid AR modules (i.e., only AR modules participate in reasoning). It can be seen that this solution can greatly simplify the effective network structure inside the speech synthesis model without reducing the quality of speech synthesis by pre-placing the extraction of conditional features outside the speech synthesis model (e.g., the NAR part in the model can be eliminated or frozen, and only the AR part can be retained as the effective network), thereby reducing the internal parameters and calculation amount of the model, and significantly improving the reasoning efficiency of the model.

[0104] Figure 7 is a schematic structural diagram of an electronic device shown in an embodiment of the present invention. Figure 7 At the hardware level, the device includes a processor 701, a network interface 702, a memory 703, a non-volatile memory 704, and an internal bus 705, and may also include hardware required for other services. One or more embodiments of the present invention may be implemented based on software, such as the processor 701 reading the corresponding computer program from the non-volatile memory 704 to the memory 703 and then running it. Of course, in addition to the software implementation, one or more embodiments of the present invention do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but may also be hardware or logic devices.

[0105] Figure 8 A block diagram of a speech synthesis device shown in an embodiment of the present invention. Figure 8 , the device can be applied to Figure 7 The device shown in the figure, in order to implement the technical solution described in the present invention, comprises:

[0106] The feature acquisition unit 801 is used to acquire a phoneme sequence corresponding to the target text; and extract conditional features of the reference speech and process the reference speech into a discretized coding representation, wherein the conditional features include acoustic features and / or semantic features;

[0107] The speech synthesis unit 802 is used to input the phoneme sequence, the conditional features and the encoded representation into a pre-trained speech synthesis model, and obtain the target speech output by the speech synthesis model that matches the target text; wherein the speech synthesis model includes a neural codec language model and an audio decoder connected thereto, and the neural codec language model includes an autoregressive AR module and does not include a non-autoregressive NAR module.

[0108] Optionally, the feature acquisition unit 801 is specifically used for:

[0109] Searching for the pronunciation of each character in the target text in order, and splitting each pronunciation into individual phonemes, and arranging the split phonemes in order according to the order of the corresponding words in the target text to form a phoneme sequence corresponding to the target text; or

[0110] The target text is input into a pre-trained grapheme-to-phoneme G2P model, and a phoneme sequence corresponding to the target text output by the G2P model is received.

[0111] Optionally, the feature acquisition unit 801 is specifically used to: input the reference speech into a pre-trained Hubert extractor, and receive the conditional features of the reference speech output after extraction by the Hubert extractor.

[0112] Optionally, the conditional features output by the Hubert extractor include multiple levels, wherein low-level features include the acoustic features and high-level features include the semantic features.

[0113] Optionally, the feature acquisition unit 801 is specifically used for:

[0114] Segmenting the reference speech into a plurality of speech segments, and calculating the residual between the original signal and the predicted signal of each speech segment;

[0115] The residuals of each speech segment are vectorized to determine matching code vectors, and each code vector is encoded to generate a discretized coded representation.

[0116] Optionally, the speech synthesis unit 802 is specifically used for:

[0117] The phoneme sequence, the conditional feature and the coded representation are concatenated and inputted into the input end of the AR module, so that the module can send the processed low-dimensional discretized features to the audio decoder;

[0118] Receive the target speech matching the target text, which is generated and output by the audio decoder according to the low-dimensional discretization features.

[0119] Optionally, it also includes a reference text processing module 803, which is used to: for a reference text matching the reference speech, obtain a reference phoneme sequence corresponding to the reference text, and associate the reference phoneme sequence with the phoneme sequence corresponding to the target text and input them into the speech synthesis model.

[0120] Optionally, the device further includes a speech synthesis unit 804, which is used to:

[0121] Acquire a first sample phoneme sequence corresponding to the first sample text; and extract a first sample conditional feature of a first sample reference speech and process the first sample reference speech into a discretized first sample encoding representation, wherein the first sample conditional feature includes a first sample acoustic feature and / or a first sample semantic feature;

[0122] Using the first sample data consisting of the first sample phoneme sequence, the first sample conditional feature and the first sample coding representation to train the codec model to be trained, the codec model to be trained also includes an AR module to be trained and a NAR module to be trained, and the difference between the first sample reference speech and the first predicted speech output by the model to be trained is used as the model loss of the codec model to be trained;

[0123] For the trained intermediate model including the AR module and the NAR module, the NAR module is removed or frozen to obtain the neural encoding and decoding language model.

[0124] Optionally, the speech synthesis unit 804 is specifically used for:

[0125] The first sample phoneme sequence, the first sample conditional feature and the first sample encoding representation are concatenated and inputted into the input end of the AR module to be trained, so that the module sends the processed first sample low-dimensional discretized feature to the audio decoder to be trained; and, receiving the first predicted speech matching the first sample text generated and output by the audio decoder to be trained at least according to the first sample low-dimensional discretized feature;

[0126] The parameters of the AR module to be trained are adjusted according to the difference between the first predicted speech and the first sample reference speech, and the training is continued until the training is terminated when a preset termination condition is met, thereby obtaining the intermediate model.

[0127] Optionally, the audio decoder is a neural vocoder built and pre-trained based on a neural network, and the speech synthesis unit 804 is specifically used for:

[0128] Acquire a sample phoneme sequence corresponding to a second sample text; and extract sample conditional features of a second sample reference speech and process the second sample reference speech into a discretized second sample encoding representation, wherein the second sample conditional features include a second sample acoustic feature and / or a second sample semantic feature;

[0129] Inputting the second sample phoneme sequence, the second sample conditional feature and the second sample encoding into the neural codec language model, and receiving the second sample low-dimensional discretization feature outputted by the neural codec language model;

[0130] Inputting the second sample low-dimensional discretized features as second sample data into a vocoder to be trained, and receiving a second predicted speech output by the vocoder to be trained that matches the second sample text;

[0131] The parameters of the vocoder to be trained are adjusted according to the difference between the second predicted speech and the second sample reference speech, and the training is continued until the training is terminated when a preset termination condition is met, thereby obtaining the neural vocoder.

[0132] Correspondingly, the present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the speech synthesis method described in any of the above embodiments is implemented.

[0133] Accordingly, the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the speech synthesis method described in any of the above embodiments.

[0134] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0135] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0136] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0137] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

Claims

1. A speech synthesis method, comprising: Get the phoneme sequence corresponding to the target text; and, extracting conditional features of a reference speech and processing the reference speech into a discretized coded representation, the conditional features comprising acoustic features and / or semantic features; The phoneme sequence, the conditional features and the encoded representation are input into a pre-trained speech synthesis model, and a target speech output by the speech synthesis model that matches the target text is obtained; wherein the speech synthesis model includes a neural codec language model and an audio decoder connected thereto, and the neural codec language model includes an autoregressive AR module and does not include a non-autoregressive NAR module.

2. The method according to claim 1, characterized in that The step of obtaining the phoneme sequence corresponding to the target text includes: Searching for the pronunciation of each character in the target text in order, and splitting each pronunciation into individual phonemes, and arranging the split phonemes in order according to the order of the corresponding words in the target text to form a phoneme sequence corresponding to the target text; or The target text is input into a pre-trained grapheme-to-phoneme G2P model, and a phoneme sequence corresponding to the target text output by the G2P model is received.

3. The method according to claim 1, characterized in that The extracting conditional features of the reference speech comprises: The reference speech is input into a pre-trained Hubert extractor, and conditional features of the reference speech output after extraction by the Hubert extractor are received.

4. The method according to claim 3, characterized in that The conditional features output by the Hubert extractor include multiple levels, wherein low-level features include the acoustic features and high-level features include the semantic features.

5. The method according to claim 1, characterized in that The step of processing the reference speech into a discretized coded representation comprises: Segmenting the reference speech into a plurality of speech segments, and calculating the residual between the original signal and the predicted signal of each speech segment; The residuals of each speech segment are vectorized to determine matching code vectors, and each code vector is encoded to generate a discretized coded representation.

6. The method according to claim 1, characterized in that The step of inputting the phoneme sequence, the conditional feature, and the encoded representation into a pre-trained speech synthesis model, and obtaining a target speech output by the speech synthesis model that matches the target text, comprises: The phoneme sequence, the conditional feature and the coded representation are concatenated and inputted into the input end of the AR module, so that the module can send the processed low-dimensional discretized features to the audio decoder; Receive the target speech matching the target text, which is generated and output by the audio decoder according to the low-dimensional discretization features.

7. The method according to claim 1, characterized in that Also includes: For a reference text matching the reference speech, a reference phoneme sequence corresponding to the reference text is obtained, and the reference phoneme sequence is associated with a phoneme sequence corresponding to the target text and input into the speech synthesis model.

8. The method according to claim 1, characterized in that: The neural encoding and decoding language model is trained in the following way: Acquire a first sample phoneme sequence corresponding to the first sample text; and extract a first sample conditional feature of a first sample reference speech and process the first sample reference speech into a discretized first sample encoding representation, wherein the first sample conditional feature includes a first sample acoustic feature and / or a first sample semantic feature; Using the first sample data consisting of the first sample phoneme sequence, the first sample conditional feature and the first sample coding representation to train the codec model to be trained, the codec model to be trained also includes an AR module to be trained and a NAR module to be trained, and the difference between the first sample reference speech and the first predicted speech output by the model to be trained is used as the model loss of the codec model to be trained; For the trained intermediate model including the AR module and the NAR module, the NAR module is removed or frozen to obtain the neural encoding and decoding language model.

9. The method according to claim 8, characterized in that The training of the codec model to be trained using the first sample data consisting of the first sample phoneme sequence, the first sample conditional feature and the first sample encoding representation comprises: The first sample phoneme sequence, the first sample conditional feature and the first sample encoding representation are concatenated and inputted into the input end of the AR module to be trained, so that the module sends the processed first sample low-dimensional discretized feature to the audio decoder to be trained; and, receiving the first predicted speech matching the first sample text generated and output by the audio decoder to be trained at least according to the first sample low-dimensional discretized feature; The parameters of the AR module to be trained are adjusted according to the difference between the first predicted speech and the first sample reference speech, and the training is continued until the training is terminated when a preset termination condition is met, thereby obtaining the intermediate model.

10. The method according to claim 8, characterized in that The audio decoder is a neural vocoder built and pre-trained based on a neural network, and training the neural vocoder includes: Acquire a sample phoneme sequence corresponding to a second sample text; and extract sample conditional features of a second sample reference speech and process the second sample reference speech into a discretized second sample encoding representation, wherein the second sample conditional features include a second sample acoustic feature and / or a second sample semantic feature; Inputting the second sample phoneme sequence, the second sample conditional feature and the second sample encoding into the neural codec language model, and receiving the second sample low-dimensional discretization feature outputted by the neural codec language model; Inputting the second sample low-dimensional discretized features as second sample data into a vocoder to be trained, and receiving a second predicted speech output by the vocoder to be trained that matches the second sample text; The parameters of the vocoder to be trained are adjusted according to the difference between the second predicted speech and the second sample reference speech, and the training is continued until the training is terminated when a preset termination condition is met, thereby obtaining the neural vocoder.

11. A speech synthesis device, characterized in that: include: A feature acquisition unit, configured to acquire a phoneme sequence corresponding to a target text; and extract conditional features of a reference speech and process the reference speech into a discretized coding representation, wherein the conditional features include acoustic features and / or semantic features; A speech synthesis unit is used to input the phoneme sequence, the conditional features and the encoded representation into a pre-trained speech synthesis model, and obtain the target speech output by the speech synthesis model that matches the target text; wherein the speech synthesis model includes a neural codec language model and an audio decoder connected thereto, and the neural codec language model includes an autoregressive AR module and does not include a non-autoregressive NAR module.

12. A TTS system, characterized in that: It includes a speech synthesis model, which includes a neural codec language model and a neural vocoder connected thereto, wherein the neural codec language model includes an autoregressive AR module and does not include a non-autoregressive NAR module; wherein, The neural codec language model is used to receive a phoneme sequence corresponding to a target text, conditional features of a reference speech, and a discretized coded representation obtained by processing the reference speech, wherein the conditional features include acoustic features and / or semantic features; and send a low-dimensional discretized feature obtained by processing the phoneme sequence, the conditional features, and the coded representation to the neural vocoder; The neural vocoder is used to generate and output a target speech matching the target text according to the low-dimensional discretization features.

13. The system according to claim 12, characterized in that The TTS system further includes at least one of the following: A G2P model is used to generate a phoneme sequence corresponding to the target text and input the phoneme sequence into the neural encoding and decoding language model; A Hubert extractor, configured to extract conditional features of the reference speech and input the conditional features into the neural codec language model; A neural encoder is used to compress the reference speech into a discretized coding representation through a residual vector quantization algorithm.

14. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 10 by running the executable instructions.

15. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

Images