A speech editing and synthesis method and system based on autoregressive model

Abstract

The present invention proposes a speech editing and synthesis method and system based on an autoregressive model, which is used for splicing real-person audio and synthesized audio, and belongs to the field of artificial intelligence technology. Discretize the real-person audio signal into audio tokens and text tokens, and train the autoregressive model; use the hidden layer state generated by the autoregressive model to train the vocoder, so that the vocoder can restore the audio according to the hidden layer state; extract the timbre features from the real-person audio to be edited, and convert them into audio tokens and text tokens, and splice the text tokens of the real-person audio with the text tokens of the synthesized audio; splice the timbre features, text tokens, and audio tokens and input them into the autoregressive model to predict the hidden layer state; preprocess the hidden layer state of the audio according to the sequence of the real-person audio and the synthesized audio, and restore the audio. The present invention solves the problem of abrupt splicing of real-person audio and synthesized audio.

Description

Technical Field

[0001] The invention belongs to the field of artificial intelligence technology, and in particular relates to a speech editing and synthesis method and system based on an autoregressive model. Background Art

[0002] Artificial intelligence technology is becoming increasingly mature. In robot customer service scenarios, the step of recording the audio of the conversation by the recording engineer can be replaced by speech synthesis technology, which reduces labor costs. In the existing technology, the audio of speech synthesis is sometimes not as good as the real person recording in terms of emotional expression, tone, rhythm, etc. In this case, the solution of splicing real person recording and speech synthesis is usually adopted.

[0003] The existing real-person recording plus speech synthesis splicing solution has the following disadvantages:

[0004] 1. The tone and rhythm of real-person recordings will vary from time to time, while the tone and rhythm of speech synthesis audio are relatively fixed. When splicing real-person recordings and synthesized audio, there will be obvious differences in timbre and rhythm between the two parts of audio.

[0005] 2. The distance between the real person and the microphone, the recording environment, the reverberation, etc. will vary from time to time, while the audio of speech synthesis is relatively fixed in the sound field. This results in a noticeable abruptness when splicing the real person recording and the synthesized sound.

[0006] How to make the result of real-person recording and speech synthesis more realistic is an issue that is constantly being studied in this field. Summary of the invention

[0007] In view of this, the present invention proposes a speech editing and synthesis method and system based on an autoregressive model, and the technical solution adopted is as follows:

[0008] In a first aspect, the present invention provides a speech editing and synthesis method based on an autoregressive model, which is used to splice real audio and synthesized audio, comprising:

[0009] Discretize the real person audio signal into audio tokens and text tokens, and use the audio tokens and text annotations to train the autoregressive model;

[0010] Use the trained autoregressive model to convert the real person audio into the hidden layer state of the audio token, and use the hidden layer state and the corresponding real person audio to train the vocoder model so that the vocoder model can restore the audio according to the hidden layer state;

[0011] For the audio to be edited, obtain the previous segment of real-person audio of the audio part to be synthesized, and if the previous segment of real-person audio does not exist, obtain the subsequent segment of real-person audio; extract timbre features from the obtained real-person audio, convert the real-person audio into audio tokens and text tokens, and concatenate the text token of the real-person audio with the text token of the audio to be synthesized; concatenate the timbre features, text tokens, and audio tokens and input them into the trained autoregressive model to predict the hidden layer state of the audio to be synthesized;

[0012] The hidden layer state of the audio is preprocessed according to the sequence of the real audio and the synthesized audio, and the preprocessed hidden layer state is used to restore the audio using a vocoder model.

[0013] Furthermore, the encoder in the dVAE model is used to discretize the real audio signal into audio tokens.

[0014] Furthermore, the training process of the dVAE model includes:

[0015] Convert human audio to mel-spectrogram;

[0016] The Mel-spectrogram is input into the dVAE model. The encoder in the dVAE model compresses the Mel-spectrogram to generate audio tokens. The decoder then restores the high-resolution Mel-spectrogram based on the audio tokens.

[0017] The dVAE model is trained by combining the loss between the true mel-spectrogram and the restored mel-spectrogram.

[0018] Furthermore, the text token generation method includes:

[0019] Use the speech recognition system to obtain the text corresponding to the real person audio;

[0020] Tokenize the text to get the text token.

[0021] Furthermore, the input of the autoregressive model is the concatenation result of timbre features, text tokens, and audio tokens, and a start marker token is inserted between the text tokens and the audio tokens; the autoregressive model gradually predicts the audio token corresponding to each text token according to a given context.

[0022] Furthermore, the hidden layer state of the audio token refers to the hidden vector corresponding to each audio token output by the penultimate layer of the autoregressive model.

[0023] Furthermore, the preprocessing of the hidden layer state of the audio according to the sequence of the real audio and the synthesized audio includes:

[0024] If the synthetic audio is spliced ​​after the real audio, the hidden layer state does not need to be preprocessed;

[0025] If the synthesized audio is spliced ​​before the real audio, the hidden layer state of the synthesized audio and the hidden layer state of the real audio are swapped.

[0026] Furthermore, for the case where the synthesized audio is spliced ​​after the real audio, after obtaining the preprocessed hidden layer state, the method further includes:

[0027] Determine whether the synthesized audio contains the real audio to be spliced;

[0028] If included, extract the timbre features from the subsequent real-person audio of the synthesized audio and convert them into audio tokens, and convert the text of the real-person audio into text tokens; concatenate the timbre features, text tokens, and audio tokens and input them into an autoregressive model, generate a hidden layer state of the subsequent real-person audio by the autoregressive model, and concatenate the hidden layer state after the preprocessed hidden layer state;

[0029] If not included, no processing is required.

[0030] Furthermore, a timbre extraction module for extracting timbre features from real-person audio is trained together with the autoregressive model.

[0031] In a second aspect, the present invention provides a speech editing and synthesis system based on an autoregressive model, which is used to splice real audio and synthesized audio to implement the above-mentioned speech editing and synthesis method.

[0032] The beneficial effects of the present invention are:

[0033] The present invention uses an autoregressive model for speech editing and synthesis. Since the autoregressive model is continuously generated according to the existing audio, the synthesized audio part is closer to the real audio part in terms of tone, rhythm, and sound field; the present invention uses the autoregressive model to regenerate the front and back audio at the splicing of the real recording and the synthesized audio, so there will be no abrupt feeling at the connection. The autoregressive model is trained quickly using speech audio containing various timbres, rhythms, and environments, and the model has stronger generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0035] Figure 1 It is an optional form of audio to be edited, where the beginning and the end are real-person recordings, and the middle is synthetic speech;

[0036] Figure 2 This is a diagram of the training of the autoregressive model;

[0037] Figure 3 Schematic diagram of speech editing and synthesis method based on autoregressive model. DETAILED DESCRIPTION

[0038] The present invention is further described and illustrated below in conjunction with specific embodiments. The embodiments are merely exemplary of the present disclosure and do not define the scope of limitation. The technical features of each embodiment of the present invention may be combined accordingly without conflicting with each other.

[0039] The accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0040] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0041] The present invention proposes a speech editing and synthesis method based on an autoregressive model, which mainly comprises the following steps:

[0042] S1, uses real-person audio dataset to train the encoding-decoding model;

[0043] S2: Use the encoder in the encoding-decoding model to discretize the audio signal into audio tokens, use the ASR model to annotate the audio dataset and convert it into text tokens, and use the audio tokens and text annotations to train the autoregressive model.

[0044] The input of the autoregressive model consists of three parts: the first part is the timbre feature, the second part is the text token, and the third part is the audio token. The timbre feature is extracted from the audio data by the timbre extraction module, and the timbre extraction module is trained together with the autoregressive model; the output of the autoregressive model is the next audio token of each audio token, and the autoregressive model can predict the next audio token according to the given context during inference.

[0045] S3, using the trained encoder-decoder model and autoregressive model to convert the real audio data set into the hidden layer state of the audio token, that is, the penultimate layer output of the autoregressive model. Then use the hidden layer state and the corresponding real audio to train the vocoder model so that the vocoder model can restore the audio according to the hidden layer state.

[0046] S4, for the audio to be edited, obtaining the previous segment of live audio of the speech part to be synthesized, and if the previous segment of live audio does not exist, obtaining the subsequent segment of live audio;

[0047] Extracting timbre features from the acquired real-person audio, converting the real-person audio into audio tokens, concatenating the text of the real-person audio and the text of the audio to be synthesized and converting them into text tokens; concatenating the timbre features, text tokens, and audio tokens and inputting them into an autoregressive model to predict the hidden layer state of the audio to be synthesized;

[0048] The hidden layer state of the audio is preprocessed according to the sequence of the real audio and the synthesized audio, and the preprocessed hidden layer state is used to restore the audio using a vocoder model.

[0049] In the above step S1, the encoding-decoding model is preferably a dVAE (Denoising Variational Autoencoder) model, which combines the generative model of the variational autoencoder and the denoising technology, and uses Gumbel-Softmax to provide a continuous approximation for discrete variables to solve the non-differentiable problem caused by the discretization method. The dVAE model is an encoding-decoding structure, the input Mel spectrogram is compressed by the encoder, and the output latent vector of the encoder is taken as the audio token; in the training stage, the audio token is restored to a high-resolution Mel spectrogram by the decoder.

[0050] Since the processing object of the dVAE model is a graph structure, the audio also needs to be processed into a Mel-spectrogram. The Mel-spectrogram is the result of converting the spectral representation of the audio signal to the Mel-spectrogram scale and performing weighted averaging of the spectrum through a set of Mel filters.

[0051] In the above step S2, the present invention uses an autoregressive model to synthesize audio. The autoregressive model structure can adopt a Tacotron2 model or a Transformer TTS model, and the Transformer TTS model is preferred. The Transformer TTS model adopts a Transformer structure in the encoding layer and the decoding layer to replace the BLSTM structure in the Tacotron2 model, and at the same time adopts its own multi-head attention mechanism to replace the position-sensitive attention mechanism.

[0052] like Figure 2 As shown, its input includes three parts, wherein the timbre feature is extracted from the input audio through a timbre extraction module, and the timbre extraction module can be a lightweight machine learning model, for example, it can be composed of 5 layers (or other layers) of one-dimensional convolution, and the timbre extraction module can learn the timbre representation of the speech according to the input audio. The text token is obtained by directly dividing the text corresponding to the audio into tokens, and the corresponding text can be recognized from the audio through the traditional ASR speech recognition system. The audio token is obtained by converting the encoder in the encoding-decoding model obtained by training the input real person audio in step S1. In the autoregressive process, the next audio token can be predicted using the given context. It should be noted that the start mark token is also inserted between the timbre feature, text token, and audio token of the input autoregressive model, which is recorded as start_token. The autoregressive model will gradually synthesize the audio token after the start_token. The start_token in the present invention is inserted between the text token and the audio token. In the training process of the autoregressive model, the cross entropy loss between the synthesized audio token and the real audio token is used to update the autoregressive model until the model converges. Based on the input timbre features, the synthetic audio token of the target timbre output by the model can be obtained, retaining the melody, rhythm and other information of the real audio.

[0053] In the above step S3, the present invention uses the hidden layer state of the penultimate layer of the autoregressive model to train the vocoder so that it can restore the audio according to the hidden layer state. The vocoder is a model that can restore the hidden layer state of the audio to obtain the audio, and often uses a neural network-based structure, such as wavenet, lpcnet and hifigan.

[0054] In the above step S4, for the audio to be edited, for example Figure 1 A three-segment audio is shown, in which the first and last parts are real-person recordings, and the middle part is synthetic speech.

[0055] It is necessary to first analyze whether there is real-person audio before the synthesized speech in the audio to be edited. If so, directly obtain the previous real-person audio for subsequent speech synthesis.

[0056] like Figure 3As shown, timbre features are extracted from the acquired real-person audio, and the real-person audio is converted into audio tokens, which are obtained by converting the encoder in the encoding-decoding model obtained by training the input real-person audio in step S1; the real-person audio is recognized into corresponding text by a traditional ASR speech recognition system, where not only the text token corresponding to the real-person audio needs to be input, but also the text token of the audio to be synthesized needs to be concatenated thereafter, which is also obtained by directly dividing the text of the audio to be synthesized into tokens; the text of the real-person audio and the text of the audio to be synthesized are concatenated and converted into text tokens; the timbre features, text tokens, and audio tokens are concatenated and input into the autoregressive model to predict the hidden layer state of the audio to be synthesized. The role of this autoregressive model is to predict the next audio token based on the previous audio token. For the real audio token prediction part, the previous real audio token is used in combination with contextual information such as timbre features and text tokens to predict the next audio token. This task can be achieved in the training stage, so the hidden layer state of the generated real audio retains the real information well; for the synthetic audio token prediction part, the previous predicted audio token is used to predict the next audio token. In the prediction process, the timbre features and text tokens and other contextual information are combined to generate the hidden layer state of the synthetic audio that fits the melody, rhythm, timbre, etc. of the real audio.

[0057] If there is real-person audio before and after the synthesized speech, then on the basis of the previous steps, the timbre features of the latter segment of real-person audio are extracted and converted into audio tokens, and the text of the real-person audio is converted into text tokens; the timbre features, text tokens, and audio tokens are concatenated and input into the autoregressive model, and the hidden layer state of the latter segment of real-person audio is generated by the autoregressive model, and the hidden layer state is concatenated after the hidden layer state generated in the previous steps.

[0058] If there is no real-person audio before the synthesized speech, but there is after it, then obtain the subsequent real-person audio, extract the timbre features from the obtained real-person audio, and convert the real-person audio into an audio token. The audio token is converted from the encoder in the encoding-decoding model obtained by training the input real-person audio in step S1. The real-person audio is recognized into the corresponding text through a traditional ASR speech recognition system. Here, it is necessary to input not only the text token corresponding to the real-person audio, but also the text token of the audio to be synthesized after it, which is also obtained by directly dividing the text of the audio to be synthesized into tokens. The text of the real-person audio and the text of the audio to be synthesized are spliced ​​and converted into text tokens; the timbre features, text tokens, and audio tokens are spliced ​​together and input into the autoregressive model to let it predict the hidden layer state of the audio to be synthesized; according to the splicing position of the real-person audio and the synthesized audio, the hidden layer states of the two are swapped.

[0059] Through the above analysis, the autoregressive model generates a complete hidden layer state containing real audio and synthetic audio, the order of the hidden layer state is consistent with the order of the audio, and the vocoder model restores the audio according to the hidden layer state to obtain a complete audio. The present invention uses the autoregressive model to regenerate the front and back audio at the splicing of the real audio recording and the synthetic audio, so there is no abrupt feeling at the connection.

[0060] In order to prove the effect of the present invention, in this embodiment, the effect is verified on three data sets: Dataset 1: WenetSpeech, Dataset 2: MagicData, and Dataset 3: CN-Celeb. The present invention is compared with the traditional speech synthesis splicing method in terms of the two objective indicators of timbre and speech speed and the subjective indicator of MOS, and the results are shown in Tables 1-3.

[0061] Table 1 Comparison results under timbre index

[0062] Average distance of timbre Dataset 1 Dataset 2 Dataset 3 Voice splicing method 0.34 0.29 0.36 Method of the present invention 0.17 0.11 0.21

[0063] Table 2 Comparison results under speech speed index

[0064] Average difference in speech rate Dataset 1 Dataset 2 Dataset 3 Voice splicing method 0.91 1.54 1.93 Method of the present invention 0.53 0.92 1.15

[0065] Table 3 Comparison results under MOS index

[0066] Average MOS value Dataset 1 Dataset 2 Dataset 3 Voice splicing method 3.64 3.96 3.82 Method of the present invention 4.15 4.31 4.33

[0067] The timbre index is obtained by extracting speaker features of real audio and synthesized audio using an open source speaker recognition model and calculating the cosine distance of the two features. The smaller the value, the closer the timbre. As can be seen from Table 1, the average timbre distance of the present invention is significantly lower than that of the speech splicing method.

[0068] The speaking speed index is obtained by calculating the average speaking speed of the real person audio and the synthesized audio, and calculating the difference between the two. The smaller the difference, the closer the speaking speeds are. As can be seen from Table 2, the average speaking speed difference of the present invention is significantly lower than that of the speech splicing method.

[0069] MOS is a subjective scoring method that evaluates the naturalness and clarity of speech through the scoring of human listeners. It is usually a scale of 1 to 5, with 1 representing the worst and 5 representing the best. This index is the most commonly used evaluation standard in speech synthesis systems. As can be seen from Table 3, the MOS scores of the present invention are all above 4 points, which is higher than the speech splicing method.

[0070] Based on the same inventive concept, in this embodiment, a speech editing and synthesis system based on an autoregressive model is also provided, which is used to splice real audio and synthesized audio, including:

[0071] Audio discretization module, which is used to discretize real-person audio signals into audio tokens and text tokens;

[0072] An autoregressive model training module, which is used to train an autoregressive model;

[0073] A vocoder model training module, which is used to convert real-person audio into hidden layer states of audio tokens using the trained autoregressive model, and train the vocoder model using the hidden layer states and the corresponding real-person audio, so that the vocoder model can restore the audio according to the hidden layer states;

[0074] The speech editing and synthesis module is used to obtain the audio to be edited, extract the timbre features from the previous segment of real-person audio of the audio to be synthesized, and if the previous segment of real-person audio does not exist, extract the timbre features from the next segment of real-person audio, and concatenate the text token of the real-person audio with the text token of the audio to be synthesized; concatenate the timbre features, text token, and audio token and input them into the trained autoregressive model to predict the hidden layer state of the audio to be synthesized; pre-process the hidden layer state of the audio according to the sequence of the real-person audio and the synthesized audio, and restore the audio with the pre-processed hidden layer state using a vocoder model.

[0075] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0076] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0077] The above-mentioned embodiments only express several implementation modes of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. For those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A speech editing and synthesis method based on an autoregressive model, for splicing real audio and synthetic audio, characterized in that: include: Discretize the real person audio signal into audio tokens and text tokens, and use the audio tokens and text annotations to train the autoregressive model; Use the trained autoregressive model to convert the real person audio into the hidden layer state of the audio token, and use the hidden layer state and the corresponding real person audio to train the vocoder model so that the vocoder model can restore the audio according to the hidden layer state; For the audio to be edited, obtain the previous segment of real-person audio of the audio part to be synthesized, and if the previous segment of real-person audio does not exist, obtain the subsequent segment of real-person audio; extract timbre features from the obtained real-person audio, convert the real-person audio into audio tokens and text tokens, and concatenate the text token of the real-person audio with the text token of the audio to be synthesized; concatenate the timbre features, text tokens, and audio tokens and input them into the trained autoregressive model to predict the hidden layer state of the audio; The hidden layer state of the audio is preprocessed according to the sequence of the real audio and the synthesized audio, and the preprocessed hidden layer state is used to restore the audio using a vocoder model.

2. The method for speech editing and synthesis based on an autoregressive model according to claim 1, characterized in that: The encoder in the dVAE model is used to discretize the real audio signal into audio tokens.

3. The speech editing and synthesis method based on the autoregressive model according to claim 2, characterized in that: The training process of the dVAE model includes: Convert human audio to mel-spectrogram; The Mel-spectrogram is input into the dVAE model. The encoder in the dVAE model compresses the Mel-spectrogram to generate audio tokens. The decoder then restores the high-resolution Mel-spectrogram based on the audio tokens. The dVAE model is trained by combining the loss between the true mel-spectrogram and the restored mel-spectrogram.

4. The method for speech editing and synthesis based on an autoregressive model according to claim 1, characterized in that: The method for generating a text token includes: Use the speech recognition system to obtain the text corresponding to the real person audio; Tokenize the text to get the text token.

5. The method for speech editing and synthesis based on an autoregressive model according to claim 1, characterized in that: The input of the autoregressive model is the concatenation result of timbre features, text tokens, and audio tokens, and a start marker token is inserted between the text tokens and the audio tokens; the autoregressive model gradually predicts the audio token corresponding to each text token according to a given context.

6. The method for speech editing and synthesis based on an autoregressive model according to claim 5, characterized in that: The hidden layer state of the audio token refers to the hidden vector corresponding to each audio token output by the penultimate layer of the autoregressive model.

7. The method for speech editing and synthesis based on an autoregressive model according to claim 1, characterized in that: The hidden layer state of the preprocessing audio according to the sequence of real audio and synthesized audio includes: If the synthetic audio is spliced ​​after the real audio, the hidden layer state does not need to be preprocessed; If the synthesized audio is spliced ​​before the real audio, the hidden layer state of the synthesized audio and the hidden layer state of the real audio are swapped.

8. The method for speech editing and synthesis based on an autoregressive model according to claim 7, characterized in that: For the case where the synthetic audio is spliced ​​after the real audio, after obtaining the preprocessed hidden layer state, it also includes: Determine whether the synthesized audio contains the real audio to be spliced; If included, extract the timbre features from the subsequent real-person audio of the synthesized audio and convert them into audio tokens, and convert the text of the real-person audio into text tokens; concatenate the timbre features, text tokens, and audio tokens and input them into an autoregressive model, generate a hidden layer state of the subsequent real-person audio by the autoregressive model, and concatenate the hidden layer state after the preprocessed hidden layer state; If not included, no processing is required.

9. The method for speech editing and synthesis based on an autoregressive model according to claim 1, characterized in that: The timbre extraction module, which is used to extract timbre features from real-person audio, is trained together with the autoregressive model.

10. A speech editing and synthesis system based on an autoregressive model, used for splicing real audio and synthetic audio, characterized in that: include: Audio discretization module, which is used to discretize real-person audio signals into audio tokens and text tokens; An autoregressive model training module, which is used to train an autoregressive model; A vocoder model training module, which is used to convert real-person audio into hidden layer states of audio tokens using the trained autoregressive model, and train the vocoder model using the hidden layer states and the corresponding real-person audio, so that the vocoder model can restore the audio according to the hidden layer states; The speech editing and synthesis module is used to obtain the audio to be edited, extract the timbre features from the previous segment of real-person audio of the audio to be synthesized, and if the previous segment of real-person audio does not exist, extract the timbre features from the next segment of real-person audio, and concatenate the text token of the real-person audio with the text token of the audio to be synthesized; concatenate the timbre features, text token, and audio token and input them into the trained autoregressive model to predict the hidden layer state of the audio to be synthesized; pre-process the hidden layer state of the audio according to the sequence of the real-person audio and the synthesized audio, and restore the audio with the pre-processed hidden layer state using a vocoder model.

Images