A high expressiveness audio generation method based on natural language description text

Abstract

A high expressiveness audio generation method based on natural language description text, comprising: S1: using a pre-trained multilingual text encoder to encode the input natural language description instruction, and extracting semantic information; S2: taking the text encoding representation as a condition, sequentially predicting the semantic unit sequence of the audio through an autoregressive model (AR), which represents the content event information appearing in the audio; S3: taking the text encoding representation and the audio semantic unit sequence output by S2 as a prefix condition, predicting the coarse-grained acoustic unit sequence of the audio through AR, which determines the overall length and basic acoustic information of the audio; S4: using a non-autoregressive model (NAR) to predict the fine-grained acoustic unit sequence of the audio, modeling the audio detail performance and high-frequency information; S5: inputting the multi-layer acoustic unit sequence into a vocoder to generate high-quality audio waveform. The method directly supports generating audio from a natural language description instruction, and improves the naturalness, expressiveness and controllability of the generated audio.

Description

TECHNICAL FIELD

[0001] The present application relates to artificial intelligence audio generation technology, in particular to a high expressiveness audio generation method based on natural language description text. BACKGROUND

[0002] Recent artificial intelligence generated content (AIGC) systems have the ability to generate digital multimedia content such as text, images and videos based on human language instructions. However, existing TTS solutions can only control the pronunciation content and style of audio through content prompts and description prompts, respectively, such as a content prompt of "Today the weather is really good" and a description prompt of "young woman, happy and cheerful, with a sense of satisfaction"; they cannot directly generate understandable and high expressiveness audio based on a natural language description, such as an instruction of "In a quiet park, the rustling of leaves, a young mother happily and cheerfully chats, "Today the weather is really good", with a slightly heavier tone on "really" and a sense of satisfaction". Therefore, the current "text-to-audio" still has a gap with other AIGC generation paradigms. In addition, the description prompt and the content prompt of the existing solution are independent of each other, and the pronunciation content text is often ignored when modeling the style of the audio, which limits the control ability of the fine-grained level of the audio (such as emphasizing the re-reading of a certain word in the description in the above example).

[0003] PromptTTS uses separate content prompts and description prompts for audio synthesis, the content prompt is the phoneme sequence of the text, and the global style control (including gender, speech rate, tone, etc.) can be achieved through natural language description. The acoustic model is a FastSpeech 2 framework, which extracts the global style representation of the audio through a style encoder, and uses a pre-trained BERT model to extract the text encoding information of the description prompt to predict the style representation.

[0004] Salle uses separate content prompts and description prompts for audio synthesis, the content input is the phoneme sequence of the text, and the style control (including gender, speech rate, emotion, etc.) can be achieved through natural language description. The acoustic model uses the GPT architecture to model the discrete acoustic units of the audio, and directly concatenates the description text sequence before the text phoneme sequence for acoustic part prediction.

[0005] Defects: The existing scheme needs to separate the explicit content prompt and the description prompt, and the description prompt can only control the overall timbre style of the audio, and cannot directly generate audio from a mixed form, flexible and diverse natural language description instruction and achieve fine control; The existing scheme has single capability, which can only achieve text description control of audio timbre style or reference audio control of audio timbre style, and cannot use a unified system to realize multiple task functions; The existing text description based speech synthesis scheme can only use description annotation data for training, and the system has limited generalization, and cannot utilize large-scale unannotated data.

[0006] In general, in the field of audio, the existing methods related to human instruction to audio generation have two limitations. First, they need to separate the input into content prompts (transcription) and description prompts (style and speaker), rather than directly supporting human instructions. This division is not natural in form and is not coordinated with other AIGC models. Second, using independent description prompts to simulate audio style without considering the transcription content limits the ability to control audio at a fine-grained level.

[0007] In addition, the existing "text-to-audio" (TTA) scheme can only generate simple audio events, such as the AudioGen and MusicLM schemes supporting input such as "leaves rustling" "car honking" "playing a sad violin piece" and synthesizing the corresponding audio segments. Since the text prompt of its input does not support given speech content, it cannot achieve high-quality understandable speech synthesis and joint and harmonious generation of audio (specifically sound effects, music, etc.) and speech. This limits the further development and application of text-to-audio and video dubbing scenarios.

[0008] It should be noted that the information disclosed in the above background section is only for understanding the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0009] The main purpose of the present application is to solve the problems existing in the background art, and to provide a high expressiveness audio generation method based on natural language description text.

[0010] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0011] In a first aspect, the present application provides a high expressiveness audio generation method based on natural language description text, comprising the following steps:

[0012] S1: using a pre-trained multi-language text encoder to encode the input natural language description instruction, and extracting all semantic information in the description text;

[0013] S2: sequentially predict a sequence of semantic units of the audio by an autoregressive model (AR) with the text encoding representation as a condition, the sequence of semantic units representing content event information occurring in the audio;

[0014] S3: predict a sequence of coarse-grained acoustic units of the audio by an autoregressive model (AR) with the text encoding representation and the sequence of semantic units of S2 as prefix conditions, determine the overall duration and basic acoustic information of the audio;

[0015] S4: predict a sequence of fine-grained acoustic units of the audio by a non-autoregressive model (NAR) in a multi-layer iterative manner, model audio detail performance and high-frequency information;

[0016] S5: input the multi-layer acoustic unit sequence into a vocoder to generate an audio waveform.

[0017] In a second aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the high-expressiveness audio generation method based on natural language description text.

[0018] The present application has the following beneficial effects:

[0019] Compared with the prior art, the present application first proposes to directly support the generation of audio from a natural language description instruction, aligns the paradigm of AIGC audio generation and other modal generation, and is applicable to various audio generation tasks, and has great improvement in the naturalness and expressiveness of generated audio.

[0020] The present application proposes a text-to-audio system (VoxInstruct for short) directly supporting completely natural language description instructions, and the embodiments of the present application have the following characteristics and advantages:

[0021] 1. The entire system includes two models of autoregression and non-autoregression. The discrete features of the audio are taken as basic modeling units, and next token prediction is taken as a training target. Thanks to the In-Context Learning ability of the language model, the present application has better performance in prosody and naturalness than the traditional TTS model in the past;

[0022] 2. In order to enable the model to automatically understand the content information in the description instruction, a pre-trained text encoder is used to obtain the text encoding information of the input instruction, and an audio semantic unit (semantic token) representation is introduced to guide the model to learn how to distinguish and extract the actual pronunciation content, without the need for an additional determined content text prompt or phoneme sequence, etc.

[0023] 3. In order to improve the compliance of synthesized audio to the description instruction, various unconditional guidance strategies CFG are designed, and by setting different CFG parameters during reasoning, the style control and pronunciation content intelligibility of synthesized audio are respectively strengthened;

[0024] 4. In order to improve the generalization of the VoxInstruct model system, a "pre-training + fine-tuning" training paradigm is adopted, which can pre-train the model on large-scale audio data (without description text annotation), further improve the naturalness and pronunciation stability of synthesized audio;

[0025] 5. By introducing language markers for multilingual modeling, the system can support Chinese and English bilingual description instruction text input and generate Chinese, English and mixed language audio results.

[0026] Compared with the prior art, the technical scheme of the present application can accept natural language description instruction text input without any additional preprocessing work, and establishes an end-to-end description text to audio generation system. The present application can be pre-trained with a large amount of non-description annotated data to promote the model to learn the association between acoustic units and semantic units of audio, thereby improving the performance of the model. The present application can use the same model to support multiple language inputs and support multiple audio generation task scenarios.

[0027] In addition, the present application integrates traditional TTS tasks, description prompt-based TTS tasks and text prompt-based TTA tasks in a general framework, proposes an audio generation unified paradigm starting from natural language description text instructions, and further supports generating high expressiveness speech or singing results with music sound effects and other environmental sounds.

[0028] The present application has important value for human-computer interaction industry and can be used in audio synthesis, voice interaction, voice assistant, audio novel generation, film and television script dubbing, music, singing, scene sound effect and other application scenarios, so that users can create personalized audio results through instruction description.

[0029] Other beneficial effects of the embodiments of the present application will be further described below. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The schematic diagram of the text-to-audio system VoxInstruct directly supported by the embodiments of the present application.

[0031] Figure 2 The algorithm block diagram of the high expressiveness audio generation method based on natural language description text of the embodiments of the present application. DETAILED DESCRIPTION

[0032] The embodiments of the present application will be described in detail below. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present application and its applications.

[0033] The audio generation method of the present application can generate corresponding high expressive audio results based on a sentence of natural language description text. Unlike traditional TTS systems that only synthesize all pronunciation content of input text word by word, the present application aims to further understand the semantic connotation and actual demand of the user input text instruction, automatically analyze and extract descriptive information about audio style and pronunciation information about audio content, so as to generate natural audio that meets the description of human instruction, and provide an AIGC audio generation scheme similar to the text-to-image and text-to-video paradigm. Referring to Figure 1 and Figure 2 , the system (hereinafter referred to as VoxInstruct) proposed in the present application first encodes the input natural language description instruction using a pre-trained multilingual text encoder, extracting all semantic information in the description text; then, taking the text encoding representation as a condition, the semantic unit sequence of the audio (representing the content event information appearing in the audio) and the coarse-grained acoustic unit sequence (determining the overall duration and basic acoustic characteristics of the audio) are predicted in turn through an autoregressive model; then, the non-autoregressive model is used to predict the fine-grained acoustic unit sequence of the audio (modeling audio detail performance and high-frequency information) in a multi-layer iterative manner; finally, the vocoder reconstructs the high-quality audio waveform according to the multi-layer acoustic unit sequence. In addition, during the prediction process, the embodiments of the present application propose various non-classifier guided strategies to improve the compliance of the synthesized audio to the instruction, in order to strengthen the control ability of the description text to the audio generation process. The system can support text instruction input forms in multiple languages such as Chinese and English, as well as mixed languages, and is applicable to various audio generation tasks, including text description to audio generation, single language / cross language timbre cloning based on reference audio, timbre style modification based on reference audio and text description, etc. The specific system capabilities are shown in the following figure, and these capabilities are realized by setting the form of the input instruction (whether the instruction contains a description of the audio timbre style, or only contains pronunciation content, whether the pronunciation content part is Chinese or English), whether to give the model an additional reference audio, etc.

[0034] VoxInstruct can adopt LlaMA architecture, including autoregressive model and non-autoregressive model, audio discrete unit encoder and vocoder, and Multinugal T5 text encoder. 1) By introducing audio semantic units as intermediate representation, the model can more accurately grasp the pronunciation content of the instruction and effectively convert it into target audio; 2) Through the pre-training + fine-tuning paradigm, large-scale unlabeled audio data can be used to improve the overall generalization of the system, and small-scale labeled data can be used to give the model fine instruction control ability; 3) In the training stage, the conditional guided strategy CFG form is realized by masking the instruction text encoding representation, semantic unit sequence and other conditions with a certain probability, and in the inference generation process, by setting the CFG coefficient, the adjustment of different requirements such as audio style description correlation, pronunciation intelligibility and reference timbre similarity is met, so as to realize more fine audio expression control.

[0035] The system flow is as follows: first, a pre-trained multilingual text encoder is used to encode the input natural language description instruction, and all semantic information in the description text is extracted; then, the text encoding representation is taken as a condition, and the semantic unit sequence of the audio (representing the content event information appearing in the audio) and the coarse-grained acoustic unit sequence (determining the overall length and basic acoustic characteristics of the audio) are predicted in turn by the autoregressive model (AR); then, the non-autoregressive model (NAR) is used to predict the fine-grained acoustic unit sequence of the audio in a multi-layer iterative manner (modeling audio detail performance and high-frequency information); finally, the vocoder generates high-quality audio waveform according to the multi-layer acoustic unit sequence.

[0036] Referring to Figure 1 and Figure 2 , the high expressiveness audio generation method based on natural language description text of the embodiment of the application comprises the following steps:

[0037] S1: using a pre-trained multilingual text encoder to encode the input natural language description instruction, and extracting all semantic information in the description text;

[0038] S2: taking the text encoding representation as a condition, predicting the semantic unit sequence of the audio by the autoregressive model (AR), wherein the semantic unit sequence represents the content event information appearing in the audio, such as the basic pronunciation unit of the voice, etc.;

[0039] S3: taking the text encoding representation and the audio semantic unit sequence output by S2 as prefix conditions, predicting the coarse-grained acoustic unit sequence of the audio by the autoregressive model (AR), which determines the overall length and basic acoustic information of the audio;

[0040] S4: predicting the fine-grained acoustic unit sequence of the audio by the non-autoregressive model (NAR) in a multi-layer iterative manner, modeling the audio detail performance and high-frequency information;

[0041] S5: inputting the multi-layer acoustic unit sequence into a vocoder to generate an audio waveform.

[0042] The high expressiveness audio generation method of the present application realizes efficient conversion of natural language description text to high-quality audio through a series of innovative steps. First, a pre-trained multilingual text encoder is used to deeply encode the input natural language description, accurately extracting semantic information in the text. Then, under the condition of the encoding representation, an autoregressive model (AR) predicts a semantic unit sequence of the audio, which directly maps to the actual pronunciation content. Further, based on the output of the AR model, a coarse-grained acoustic unit sequence is predicted, which determines the overall duration and basic acoustic features of the audio, laying the foundation for generating natural and fluent audio. In order to enrich the details and high-frequency information of the audio, a non-autoregressive model (NAR) is used to predict a fine-grained acoustic unit sequence in a multi-layer iterative manner, thereby enhancing the expressiveness and realism of the audio. Finally, the multi-layer acoustic unit sequence is converted into a high-quality audio waveform by a vocoder, completing the whole process from text to audio. The whole process not only improves the naturalness and expressiveness of audio synthesis, but also greatly widens the application range and flexibility of audio generation technology by supporting multilingual and mixed language inputs. In addition, this method realizes end-to-end audio generation without complex preprocessing and postprocessing steps, simplifying the operation process and improving efficiency.

[0043] The audio generation method of the present application is applicable to general audio scenarios, including the generation of speech, music, singing, sound effects, etc., as well as their combined forms. For different scenarios, corresponding types of training data can be introduced. For example, the description instructions input to the model can be: example 1. At the seaside, the sea breeze blows, a five-year-old boy happily shouts "Wow, what a big lake!" Then the sound of seagulls is heard. Example 2. In the sound of a piano and violin duet, she gradually becomes intoxicated, humming softly, "A river wide and flowing …")

[0044] In some embodiments, in step S1, a multilingual T5 encoder supporting multiple languages is used as a text encoding modeling module to encode the input natural language description instructions and extract all semantic information in the description text; the text encoding representation can process input texts in multiple languages, including but not limited to Mandarin and English, as well as mixed language text instructions; using the text encoding representation, an audio sample corresponding to the text description instruction is generated to ensure that the generated audio is consistent with the input text in terms of semantics and acoustic features.

[0045] In the preferred embodiment, the pre-trained multilingual text encoder is used to encode the input natural language description instruction in step S1 to extract all semantic information in the description text; a multilingual T5 base model (MT5-base) is selected as the text encoder, and a pre-trained text encoder thereof is used with a trainable low-rank adaptation (LoRA) adapter inserted to support multilingual instruction input; the original text of the human instruction is passed to the MT5 encoder to derive a text embedding sequence E with a number of subwords T after text tokenization E .

[0046] In some embodiments, in step S2, high-level scene semantic features (more specifically, including how to extract and represent content event information of audio in the data processing flow) such as "cross-talk stage" and "dialogue scene" are introduced in the data processing flow, and it is ensured that these features are reflected in the generated audio; when predicting the semantic unit sequence through the autoregressive model (AR), the high-level scene information in the description text is considered to generate style intonation, background environment sound, music melody, sound effect event, etc.; the user is allowed to control the local performance of the generated audio, such as emphasis, through instructions in the natural language description, such as "emphasize a certain word in the tone";

[0047] In the preferred embodiment, the self-supervised representation model HuBERT or the audio self-supervised representation model AudioMAE is used in step S2 to extract semantic unit features (i.e., event content appearing in the audio) of the audio as the prediction target of the AR model in the S2 stage during training; the embeddings extracted by HuBERT are discretized through a k-means clustering algorithm to generate discrete semantic tags (ST); consecutive repeated tags in the generated semantic tag sequence are removed to form a clear and non-redundant semantic unit sequence to indicate the language information of the generated audio; a language tag (such as an audio part, otherwise a special empty tag) is added in front of the semantic tag sequence to represent the language information of the audio content.

[0048] Next, an autoregressive model (AR) can be used to predict the semantic unit sequence of the audio as a condition with the text encoding representation and the semantic tag sequence, which directly maps to the event content and order information appearing in the to-be-synthesized audio, and can include basic pronunciation units of the audio part, basic audio event units of the sound effect part, and musical instrument note units of the music part, etc., which still form a semantic unit sequence.

[0049] In some embodiments, in the timbre cloning or cross-language timbre cloning scenario in step S3, the user provides the text to be synthesized as an instruction; during inference, the semantic and acoustic units of the reference audio provided by the user are extracted and set as the sequence prefix in the self-recursive model inference process; using the context learning ability of the self-recursive model, a conditional continuation task is performed to achieve accurate imitation of the reference audio timbre style;

[0050] In some embodiments, in the sound style modification scenario in step S4, the user provides the style description to be converted and the text to be synthesized as an instruction; during inference, a conditional continuation task is performed while adjusting the classifier-free guidance (CFG) parameters; by adjusting the CFG parameters, it is determined whether the generated audio is more dependent on instruction control or reference audio control, allowing the user to generate audio that is more consistent with the instruction description or closer to the characteristics of the reference audio according to the user's needs.

[0051] During the training process of the self-recursive model, an unconditional guidance strategy such as masked text encoding or semantic tagging is used with a certain probability to enhance the model's understanding of the instruction text and the accuracy of the generated audio; during the inference process, the parameters of the classifier-free guidance (CFG) are adjusted to optimize the semantic relevance, pronunciation intelligibility, and reference timbre similarity of the generated audio.

[0052] In some embodiments, in the pre-training phase, a large-scale unlabeled audio data is used to train the pre-trained multi-language text encoder to extract and understand the semantic information in the natural language description instructions; in the pre-training phase, a large-scale unlabeled audio data is used to train the self-recursive model (AR) and the non-self-recursive model (NAR) to generate coarse-grained and fine-grained acoustic unit sequences. In the fine-tuning phase, a small-scale description labeled data is used to fine-tune the pre-trained text encoder to enhance the model's understanding of descriptive text; in the fine-tuning phase, a small-scale description labeled data is used to fine-tune the self-recursive model (AR) to improve the accuracy and expressiveness of the generated semantic unit sequence; in the fine-tuning phase, a small-scale description labeled data is used to fine-tune the non-self-recursive model (NAR) to optimize the generated fine-grained acoustic unit sequence, enhance the details and high-frequency information of the audio; in the fine-tuning phase, the parameters of the classifier-free guidance (CFG) are adjusted to meet different requirements for audio style description relevance, pronunciation intelligibility, and reference timbre similarity; in the fine-tuning phase, the parameters of the vocoder are optimized to generate higher quality audio waveforms, ensuring that the generated audio is optimal in naturalness and clarity.

[0053] The text encoder, AR model, and NAR model are jointly trained, including two training processes, 1) model pre-training on large-scale audio data without descriptive annotations (only audio transcription text automatically annotated by ASR or basic category labels of audio can be provided), and 2) model fine-tuning on high-quality audio data with descriptive annotations (various descriptive instruction examples are listed above).

[0054] Through the above pre-training and fine-tuning processes, high-expression audio generation for natural language description text is achieved, and the generated audio can accurately reflect the semantic and style characteristics of the input text.

[0055] In the preferred embodiment, in the training phase of the autoregressive model (AR), the text encoding representation or semantic unit sequence is masked by a certain probability to realize the training in the form of the unconditionally guided strategy CFG to enhance the understanding of natural language description and the accuracy of audio generation by the model. In the training phase of the autoregressive model (AR), based on the partially masked text encoding representation, a coarse-grained acoustic unit sequence is predicted to generate the overall duration and basic acoustic features of the audio; in the training phase of the non-autoregressive model (NAR), the output of the autoregressive model is used as a condition to predict a fine-grained acoustic unit sequence through multi-layer iteration, while implementing the unconditionally guided strategy to enhance the detail performance and high-frequency information of the audio.

[0056] In the preferred embodiment, the autoregressive model (AR) is used to predict coarse-grained acoustic markers, which contain the basic acoustic content structure of the audio, the main prosody and basic timbre of the singing part of the audio, and determine the overall audio duration. In the training phase of the non-autoregressive model (NAR), the output of the autoregressive model is used as a condition to predict a fine-grained acoustic marker through multi-layer iteration, while implementing the unconditionally guided strategy to enhance the detail performance and high-frequency information of the audio.

[0057] In some embodiments, in the inference generation process, the CFG coefficient is set to adjust the semantic relevance, pronunciation intelligibility, and reference timbre similarity of the generated audio, to achieve fine control of the audio style; in the inference generation process, the CFG coefficient is dynamically adjusted according to the input natural language description instruction to meet the demand for audio expressiveness in different scenarios, to generate audio with high naturalness and expressiveness.

[0058] In some embodiments, in the inference process, the instruction text is embedded and the audio prompt (if provided) is used as input using a language model to produce target acoustic markers; after generating the acoustic markers, an acoustic marker decoder such as Vocos is used instead of the original Encodec decoder to synthesize audio waveforms to provide better audio reconstruction quality.

[0059] According to a preferred embodiment of the present application, in the training stage of the autoregressive model (AR), a classifier-free guidance strategy (CFG) is adopted to mask the text embedding representation or the semantic unit sequence by a certain probability, realizing the training of the unconditional guidance strategy to enhance the understanding of the natural language description and the accuracy of the generated audio. When predicting the semantic label, the text embedding sequence is masked, so that the model predicts in the form of unconditional generation, thereby improving the compliance with the natural language instruction; when predicting the coarse-grained acoustic label, the text embedding sequence or the semantic label sequence is masked, and the form of unconditional generation is also adopted, further enhancing the understanding of the model to the instruction and the naturalness of the generated audio. In the training stage of the autoregressive model (AR), based on the partially masked text embedding representation, the coarse-grained acoustic unit sequence is predicted to generate the overall duration and basic acoustic features of the audio. In the training stage of the non-autoregressive model (NAR), the output of the autoregressive model is used as a condition to predict the fine-grained acoustic unit sequence through multi-layer iteration, and the unconditional guidance strategy is implemented to enhance the detail performance and high-frequency information of the audio. In the inference generation process, by setting the CFG coefficient, the semantic relevance, pronunciation intelligibility and reference timbre similarity of the generated audio are adjusted to realize fine control of the audio style. In the logarithmic space, the semantic label is sampled, and the unconditional guidance strategy is combined to optimize the semantic relevance and naturalness of the generated audio. When sampling the coarse-grained acoustic label, two types of CFG are used simultaneously, allowing the generated acoustic label to focus on different aspects, such as the control of audio features and the intelligibility of audio content. In the inference process, the CFG coefficient is dynamically adjusted to meet the demand for audio expressiveness in different scenarios, generating audio with high naturalness and expressiveness.

[0060] According to a preferred embodiment of the present application, in the pre-training stage, a large public audio dataset containing only text transcription is used to train the pre-trained multilingual text encoder to extract and understand the semantic information in the natural language description instructions; the original transcription text is used as human instructions, enclosed in quotation marks, to ensure that the encoding and decoding language model has strong text-to-audio synthesis (intelligibility) and zero-shot sound cloning (generalization) capabilities; in the fine-tuning stage, the pre-trained text encoder is fine-tuned using instruction-audio paired data to give the model the ability to understand the descriptive information in human instructions; in the fine-tuning stage, a progressive fine-tuning strategy is adopted, first using small-scale description annotated data to fine-tune the autoregressive model (AR) to improve the accuracy and expressiveness of the generated semantic unit sequence; further fine-tuning the model using a small dataset of fine-grained instructions, so that the model has the ability to control the audio features in detail, including but not limited to fine-grained attributes such as accent marks; in the fine-tuning stage, the non-autoregressive model (NAR) is fine-tuned using small-scale description annotated data to optimize the generated fine-grained acoustic unit sequence and enhance the detail performance and high-frequency information of the audio; in the fine-tuning stage, the parameters of the classifier-free guidance strategy (CFG) are adjusted to meet the adjustment of different requirements such as audio style description correlation, pronunciation intelligibility, and reference timbre similarity; in the fine-tuning stage, the parameters of the vocoder are optimized to generate higher quality audio waveforms, ensuring that the generated audio reaches the optimal in naturalness and clarity.

[0061] It should be noted that the various audio generation tasks supported by the present application do not require separate training and parameter switching, but only need to define different input formats in the inference process.

[0062] 1. Multilingual description instruction text support: The text encoding modeling module of the present scheme uses a multilingual T5 encoder that supports multiple languages, so it can directly support input text in multiple languages; The semantic and acoustic representation units of the audio used by the present model also support the modeling of multiple language audios, and can generate Mandarin, English, and Mandarin-English switching audio samples, which has been verified in experiments.

[0063] 2. Understanding and fine-grained control of high-level scene information in description text: Unlike previous schemes that only control attributes such as speaker timbre, speech rate, and emotion, the present scheme further introduces higher-level scene semantic features such as "cross-talk stage" and "dialogue scene" in the data processing process, and reflects this in the style and tone of the generated audio and background environmental sound; In addition, it can also control the local emphasis phenomenon of the generated audio by describing "emphasizing a certain word in the tone", which is made possible by the joint representation and modeling of content cues and description cues in the present scheme.

[0064] 3. Voice cloning / cross-lingual voice cloning: The instruction provided by the user in this scenario only needs to be the text to be synthesized. When reasoning, a reference audio provided by the user is obtained, the semantic and acoustic units of the reference audio are first extracted, and they are set as the sequence prefix in the reasoning process of the autoregressive model, so as to perform a kind of conditional continuation task, and the powerful context learning ability of the LM can be used to realize the accurate imitation of the voice style of the reference audio.

[0065] 4. Voice style modification: The instruction provided by the user in this scenario is the style to be converted and the text to be synthesized. The reasoning process is consistent with the last one, and a conditional continuation task is also performed, and the CFG strategy will be adjusted according to the style to be converted. The whole CFG parameter decides whether the effect of generating audio is more dependent on instruction control or reference audio control, The voice quality of PromptTTS is relatively low, which may be attributed to its Transformer-decoder This can allow the user to decide whether the generated audio is more in line with the description of the instruction or closer to the characteristics of the reference audio.

[0066] The core of the present application is to use a language model to build a model to describe the text-to-audio generation task, introduce a semantic unit sequence to ensure that the model has a clear instruction- pronunciation content understanding ability, and design multiple CFG strategies to enable the model to synthesize audio results that are more in line with the requirements of the instruction. Therefore, the pre-trained text encoder MT5 can be replaced by other pre-trained language models such as BERT, BART, GPT, etc., the HuBERT that extracts the semantic units of the audio can be replaced by other acoustic semantic models such as W2V-BERT, WavLM, etc., the Encodec that extracts the acoustic units of the audio can be replaced by SoundStream, HiFi-Code, cDAC, etc., and the Vocos that synthesizes the audio according to the acoustic units can also be replaced by other vocoders.

[0067] Experiment

[0068] According to the pre-training and fine-tuning paradigm of scene intent, a large amount of data with different annotation granularity is integrated, including only transcription data, instruction data and fine-grained instruction data. Large-scale publicly available speech datasets, including WenetSpeech Chinese corpus and GigaSpeech English corpus, are first involved in the pre-training phase. After screening out samples shorter than 3 seconds and low-quality samples, a total of 13.4K hours of speech are obtained. Subsequently, using the annotation system, a series of instruction-speech pair datasets are used to comprehensively and deeply explain the expressiveness of speech through diverse natural language instructions. Speech instructions are characterized by spoken content, acoustic properties, speaker identity, emotional tone and scene background, with a fine-grained description of the subset of word emphasis. Details of the annotation system, including expert classifiers and caption models, are followed by LLMs for instruction rewriting. Experimental examples automatically annotate instructions on some open-source datasets. In addition, to enhance the generalization ability, a considerable amount of scene-rich wild audio data is collected from the Internet. The corpus includes various explicit contextual information from live e-commerce and news broadcasts to classroom lectures and game reviews, giving the model stronger generalization ability in specific scenes. The instruction dataset and the fine-grained instruction dataset contain 2.4K and 200 hours of speech, respectively.

[0069] Model training is performed on 8 NVIDIA A100 GPUs. Initially, the model is pre-trained for 1M iterations with a batch size of 64 using a learning rate that starts at 10-4 and gradually decays. A warm-up strategy is adopted in the first 10,000 iterations. Subsequently, the model is fine-tuned on the instruction data for 800K iterations with a batch size of 32 and an additional 100K iterations on the fine-grained instruction dataset. In terms of model configuration, the AR model and the NAR model are built on the LLaMA architecture, including 12 layers of Transformer with a hidden dimension of 1024 and a feed-forward network dimension of 4096. The r value of the LoRA adapter inserted into the MT5 text encoder is 16. For the AR model, to facilitate unconditional generation as part of CFG, the entire text embedding sequence or semantic token sequence is masked with a probability of 0.1 during training. For the NAR model, to support optional speech cue input, a probability of 0.3 is set not to use any prefix acoustic segment. To enable iterative decoding, a cosine schedule is used to randomly mask the acoustic token part of the quantizer of the current layer.

[0070] To verify the effectiveness of the proposed audio generation model, multiple subjective and objective evaluation metrics are used. Given the model’s capability in both instruction-to-audio generation and voice cloning, evaluation metrics focusing on these two aspects are particularly introduced. For instruction-to-audio generation, two Mean Opinion Score (MOS) tests are adopted to evaluate the quality and controllability of the generated audio: MOS-Q measures the quality of the speech, with higher values indicating better speech quality, naturalness, and expressiveness; MOS-I measures the degree to which the speech follows the given human instruction, with higher values indicating better control over the speech attributes of the descriptive instruction. In terms of objective metrics, the generated audio is subjected to automatic speech recognition (ASR) using the Whisper medium model, and the word error rate (WER) with the original transcribed text is calculated. The accuracy of several speech attribute factors of the generated audio is also calculated, using the corresponding classification models.

[0071] For voice cloning, MOS-S is used to measure the similarity between the speech prompt and the generated audio. In terms of objective metrics, Mel-Cepstral Distortion (MCD) is adopted to evaluate the difference between the generated audio and the real speech. The speaker embedding cosine similarity (SECS) between the generated audio and the speech prompt is also evaluated. Specifically, Resemblyzer6 is used to extract the sentence-level speaker embedding to calculate the cosine similarity. For all subjective MOS tests, 20 participants are involved in the evaluation, and score on a rating scale of 1 to 5 with 1-point intervals.

[0072] The proposed audio generation model VoxInstruct is compared with several text-prompted TTS and speech-prompted TTS systems. For text-prompted TTS, multilingual versions of PromptTTS and Salle are replicated and trained on the instruction dataset and the fine-grained instruction dataset. Specifically, the instruction text prompts are processed to exclude the content part, consistent with the original setup that models content and style separately. And both use Vocos decoder as their vocoder. For speech-prompted TTS, monolingual Vall-E

[28] and cross-lingual Vall-E X

[34] are chosen as the baseline. Due to the high replication cost, some audio samples are directly collected from their demo pages7 for comparison.

[0073] To demonstrate the ability of VoxInstruct to convert human instructions into expressive speech, subjective and objective experiments were first conducted on the English test set. The test samples were taken from GigaSpeech-s, which were unseen during training. The corresponding human instructions were generated for these samples using the instruction annotation pipeline. Since there is real speech as a reference, MCD and SECS can be calculated using real speech as a reference in this section. It was found that VoxInstruct achieved the highest 4.22 on MOS-Q and 3.76 on MOS-I, significantly outperforming the two baseline models. Replication For example, the "Advanced Scene Semantic Understanding" example can be expanded on data collection In the meta, it can also be a single unit, or two or more units architecture and using MSE loss. This severely affected its subjective evaluation results and WER values. In objective evaluation, VoxInstruct also achieved the highest average classification accuracy on speech attribute factors and the closest similarity to real speech, and achieved a WER value of 2.5 comparable to the current other most advanced TTS systems. This shows that VoxInstruct has the ability to understand unified human instructions, can recognize the description of sound characteristics and accurate spoken content in the instructions, and generate expressive speech consistent with the given instructions. In addition, it can be observed that the introduction of the pre-training stage brings a slight improvement in speech attribute control and a more significant improvement in intelligibility, which is intuitively expected.

[0074] In addition, Chinese experiments were also conducted. Unlike English tests, Chinese instructions were generated by first randomly sampling speech attributes and then rewriting them using a large language model (LLM).

[0075] Although VoxInstruct performs comparable or slightly inferior to the baseline models on objective accuracy indicators, it significantly outperforms other models with high scores of 4.01 and 3.83 on subjective indicators MOS-Q and MOS-I. This shows that the present invention also performs well in understanding Chinese instructions and generating audio. In addition, it is revealed that VoxInstruct can inherently understand mixed language instructions and directly produce code-switched speech, eliminating the need for any grapheme-to-phoneme (G2P) conversion.

[0076] To demonstrate the superior fine-grained control of the present application's unified instruction-based audio generation method in controlling speech, all these models were fine-tuned on a fine-grained instruction dataset and evaluated using an internal accent detection model. 200 instructions containing detailed emphasis information were used to synthesize test samples. The accuracy of correctly detecting accent words in all words in synthesized speech (Acc_word), and the accuracy of correctly identifying accent words in all sentences (Acc_sentence). The results show that the method adopted by PromptTTS, which focuses on the mapping between text prompts and overall speech style embeddings, has difficulty in achieving fine-grained control. In contrast, the method of the present application uses unified instruction prompts as input, and outperforms Salle's method in fine-grained control ability, which models content and style prompts separately.

[0077] The present application's VoxInstruct was compared with zero-shot TTS models VALL-E and Vall-E X, which focus on monolingual and cross-lingual scenarios, respectively. The results reflect that the present application achieves comparable performance to the current leading zero-shot voice cloning TTS model. Although VoxInstruct is fine-tuned on an instruction dataset, it still retains the strong ability to mimic the voice in the speech prompt. In addition, the present application is significantly superior to VALL-E in naturalness and speech quality. The Vocos decoder and pre-trained MT5 text encoder provide rich semantic information.

[0078] Those skilled in the art will understand that the method of the present application is suitable for various types of audio generation capabilities, ​ ​ ("ocean wave sound" "piano sound" is also understood as a kind of scene description); in addition, the modeling way of song and speech is highly consistent, which can be understood as a kind of speech with specific intonation and rhythm.

[0079] The embodiments of the present application also provide a storage medium for storing a computer program, which is executed to perform at least the method as described above.

[0080] The embodiments of the present application also provide a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute the computer program to perform at least the method as described above.

[0081] The embodiments of the present application also provide a processor, which executes a computer program to perform at least the method as described above.

[0082] The storage medium can be implemented by any type of nonvolatile storage device, or a combination thereof. The nonvolatile memory can be a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Ferromagnetic Random Access Memory (FRAM), a Flash Memory, a magnetic surface storage, an optical disc, or a Compact Disc Read-Only Memory (CD-ROM). The magnetic surface storage can be a magnetic disc memory or a magnetic tape memory. The storage medium described in the embodiments of the present application is intended to include, but is not limited to, these and any other suitable type of memory.

[0083] In several embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other manners. The described device embodiments are merely schematic, and the division of the units is merely a logical function division. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling, or the direct coupling or communication connection between the components can be indirect coupling or communication connection through some interfaces, devices, or units, and can be electrical, mechanical, or in other forms.

[0084] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0085] In addition, each functional unit in the embodiments of the present application can be integrated in one processing unit, or each of the functional units can exist alone physically, or two or more functional units can be integrated in one unit. ​ ​ The integrated unit can be implemented in the form of hardware, or in the form of hardware plus software functional units.

[0086] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program performs the steps of the above-mentioned method embodiments when executed; and the foregoing storage medium includes a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disc or an optical disc and various storage medium capable of storing program codes.

[0087] Alternatively, the integrated unit of the present application can be stored in a computer readable storage medium if it is realized in the form of a software function module and sold or used as an independent product. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes a mobile storage device, a ROM, a RAM, a magnetic disc or an optical disc and various storage medium capable of storing program codes.

[0088] The methods disclosed in the several method embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0089] The features disclosed in the several product embodiments provided by the present application can be combined arbitrarily without conflict to obtain new product embodiments.

[0090] The features disclosed in the several method or device embodiments provided by the present application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

[0091] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art, without departing from the concept of the present application, a number of equivalent substitutions or obvious modifications can be made, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present application.

Claims

1. A method for high expressivity audio generation based on natural language description text, characterized in that, The method comprises the following steps: S1: encode the input natural language description instruction using a pre-trained multilingual text encoder to extract all semantic information in the description text; S2: use the text encoding representation as a condition to predict the semantic unit sequence of the audio in sequence through an autoregressive model (AR), wherein the semantic unit sequence represents the content event information appearing in the audio; S3: use the text encoding representation and the audio semantic unit sequence output by S2 as a prefix condition to predict the coarse-grained acoustic unit sequence of the audio through an autoregressive model (AR), to determine the overall duration and basic acoustic information of the audio; S4: use a non-autoregressive model (NAR) to predict the fine-grained acoustic unit sequence of the audio in a multi-layer iterative manner, to model the audio detail performance and high-frequency information; during inference, perform a conditional continuation task while adjusting the classifier-free guidance (CFG) parameter to determine whether the generated audio is more dependent on instruction control or reference speech control; S5: input the multi-layer acoustic unit sequence into a vocoder to generate an audio waveform.

2. The method of claim 1, wherein the natural language description text-based high expressivity audio generation method is characterized by, In step S1, a multilingual text encoding modeling module is used to encode the input natural language description instruction to extract all semantic information in the description text; and the text encoding representation is used to generate an audio sample corresponding to the text description instruction, to ensure that the generated audio is consistent with the input text in terms of semantics and acoustic features.

3. The method of claim 2, wherein the natural language description text-based high expressivity audio generation method is characterized by, In step S1, a pre-trained multilingual text encoder is used to encode the input natural language description instruction to extract all semantic information in the description text; a multilingual T5 base model (MT5-base) is selected as the text encoder, and a pre-trained text encoder of the MT5-base model is used with a trainable low-rank adaptation (LoRA) adapter inserted to support multilingual instruction input; the original text of the human instruction is passed to the MT5 encoder to derive a text embedding sequence.

4. The method of claim 1, wherein the natural language description text-based high expressivity audio generation method is characterized by, In step S2, advanced scene semantic features are introduced in the data processing flow and embodied in the generated audio; when predicting the semantic unit sequence through the autoregressive model (AR), the advanced scene information in the description text is considered, including the generation of style intonation, background environment sound, music melody, and sound effect event; the local performance of the generated audio is allowed to be controlled through the instruction in the natural language description.

5. The method of claim 4, wherein, In step S2, a self-supervised representation model HuBERT or an audio self-supervised representation model AudioMAE is used to extract semantic unit features of the audio; the embeddings extracted by HuBERT are discretized through a k-means clustering algorithm to generate discrete semantic tags (ST); continuous repeated tags in the generated semantic tag sequence are removed to form a clear and non-redundant semantic unit sequence; a language tag is added in front of the semantic tag sequence to indicate the language information of the generated audio; An autoregressive model (AR) is used to predict the semantic unit sequence of the audio based on the text encoding representation and the semantic tag sequence as conditions, wherein the semantic unit sequence directly maps to the event content and sequence information appearing in the synthesized audio.

6. The high expressivity audio generation method based on natural language description text of claim 1 or 2, wherein, In step S3, in the timbre cloning or cross-language timbre cloning scenario, the text to be synthesized is provided as an instruction; in the inference, the semantic and acoustic units of the reference speech provided by the user are extracted and set as the sequence prefix in the inference process of the autoregressive model; Using the context learning ability of the autoregressive model, the conditional continuation task is performed to imitate the timbre style of the reference speech; In step S4, in the sound style modification scenario, the user provides a style description and the text to be synthesized as an instruction.

7. The method of claim 1, wherein the natural language description text-based high expressivity audio generation method is characterized by, Further comprising: Joint training of the text encoder, the AR model and the NAR model; Training the pre-trained multi-language text encoder using large-scale description-free annotated audio data to extract and understand the semantic information in the natural language description instruction, which only provides the ASR automatically annotated speech transcription text or the basic category label of the audio; training the autoregressive model (AR) and the non-autoregressive model (NAR) using large-scale description-free annotated audio data to generate coarse-grained and fine-grained acoustic unit sequences; Fine-tuning the pre-trained text encoder using small-scale description-annotated audio data to enhance the model's understanding of descriptive text; fine-tuning the autoregressive model (AR) using small-scale description-annotated data to improve the accuracy and expressiveness of the generated semantic unit sequence; fine-tuning the non-autoregressive model (NAR) using small-scale description-annotated data to optimize the generated fine-grained acoustic unit sequence and enhance the detail performance and high-frequency information of the audio.

8. The method of claim 1, wherein the natural language description text-based high expressivity audio generation method is characterized by, Further comprising: In the training phase of the autoregressive model (AR), the text encoding representation or the semantic unit sequence is masked by a certain probability to realize the training in the form of the conditional guided strategy CFG; based on the partially masked text encoding representation, the coarse-grained acoustic unit sequence is predicted to generate the overall duration and basic acoustic features of the audio; In the training phase of the non-autoregressive model (NAR), the output of the autoregressive model is used as a condition to predict the fine-grained acoustic unit sequence through multi-layer iteration, while implementing the unconditional guided strategy to enhance the detail performance and high-frequency information of the audio; In the inference generation process, by adjusting the parameters of the classifier-free guided strategy (CFG), the semantic description relevance of the generated audio, the accuracy of the audio content, the intelligibility of the pronunciation content for speech, and the similarity of the reference audio style are adjusted to realize fine control of the audio content style; according to the input natural language description instruction, the CFG coefficient is dynamically adjusted to meet the demand for audio expressiveness and controllability in different scenarios.

9. The method of claim 1, wherein, In the training phase of the autoregressive model (AR), the autoregressive model (AR) is used to predict coarse-grained acoustic markers, which include the basic acoustic content structure of the audio, the main prosody and basic timbre of the speech and singing parts, and determine the overall audio duration; In the inference process, the instruction text is embedded and the reference audio prompt is used as input using the encoder-decoder language model to produce target acoustic markers; After generating the acoustic markers, the audio waveform is synthesized using the acoustic decoder.

10. The high expressiveness audio generation method based on natural language description text of claim 1, wherein, In predicting semantic labels, the text embedding sequence is masked, so that the model makes predictions in a form of unconditional generation, thereby improving the compliance with natural language instructions; In predicting coarse-grained acoustic labels, the text embedding sequence or the semantic label sequence is masked, also in a form of unconditional generation, thereby further enhancing the model's understanding of the instructions and the naturalness of the generated audio; In the training phase of the autoregressive model (AR), based on the partially masked text encoding representation, the coarse-grained acoustic unit sequence is predicted to generate the overall duration and basic acoustic features of the audio; In the training phase of the non-autoregressive model (NAR), using the output of the autoregressive model as a condition, the fine-grained acoustic unit sequence is predicted through multi-layer iteration, while implementing an unconditional guidance strategy to enhance the details and high-frequency information of the audio; In the log space, the semantic labels are sampled, combined with the unconditional guidance strategy, to optimize the semantic relevance and naturalness of the generated audio; In sampling the coarse-grained acoustic labels, two types of CFGs are used simultaneously, allowing the generated acoustic labels to focus on different aspects, including the control of audio features and the intelligibility of audio content; In the inference process, the CFG coefficients are dynamically adjusted to meet the demand for audio expressiveness in different scenarios.

11. The high expressiveness audio generation method based on natural language description text of claim 1, wherein, The text encoder, the AR model and the NAR model are jointly trained; In the pre-training phase, a pre-trained multilingual text encoder is trained using public audio data without description annotation to extract and understand the semantic information in natural language description instructions, the public audio data without description annotation only provides ASR automatically annotated speech transcription text or basic category labels of audio; through model pre-training, the model has basic text-audio association modeling capability and audio generation capability, as well as stable and clear text-to-speech synthesis and zero-shot sound cloning capability; In the fine-tuning phase, the pre-trained text encoder is fine-tuned using instruction-audio paired data to give the model the ability to understand the descriptive information in human instructions; in the fine-tuning phase, a progressive fine-tuning strategy is adopted, first, a small-scale description annotated data is used to fine-tune the autoregressive model (AR) to improve the accuracy and expressiveness of the generated semantic unit sequence; in the fine-tuning phase, the parameters of the classifier-free guidance strategy (CFG) are adjusted to meet the adjustment of different needs including audio style description relevance, pronunciation intelligibility, and reference timbre similarity; thereby, the model is given the ability to understand fine description information in human instructions, and the accuracy of the generated semantic unit sequence and the expressiveness of the generated audio are improved.

12. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the high expressiveness audio generation method based on natural language description text of any one of claims 1-10.

Images