High-expressive-force audio generation method based on natural language description text

Abstract

A high-expressive-force audio generation method based on a natural language description text comprises the steps that S1, a pre-training multi-language text encoder is used for encoding an input natural language description instruction, and semantic information is extracted; s2, with text coding representation as a condition, semantic unit sequences of the audio are sequentially predicted through an autoregression model (AR), and the semantic unit sequences represent content event information appearing in the audio; s3, taking the text coding representation and the audio semantic unit sequence output in the S2 as a prefix condition, predicting a coarse-grained acoustic unit sequence of the audio through AR, and determining the overall duration and basic acoustic information of the audio; s4, predicting a fine-grained acoustic unit sequence of the audio by using a non-autoregressive model (NAR), and modeling audio detail representation and high-frequency information; and S5, inputting the multi-layer acoustic unit sequence into a vocoder to generate a high-quality audio waveform. The method directly supports the generation of the audio by a sentence of natural language description instruction, and improves the naturalness, expressive force and controllability of the generated audio.

Description

Technical Field

[0001] The present invention relates to artificial intelligence audio generation technology, and in particular to a high-expressive audio generation method based on natural language description text. Background Art

[0002] Recent artificial intelligence generated content (AIGC) systems have the ability to generate digital multimedia content, such as text, images, and videos, based on human language instructions. However, existing TTS solutions can only control the pronunciation content and style of audio through content prompts and description prompts, for example, the content prompt is "The weather is really nice today" and the description prompt is "Young woman, happy and satisfied"; it is not possible to directly generate understandable and highly expressive audio based entirely on a natural language description, for example, the instruction is "In a quiet park, the leaves rustle, and a young mother happily chats, "The weather is really nice today", and the tone of "really" is slightly emphasized, showing a sense of satisfaction." Therefore, the current "Wensheng Audio" still has a gap with other AIGC generation paradigms. In addition, the description prompts and content prompts of existing solutions are independent of each other, and the pronunciation content text is often ignored when modeling audio style, which limits the ability to control the audio at a fine-grained level (for example, in the above example, when describing, you want to emphasize the stress of a certain word).

[0003] PromptTTS uses separate content prompts and description prompts for audio synthesis. The content prompt is a text phoneme sequence, which can achieve global style control (including gender, speaking speed, pitch, etc.) through natural language description. The acoustic model is the FastSpeech 2 framework, which extracts the global style representation of the audio through the style encoder, and uses the pre-trained BERT model to extract the text encoding information of the description prompt to predict the style representation.

[0004] Salle uses separate content prompts and description prompts for audio synthesis. The content input is a text phoneme sequence, and style control (including gender, speaking speed, emotion, etc.) can be achieved through natural language description. The acoustic model uses the GPT architecture to model the discrete acoustic units of the audio, and directly concatenates the description text sequence before the text phoneme sequence to predict the acoustic part.

[0005] Disadvantages: Existing solutions require clear and separate content prompts and description prompts, and description prompts can only control the overall timbre style of the audio. It is impossible to directly generate audio from a mixed form, flexible and diverse natural language description instruction and achieve fine control; the capabilities of existing solutions are single, and can only control the timbre style of audio by text description or by using reference audio, and cannot use a unified system to achieve multiple task functions; existing text description-based speech synthesis solutions can only use description annotated data for training, the system has limited generalization, and cannot utilize large-scale unlabeled data.

[0006] In general, in terms of audio, existing methods related to human-instruction-to-audio generation have two limitations. First, they need to separate the input into content cues (transcriptions) and description cues (style and speaker) instead of directly supporting human instructions. This division is not natural enough in form and is inconsistent with other AIGC models. Second, using independent description cues to model audio style without considering the transcription content limits the ability to control the audio at a fine-grained level.

[0007] In addition, the existing "Text-to-Audio" (TTA) solutions can only generate simple audio events. For example, AudioGen and MusicLM solutions support inputting descriptive texts such as "rustling leaves", "car honking", and "playing a sad violin piece" and synthesizing corresponding audio clips. Since the input text prompt does not support the given speaking content, it is impossible to achieve high-quality understandable speech synthesis and the joint and harmonious generation of audio (specifically sound effects, music, etc.) and speech. This limits the further development and application of scenarios such as Text-to-Audio and video dubbing.

[0008] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0009] The main purpose of the present invention is to solve the problems existing in the above-mentioned background technology and provide a highly expressive audio generation method based on natural language description text.

[0010] To achieve the above object, the present invention adopts the following technical solutions:

[0011] In a first aspect, the present invention provides a method for generating highly expressive audio based on a natural language description text, comprising the following steps:

[0012] S1: Use the pre-trained multilingual text encoder to encode the input natural language description instructions and extract all semantic information in the description text;

[0013] S2: using the text encoding representation as a condition, sequentially predicting a semantic unit sequence of the audio through an autoregressive model (AR), wherein the semantic unit sequence represents content event information appearing in the audio;

[0014] S3: Using the text encoding representation and the audio semantic unit sequence output by S2 as prefix conditions, the coarse-grained acoustic unit sequence of the audio is predicted through an autoregressive model (AR) to determine the overall duration of the audio and basic acoustic information;

[0015] S4: Use a non-autoregressive model (NAR) to predict the fine-grained acoustic unit sequence of the audio in a multi-layer iterative manner, modeling the audio detail representation and high-frequency information;

[0016] S5: Input the multi-layer acoustic unit sequence into a vocoder to generate an audio waveform.

[0017] In a second aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method for generating highly expressive audio based on natural language description text.

[0018] The present invention has the following beneficial effects:

[0019] Compared with the existing technology, this application proposes for the first time to directly support the generation of audio by describing instructions in a natural language, aligns the paradigms of AIGC audio generation and other modal generation, and is applicable to a variety of audio generation tasks, and also greatly improves the naturalness and expressiveness of the generated audio.

[0020] The present invention proposes a Vincent audio system (VoxInstruct for short) that directly supports fully natural language description instructions. The embodiments of the present invention have the following features and advantages:

[0021] 1. The entire system includes two models: autoregressive and non-autoregressive. The discrete features of the audio are used as the basic modeling unit, and the next token prediction is used as the training target. Thanks to the in-context learning capability of the language model, it has better performance in rhythm and naturalness than traditional TTS models.

[0022] 2. In order to enable the model to automatically understand the content information in the description instructions, a pre-trained text encoder is used to obtain the text encoding information of the input instructions, and an audio semantic unit (semantic token) representation is introduced to guide the model to learn how to distinguish and extract the actual pronunciation content, without the need to give an additional specific content text prompt or phoneme sequence;

[0023] 3. In order to improve the compliance of synthesized audio with description instructions, a variety of unconditional guidance strategies CFG are designed. By setting different CFG parameters during reasoning, the style control and pronunciation content intelligibility of synthesized audio are strengthened respectively;

[0024] 4. In order to improve the generalization of the VoxInstruct model system, a "pre-training + fine-tuning" training paradigm is adopted, which allows the model to be pre-trained on large-scale audio data (without descriptive text annotations), further improving the naturalness and pronunciation stability of the synthesized audio;

[0025] 5. By introducing language tags for multilingual modeling, the system can support bilingual description instruction text input in Chinese and English, and generate audio results in Chinese, English, and mixed languages.

[0026] Compared with the prior art, the technical solution of the present invention can accept natural language description instruction text input without any additional preprocessing work, and establishes an end-to-end description text to audio generation system. The present invention can use a large amount of non-description annotated data for pre-training, promote the model to learn the association between acoustic units and semantic units of audio, and improve the performance of the model. The present invention can use the same model to support input in multiple languages ​​and support multiple audio generation task scenarios.

[0027] In addition, the present invention integrates traditional TTS tasks, TTS tasks based on description prompts and content prompts, and TTA tasks based on text prompts into a common framework, proposes a unified paradigm for audio generation based on natural language description text instructions, and further supports the generation of highly expressive speech or singing results with ambient sounds such as music effects.

[0028] The present invention has important value for the human-computer interaction industry and can be used in a variety of application scenarios such as audio synthesis, voice interaction, voice assistant, audio novel generation, film and television script dubbing, music, singing, scene sound effects, etc. It allows users to create the audio results they want in a personalized way through command descriptions.

[0029] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of the Vincent audio system VoxInstruct that directly supports fully natural language description instructions according to an embodiment of the present invention.

[0031] Figure 2 This is an algorithm block diagram of a method for generating highly expressive audio based on natural language description text according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0033] The audio generation method of the present invention can generate a corresponding highly expressive audio result based on a natural language description text. Unlike the traditional TTS system that only synthesizes all the pronunciation content of the input text word by word, the present invention aims to further understand the semantic connotation and actual needs of the text instructions input by the user, automatically analyze and extract the descriptive information about the audio style and the pronunciation information about the audio content, so as to generate natural audio that conforms to the human instruction description, and provide an AIGC audio solution similar to the paradigm of text image and text video. Figure 1 and Figure 2 The system proposed in the present invention (hereinafter referred to as VoxInstruct) first uses a pre-trained multilingual text encoder to encode the input natural language description instructions and extract all semantic information in the description text; then, using the text encoding representation as a condition, the autoregressive model predicts the audio semantic unit sequence (indicating the content event information appearing in the audio) and the coarse-grained acoustic unit sequence (determining the overall duration and basic acoustic features of the audio) in sequence; then the non-autoregressive model predicts the audio fine-grained acoustic unit sequence (modeling audio detail performance and high-frequency information) in a multi-layer iterative manner; finally, the vocoder reconstructs a high-quality audio waveform based on the multi-layer acoustic unit sequence. In addition, in the prediction process, the embodiment of the present invention proposes a variety of classifier-free guidance strategies to improve the compliance of the synthesized audio with the instructions, so as to strengthen the control ability of the description text on the audio generation process. The system can support text instruction input forms in multiple languages ​​such as Chinese and English and mixed languages, and is suitable for a variety of audio generation tasks, including text description to audio generation, single language / cross-language timbre cloning based on reference audio, timbre style modification based on reference audio and text description, etc. The specific system capabilities are shown in the figure below. These capabilities are realized through settings such as input instructions (whether the instructions contain a description of the audio timbre style, etc., or only contain pronunciation content, and whether the pronunciation content is in Chinese or English), and whether to give the model an additional reference audio.

[0034] VoxInstruct can adopt the LlaMA architecture, including autoregressive and non-autoregressive models, audio discrete unit encoders and vocoders, and Multinugal T5 text encoders. 1) By introducing audio semantic units as intermediate representations, the model can more accurately grasp the pronunciation content of instructions and effectively convert them into target audio; 2) Through the pre-training + fine-tuning paradigm, large-scale undescribed annotated audio data can be used to improve the overall generalization of the system, and small-scale described annotated data can be used to give the model fine instruction control capabilities; 3) In the training stage, the training of the unconditional guidance strategy CFG form is realized by masking the instruction text encoding representation, semantic unit sequence and other conditions with a certain probability, and in the inference generation process, by setting the CFG coefficient, the adjustment of different requirements such as audio style description relevance, pronunciation intelligibility, and reference timbre similarity can be met, thereby achieving more refined audio expression control.

[0035] The system process is as follows: first, the pre-trained multilingual text encoder is used to encode the input natural language description instructions and extract all semantic information in the description text; then, using the text encoding representation as a condition, the autoregressive model (AR) is used to predict the audio's semantic unit sequence (representing the content event information appearing in the audio) and coarse-grained acoustic unit sequence (determines the overall audio duration and basic acoustic features) in sequence; then, the non-autoregressive model (NAR) is used to predict the audio's fine-grained acoustic unit sequence (modeling audio detail performance and high-frequency information) in a multi-layer iterative manner; finally, the vocoder generates a high-quality audio waveform based on the multi-layer acoustic unit sequence.

[0036] See also Figure 1 and Figure 2 The method for generating highly expressive audio based on natural language description text according to an embodiment of the present invention comprises the following steps:

[0037] S1: Use the pre-trained multilingual text encoder to encode the input natural language description instructions and extract all semantic information in the description text;

[0038] S2: using the text encoding representation as a condition, predicting the semantic unit sequence of the audio in sequence through an autoregressive model (AR), wherein the semantic unit sequence represents content event information appearing in the audio, such as basic pronunciation units of speech, etc.;

[0039] S3: Using the text encoding representation and the audio semantic unit sequence output by S2 as prefix conditions, the coarse-grained acoustic unit sequence of the audio is predicted through an autoregressive model (AR) to determine the overall duration of the audio and basic acoustic information;

[0040] S4: Use a non-autoregressive model (NAR) to predict the fine-grained acoustic unit sequence of the audio in a multi-layer iterative manner, modeling the audio detail representation and high-frequency information;

[0041] S5: Input the multi-layer acoustic unit sequence into a vocoder to generate an audio waveform.

[0042] The highly expressive audio generation method of the present invention realizes the efficient conversion of natural language description text to high-quality audio through a series of innovative steps. The method first uses a pre-trained multilingual text encoder to deeply encode the input natural language description and accurately extract the semantic information in the text. Then, based on the encoded representation, the autoregressive model (AR) predicts the semantic unit sequence of the audio, which is directly mapped to the actual pronunciation content. Further, based on the output of the AR model, a coarse-grained acoustic unit sequence is predicted, which determines the overall duration and basic acoustic features of the audio, laying the foundation for generating natural and smooth audio. In order to enrich the details and high-frequency information of the audio, a non-autoregressive model (NAR) is used to predict a fine-grained acoustic unit sequence in a multi-layer iterative manner, thereby enhancing the expressiveness and realism of the audio. Finally, the multi-layer acoustic unit sequence is converted into a high-quality audio waveform by a vocoder, completing the whole process from text to audio. The whole process not only improves the naturalness and expressiveness of audio synthesis, but also greatly broadens the application scope and flexibility of audio generation technology by supporting multilingual and mixed language input. In addition, this method achieves end-to-end audio generation without complex pre-processing and post-processing steps, which simplifies the operation process and improves efficiency.

[0043] The audio generation method of the present invention is applicable to general audio scenarios, including the generation of speech, music, singing, sound effects, etc., and the generation of their combined forms. Corresponding types of training data can be introduced for different scenarios. For example, the description instruction input to the model can be: Example 1. At the seaside, the sea breeze blew, and a five-year-old boy shouted happily, "Wow, what a big lake!" Then the cry of seagulls was heard. Example 2. In the melodious piano and violin ensemble, she gradually became intoxicated and hummed in a low voice, "A big river with wide waves...").

[0044] In some embodiments, in step S1, a multilingual T5encoder supporting multiple languages ​​is used as a text encoding modeling module to encode the input natural language description instructions and extract all semantic information in the description text; the text encoding representation can process input texts in multiple languages, including but not limited to Mandarin and English, and text instructions in a mixture of Chinese and English; using the text encoding representation, an audio sample corresponding to the text description instruction is generated to ensure that the generated audio is consistent with the input text in terms of semantic and acoustic features.

[0045] In a preferred embodiment, in step S1, a pre-trained multilingual text encoder is used to encode the input natural language description instruction to extract all semantic information in the description text; a multilingual T5 base model (MT5-base) is selected as the text encoder, and its pre-trained text encoder is used to insert a trainable low-rank adaptation (LoRA) adapter to support multilingual instruction input; the original text of the human instruction is passed to the MT5 encoder to derive a text embedding sequence E, which has the number of subwords T after text tokenization E .

[0046] In some embodiments, in step S2, high-level scene semantic features are introduced into the data processing flow (more specifically, including how to extract and represent content event information of audio in the data processing flow), such as "crosstalk stage" and "dialogue scene", and these features are ensured to be reflected in the generated audio; when predicting the semantic unit sequence through the autoregressive model (AR), the high-level scene information in the description text is considered to generate style intonation, background ambient sound, music melody, sound effect events, etc.; the user is allowed to control the local performance of the generated audio, such as priority, through instructions in the natural language description, such as "emphasize a word in the tone";

[0047] In a preferred embodiment, in step S2, the self-supervised representation model HuBERT or the audio self-supervised representation model AudioMAE is used to extract the semantic unit features of the audio (that is, the event content appearing in the audio) as the prediction target of the AR model in the S2 stage during training; the embedding extracted by HuBERT is discretized by the k-means clustering algorithm to generate discrete semantic tags (ST); the continuous repeated tags in the generated semantic tag sequence are removed to form a clear and non-redundant semantic unit sequence to indicate the language information of the generated audio; and a language tag is added before the semantic tag sequence (if there is an audio part, otherwise it is set to a special empty tag) to represent the language information of the audio content.

[0048] Next, the autoregressive model (AR) can be used to predict the semantic unit sequence of the audio with the text encoding representation and the semantic tag sequence as conditions. The semantic unit sequence is directly mapped to the event content and order information appearing in the audio to be synthesized, and may include the basic pronunciation units of the audio part, the basic audio event units of the sound effect part, the instrument note units of the music part, etc. These units still constitute a semantic unit sequence.

[0049] In some embodiments, in step S3, in a timbre cloning or cross-language timbre cloning scenario, the user provides the text to be synthesized as an instruction; during reasoning, the semantic and acoustic units of the reference audio provided by the user are extracted and set as sequence prefixes in the autoregressive model reasoning process; the context learning ability of the autoregressive model is used to perform a conditional continuation task to achieve accurate imitation of the timbre style of the reference audio;

[0050] In some embodiments, in step S4, in the sound style modification scenario, the user provides the style description to be converted and the text to be synthesized as instructions; in the reasoning process, a conditional continuation task is performed, and the classifier-free guidance strategy (CFG) parameters are adjusted; by adjusting the CFG parameters, it is determined whether the effect of the generated audio is more dependent on instruction control or reference audio control, so that the user can generate audio that is more in line with the instruction description or closer to the characteristics of the reference audio according to needs.

[0051] During the training process of the autoregressive model, an unconditional guidance strategy with a certain probability, such as masked text encoding or semantic tagging, is adopted to enhance the model's understanding of the instruction text and the accuracy of generated audio. During the inference process, the semantic relevance, pronunciation intelligibility and reference timbre similarity of the generated audio are optimized by adjusting the parameters of the classifier-free guidance strategy (CFG).

[0052] In some embodiments, during the pre-training stage, a pre-trained multilingual text encoder is trained using large-scale audio data without description annotations to extract and understand semantic information in natural language description instructions; during the pre-training stage, an autoregressive model (AR) and a non-autoregressive model (NAR) are trained using large-scale audio data without description annotations to generate coarse-grained and fine-grained acoustic unit sequences. In the fine-tuning stage, the pre-trained text encoder is fine-tuned using a small-scale description annotation data to enhance the model's ability to understand descriptive text; in the fine-tuning stage, the autoregressive model (AR) is fine-tuned using a small-scale description annotation data to improve the accuracy and expressiveness of the generated semantic unit sequence; in the fine-tuning stage, the non-autoregressive model (NAR) is fine-tuned using a small-scale description annotation data to optimize the generated fine-grained acoustic unit sequence and enhance the detail expression and high-frequency information of the audio; in the fine-tuning stage, the parameters of the classifier-free guidance strategy (CFG) are adjusted to meet the adjustment of different requirements such as audio style description relevance, pronunciation intelligibility, and reference timbre similarity; in the fine-tuning stage, the parameters of the vocoder are optimized to generate higher quality audio waveforms to ensure that the generated audio is optimal in naturalness and clarity.

[0053] The text encoder, AR model, and NAR model are jointly trained, including two training steps: 1) model pre-training on large-scale audio data without description annotations (only audio transcription text automatically annotated by ASR or basic category labels of audio can be provided), and 2) model fine-tuning on audio data with high-quality description annotations (such as the various description instruction examples listed above).

[0054] Through the above pre-training and fine-tuning process, highly expressive audio generation of natural language description text is achieved, and the generated audio can accurately reflect the semantic and style characteristics of the input text.

[0055] In a preferred embodiment, during the training phase of the autoregressive model (AR), the text encoding representation or semantic unit sequence is masked with a certain probability to implement training in the form of an unconditional guidance strategy CFG, so as to enhance the model's understanding of natural language descriptions and the accuracy of generated audio. During the training phase of the autoregressive model (AR), a coarse-grained acoustic unit sequence is predicted based on a partially masked text encoding representation to generate the overall duration and basic acoustic features of the audio; during the training phase of the non-autoregressive model (NAR), the output of the autoregressive model is used as a condition to predict a fine-grained acoustic unit sequence through a multi-layer iterative method, and an unconditional guidance strategy is implemented at the same time to enhance the detailed performance and high-frequency information of the audio.

[0056] In a preferred embodiment, an autoregressive model (AR) is used to predict coarse-grained acoustic markers, which contain the basic acoustic content structure of the audio, the main rhythm and basic timbre of the audio singing part, and determine the overall audio duration. In the training phase of the non-autoregressive model (NAR), the output of the autoregressive model is used as a condition to predict fine-grained acoustic markers through a multi-layer iterative method, and an unconditional guidance strategy is implemented to enhance the detail performance and high-frequency information of the audio;

[0057] In some embodiments, during the inference generation process, the semantic relevance, pronunciation intelligibility and reference timbre similarity of the generated audio are adjusted by setting the CFG coefficient to achieve fine control of the audio style; during the inference generation process, the CFG coefficient is dynamically adjusted according to the input natural language description instructions to meet the needs of audio expressiveness in different scenarios and generate audio with high naturalness and expressiveness.

[0058] In some embodiments, during the inference process, the instruction text embedding and audio prompts (if provided) are taken as input using a codec language model to generate a target acoustic tag; after the acoustic tag is generated, an acoustic tag decoder such as Vocos is used to synthesize the audio waveform instead of the original Encodec decoder to provide better audio reconstruction quality.

[0059] According to a preferred embodiment of the present invention, in the training stage of the autoregressive model (AR), a classifier-free guidance strategy (CFG) is adopted to mask the text encoding representation or semantic unit sequence with a certain probability to realize the training of the unconditional guidance strategy, so as to enhance the model's understanding of natural language descriptions and the accuracy of generated audio. When predicting semantic tags, the text embedding sequence is masked so that the model makes predictions in the form of unconditional generation, thereby improving the compliance with natural language instructions; when predicting coarse-grained acoustic tags, the text embedding sequence or semantic tag sequence is masked, and the form of unconditional generation is also adopted to further enhance the model's understanding of instructions and the naturalness of generated audio. In the training stage of the autoregressive model (AR), based on the partially masked text encoding representation, the coarse-grained acoustic unit sequence is predicted to generate the overall duration and basic acoustic features of the audio. In the training stage of the non-autoregressive model (NAR), the output of the autoregressive model is used as a condition to predict the fine-grained acoustic unit sequence through a multi-layer iterative method, and the unconditional guidance strategy is implemented at the same time to enhance the detailed performance and high-frequency information of the audio. During the inference generation process, the semantic relevance, pronunciation intelligibility, and reference timbre similarity of the generated audio are adjusted by setting the CFG coefficients to achieve fine control of the audio style. Semantic markers are sampled in logarithmic space and combined with an unconditional guidance strategy to optimize the semantic relevance and naturalness of the generated audio. When sampling coarse-grained acoustic markers, two types of CFG are used simultaneously, allowing the generation of acoustic markers to focus on different aspects, such as the control of audio features and the intelligibility of audio content. During the inference process, the CFG coefficients are dynamically adjusted to meet the needs of audio expressiveness in different scenarios, generating audio with high naturalness and expressiveness.

[0060] According to a preferred embodiment of the present invention, in the pre-training stage, a large public audio dataset containing only text transcriptions is used to train a pre-trained multilingual text encoder to extract and understand the semantic information in natural language description instructions; the original transcribed text is used as a human instruction and enclosed in quotation marks to ensure that the codec language model has strong text-to-audio synthesis (comprehensibility) and zero-sample sound cloning (generalization) capabilities; in the fine-tuning stage, the pre-trained text encoder is fine-tuned using instruction-audio pairing data to give the model the ability to understand the descriptive information in human instructions; in the fine-tuning stage, a progressive fine-tuning strategy is adopted, and the autoregressive model (AR) is first fine-tuned using a small-scale description annotation data to improve the generation of semantic unit sequences. accuracy and expressiveness; the model is further fine-tuned using a small dataset of fine-grained instructions, so that the model has the ability to control the audio features in detail, including but not limited to fine-grained attributes such as accent marks; in the fine-tuning stage, the non-autoregressive model (NAR) is fine-tuned using a small-scale description annotation data to optimize the generated fine-grained acoustic unit sequence and enhance the detail expression and high-frequency information of the audio; in the fine-tuning stage, the parameters of the classifier-free guidance strategy (CFG) are adjusted to meet the adjustment of different requirements such as audio style description relevance, pronunciation intelligibility, and reference timbre similarity; in the fine-tuning stage, the parameters of the vocoder are optimized to generate higher quality audio waveforms to ensure that the generated audio is optimal in naturalness and clarity.

[0061] It should be noted that the various audio generation tasks supported by the present invention do not require separate training and parameter switching, and can be achieved by simply defining different input formats during the inference process.

[0062] 1. Support for multi-language description instruction text: The text encoding modeling module of this solution uses the multilingual T5 encoder that supports multiple languages, so it can directly support multi-language input text; the audio semantic and acoustic representation units used in this model also support the modeling of multi-language audio, and can generate audio samples in Mandarin, English, and switching between Chinese and English, which has also been verified in the experiment.

[0063] 2. Understanding and fine-grained control of high-level scene information in descriptive texts: Unlike previous solutions that only control attributes such as the speaker's timbre, speaking speed, and emotion, this solution further introduces more advanced scene semantic features such as "crosstalk stage" and "dialogue scenario" in the data processing flow, and reflects this in the style, tone, and background sound of the generated audio; in addition, the local importance and urgency of the generated audio can be further controlled through the description of "emphasizing a certain word in the tone". This capability is due to the joint representation and modeling of content prompts and description prompts in this solution.

[0064] 3. Voice cloning / cross-language voice cloning: In this scenario, the instructions provided by the user only need to be the text to be synthesized. During reasoning, a reference audio provided by the user is obtained. The semantic and acoustic units of the reference audio are first extracted and set as sequence prefixes in the autoregressive model reasoning process, so that it can perform a conditional continuation task. Thanks to the powerful contextual learning ability of LM, the timbre style of the reference audio can be accurately imitated.

[0065] 4. Voice style modification: In this scenario, the instructions provided by the user are the style to be converted and the text to be synthesized. The reasoning process is the same as the previous one. The conditional continuation task is also performed, and the Adjust the CFG parameters to determine the generated audio The effect depends more on command control or reference audio control. This allows the user to decide for themselves whether the generated audio is more in line with the instruction description requirements or closer to the reference audio characteristics.

[0066] The core of the present invention is to use the language model architecture to model and describe the text-to-audio generation task, introduce a sequence of semantic units to ensure that the model has a clear command-pronunciation content understanding ability, and design a variety of CFG strategies to enable the model to synthesize audio results that are more in line with the command requirements. Therefore, the pre-trained text encoder MT5 can be replaced with other pre-trained language models such as BERT, BART, GPT, etc., the HuBERT that extracts audio semantic units can be replaced with other acoustic semantic models such as W2V-BERT, WavLM, etc., the Encodec that extracts audio acoustic units can be replaced with SoundStream, HiFi-Code, cDAC, etc., and the corresponding Vocos that synthesizes audio according to acoustic units can also be replaced with other vocoders.

[0067] experiment

[0068] According to the scenario intention of the pre-training and fine-tuning paradigm, a large amount of data with different annotation granularity is integrated, including transcription-only data, instruction data, and fine-grained instruction data. Large-scale publicly available speech datasets including WenetSpeech Chinese Corpus and GigaSpeech English Corpus are first involved in the pre-training phase. Samples shorter than 3 seconds and low quality are screened out, and a total of 13.4K hours of speech are obtained. Subsequently, a series of instruction-speech pairing datasets are adopted to comprehensively and deeply explain the expressiveness of speech through diverse natural language instructions using the annotation system. Speech instructions characterize speech in terms of spoken content, acoustic properties, speaker identity, emotional tone, and scene context, among which there is a subset of fine-grained descriptions of word emphasis. Details of the annotation system, including expert classifiers and caption models, followed by LLM for instruction rewriting. Experimental examples automatically annotate instructions on some open source datasets. In addition, to enhance generalization capabilities, a considerable amount of scene-rich wild audio data is collected from the Internet. The corpus includes a variety of explicit contextual information, ranging from live e-commerce and news broadcasts to classroom lectures and game reviews, giving the model stronger generalization capabilities in specific scenarios. The instruction dataset and fine-grained instruction dataset contain 2.4K and 200 hours of speech, respectively.

[0069] Model training was performed on 8 NVIDIA A100 GPUs. Initially, the model was pre-trained for 1M iterations with a batch size of 64, using a learning rate that gradually decayed from 10^-4. A warm-up strategy was used in the first 10,000 iterations. After that, the model was fine-tuned on the instruction data for 800K iterations with a batch size of 32, and fine-tuned for an additional 100K iterations on the fine-grained instruction dataset. In terms of model configuration, the AR model and the NAR model are built on the LLaMA architecture, including a 12-layer Transformer with a hidden dimension of 1024 and a feed-forward network dimension of 4096. The r value of the LoRA adapter inserted into the MT5 text encoder is 16. For the AR model, in order to promote unconditional generation as part of the CFG, the entire text embedding sequence or semantic tag sequence is masked with a probability of 0.1 during training. For the NAR model, in order to support optional voice prompt input, a probability of 0.3 is set not to use any prefix acoustic segment. To enable iterative decoding, a cosine schedule is employed to randomly mask the acoustic signature part of the quantizer of the current layer.

[0070] In order to verify the effectiveness of the proposed audio generation model, a variety of subjective and objective evaluation metrics are used. In view of the model's capabilities in command-to-audio generation and voice cloning, evaluation metrics focusing on these two aspects are particularly introduced. For command-to-audio generation, two mean opinion score (MOS) tests are used to evaluate the quality and controllability of the generated audio: MOS-Q measures the quality of the speech, with higher values ​​indicating better speech quality, naturalness, and expressiveness; MOS-I measures the extent to which the speech follows the given human instructions, with higher values ​​indicating better control over the speech attributes of the descriptive instructions. In terms of objective metrics, the generated audio is subjected to automatic speech recognition (ASR) using the Whisper medium model, and the word error rate (WER) is calculated with the original transcribed text. The accuracy of several speech attribute factors of the generated audio is also calculated, using the corresponding classification model.

[0071] For voice cloning, MOS-S is used to measure the similarity between the voice prompt and the generated audio. In terms of objective indicators, Mel-Cepstral Distortion (MCD) is used to evaluate the difference between the generated audio and the real speech. The Speaker Embedding Cosine Similarity (SECS) between the generated audio and the voice prompt is also evaluated. Specifically, Resemblyzer6 is used to extract sentence-level speaker embeddings to calculate cosine similarity. For all subjective MOS tests, 20 participants participated in the evaluation and rated in 1-point intervals on a rating scale of 1 to 5.

[0072] The proposed audio generation model VoxInstruct is compared with several text-prompt-based TTS and speech-prompt-based TTS systems. For text-prompt-based TTS, multilingual versions of PromptTTS and Salle are copied and trained on the Instructions dataset and the Fine-grained Instructions dataset. Specifically, the instruction text prompts are processed to exclude the content part, consistent with their original settings of modeling content and style respectively. And both use the Vocos decoder as their vocoder. For speech-prompt-based TTS, the monolingual Vall-E

[28] and the cross-lingual Vall-EX

[34] are selected as baselines. Due to the high cost of copying, some audio samples are collected directly from their demonstration pages7 for comparison.

[0073] In order to demonstrate the ability of the proposed VoxInstruct to convert human instructions into expressive speech, subjective and objective experiments were first conducted on an English test set. The test samples were taken from GigaSpeech-s and were not seen during training. The instruction annotation pipeline was used to generate corresponding human instructions for these samples. Since there is real speech as a reference, it is possible to calculate MCD and SECS using real speech as a reference in this section. It is found that VoxInstruct achieved the highest 4.22 on MOS-Q and 3.76 on MOS-I, significantly outperforming the two baseline models. PromptTTS's voice quality is relatively This may be attributed to its Transformer-decoder architecture and uses MSE loss. This seriously affects its subjective evaluation results and WER values. In the objective evaluation, VoxInstruct also achieved the highest average classification accuracy on the voice attribute factor and the closest similarity to the real speech, and obtained a WER value of 2.5, which is comparable to other current state-of-the-art TTS systems. This shows that VoxInstruct has the ability to understand unified human instructions, recognize the description of sound features and accurate spoken content in the instructions, and generate expressive speech consistent with the given instructions. In addition, it can be observed that the introduction of the pre-training stage brings slight improvements in voice attribute control and more significant improvements in comprehensibility, which is intuitively expected.

[0074] In addition, Chinese experiments were also conducted. Different from the English test, the Chinese instructions were generated by first randomly sampling phonetic attributes and then rewriting them using a large language model (LLM).

[0075] Although VoxInstruct performs on par or slightly worse than the baseline models on objective accuracy metrics, it significantly outperforms other models on subjective metrics MOS-Q and MOS-I with high scores of 4.01 and 3.83, respectively. This suggests that the invention also performs well in understanding Chinese instructions and generating audio. In addition, it is revealed that VoxInstruct is able to inherently understand mixed-language instructions and directly generate code-switched speech, eliminating the need for any grapheme-to-phoneme (G2P) conversion.

[0076] To demonstrate the superior fine-grained ability of the unified instruction-based audio generation method of the present invention in controlling speech, all these models were fine-tuned on a fine-grained instruction dataset and the performance was evaluated using an internal stress detection model. 200 instructions containing detailed emphasis information were used to synthesize the test samples. The accuracy of correctly detecting stressed words in all words in the synthesized speech (Acc_word), as well as the accuracy of correctly identifying stressed words in all sentences (Acc_sentence). The results show that the method adopted by PromptTTS, which focuses on the mapping between text prompts and overall speech style embeddings, has difficulties in achieving fine-grained control. In contrast, the method of the present invention, using a unified instruction prompt as input, performs well in fine-grained control capabilities compared to Salle's method, which models content and style cues separately.

[0077] VoxInstruct of the present invention is compared with zero-shot TTS models VALL-E and Vall-EX, which focus on monolingual and cross-lingual scenarios respectively. The results reflect that the present invention achieves comparable performance to the current leading zero-shot voice cloning TTS models. Although VoxInstruct is fine-tuned on the instruction dataset, it still retains a strong ability to imitate the voice in the voice prompt. In addition, the present invention significantly outperforms VALL-E in terms of naturalness and voice quality. The Vocos decoder and pre-trained MT5 text encoder provide rich semantic information.

[0078] Those skilled in the art will appreciate that the method of the present invention is suitable for various types of audio generation capabilities. For example, refer to "Advanced scene semantic understanding" example, which can be achieved by expanding on data collection (The "sound of waves" and "sound of piano" can also be understood as a scene description). In addition, the modeling methods of singing and speech are highly consistent, and can be understood as a speech with a specific intonation and rhythm.

[0079] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0080] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0081] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0082] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0083] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0084] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0085] In addition, all functional units in the embodiments of the present invention can be integrated into one processing unit. Yuanzhong, you can also Each unit can be used as a unit separately, or two or more units can be used as a unit. The elements are integrated into a unit; the above-mentioned integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0086] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0087] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0088] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0089] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0090] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0091] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for generating highly expressive audio based on natural language description text, characterized in that: The following steps are involved: S1: Use the pre-trained multilingual text encoder to encode the input natural language description instructions and extract all semantic information in the description text; S2: using the text encoding representation as a condition, sequentially predicting a semantic unit sequence of the audio through an autoregressive model (AR), wherein the semantic unit sequence represents content event information appearing in the audio; S3: Using the text encoding representation and the audio semantic unit sequence output by S2 as prefix conditions, the coarse-grained acoustic unit sequence of the audio is predicted through an autoregressive model (AR) to determine the overall duration of the audio and basic acoustic information; S4: Use a non-autoregressive model (NAR) to predict the fine-grained acoustic unit sequence of the audio in a multi-layer iterative manner, modeling the audio detail representation and high-frequency information; S5: Input the multi-layer acoustic unit sequence into a vocoder to generate an audio waveform.

2. The method for generating highly expressive audio based on natural language description text according to claim 1, characterized in that: In step S1, a text encoding modeling module supporting multiple languages ​​is used to encode the input natural language description instruction and extract all semantic information in the description text; an audio sample corresponding to the text description instruction is generated using the text encoding representation to ensure that the generated audio is consistent with the input text in terms of semantic and acoustic features; Preferably, in step S1, a pre-trained multilingual text encoder is used to encode the input natural language description instructions to extract all semantic information in the description text; a multilingual T5 base model (MT5-base) is selected as the text encoder, and its pre-trained text encoder is used to insert a trainable low-rank adaptation (LoRA) adapter to support multilingual instruction input; and the original text of the human instruction is passed to the MT5 encoder to derive a text embedding sequence.

3. The method for generating highly expressive audio based on natural language description text according to claim 1, characterized in that: In step S2, high-level scene semantic features are introduced into the data processing flow and reflected in the generated audio; when predicting the sequence of semantic units through the autoregressive model (AR), high-level scene information in the description text is considered to generate style intonation, background ambient sound, music melody, sound effect events, etc.; and the local performance of the generated audio is allowed to be controlled by instructions in the natural language description; Preferably, in step S2, the semantic unit features of the audio are extracted using a self-supervised representation model HuBERT or an audio self-supervised representation model AudioMAE; the embedding extracted by HuBERT is discretized by a k-means clustering algorithm to generate discrete semantic tags (ST); continuous repeated tags in the generated semantic tag sequence are removed to form a clear and non-redundant semantic unit sequence; a language tag is added before the semantic tag sequence to indicate the language information of the generated audio; An autoregressive model (AR) is used to predict the semantic unit sequence of the audio with the text encoding representation and the semantic tag sequence as conditions. The semantic unit sequence is directly mapped to the event content and order information appearing in the audio to be synthesized.

4. The method for generating highly expressive audio based on natural language description text according to claim 1 or 2, characterized in that: In step S3, in the scenario of voice cloning or cross-language voice cloning, the text to be synthesized is provided as an instruction; during inference, the semantic and acoustic units of the reference speech provided by the user are extracted and set as sequence prefixes in the inference process of the autoregressive model; Using the contextual learning ability of the autoregressive model, a conditional continuation task is performed to imitate the timbre and style of the reference speech. In step S4, in the sound style modification scenario, the user provides the style description that he wants to convert and the text to be synthesized as instructions; during reasoning, a conditional continuation task is performed, and the classifier-free guidance strategy (CFG) parameters are adjusted to determine whether the effect of the generated audio depends more on instruction control or reference voice control.

5. The method for generating highly expressive audio based on natural language description text according to claim 1, characterized in that: Also includes: Jointly train the text encoder, AR model, and NAR model; Using large-scale audio data without description annotations to train pre-trained multilingual text encoders to extract and understand semantic information in natural language description instructions, the audio data without description annotations only provides basic category labels of speech transcription text or audio automatically annotated by ASR; using large-scale audio data without description annotations to train autoregressive models (AR) and non-autoregressive models (NAR) to generate coarse-grained and fine-grained acoustic unit sequences; Use small-scale descriptive annotated audio data to fine-tune the pre-trained text encoder to enhance the model's ability to understand descriptive text; use small-scale descriptive annotated data to fine-tune the autoregressive model (AR) to improve the accuracy and expressiveness of generating semantic unit sequences; use small-scale descriptive annotated data to fine-tune the non-autoregressive model (NAR) to optimize the generated fine-grained acoustic unit sequence and enhance the audio detail expression and high-frequency information.

6. The method for generating highly expressive audio based on natural language description text according to claim 1, characterized in that: Also includes: In the training phase of the autoregressive model (AR), the text encoding representation or semantic unit sequence is masked with a certain probability to achieve the training of the unconditional guidance strategy CFG form; based on the partially masked text encoding representation, the coarse-grained acoustic unit sequence is predicted to generate the overall duration and basic acoustic features of the audio; In the training phase of the non-autoregressive model (NAR), the output of the autoregressive model is used as a condition to predict the fine-grained acoustic unit sequence through a multi-layer iterative method, while implementing an unconditional guidance strategy to enhance the audio details and high-frequency information. During the inference generation process, by adjusting the parameters of the classifier-free guidance strategy (CFG), the semantic description relevance of the generated audio, the accuracy of the audio content, and for speech, the pronunciation content intelligibility and the similarity to the reference audio style are adjusted to achieve fine control of the audio content style; according to the input natural language description instructions, the CFG coefficients are dynamically adjusted to meet the requirements of audio expressiveness and controllability in different scenarios.

7. The method for generating highly expressive audio based on natural language description text according to claim 1, characterized in that: In the training phase of the autoregressive model (AR), the coarse-grained acoustic markers are predicted using the autoregressive model (AR), wherein the markers include the basic acoustic content structure of the audio, the main rhythm and basic timbre of the speech and singing part, and determine the overall audio duration; During inference, the instruction text embedding and the reference audio cue are taken as input using an encoder-decoder language model to produce the target acoustic token. After generating the acoustic markers, the acoustic decoder is used to synthesize the audio waveform.

8. The method for generating highly expressive audio based on natural language description text according to claim 1, characterized in that: When predicting semantic tags, masking the text embedding sequence allows the model to make predictions in an unconditional generation form, thereby improving compliance with natural language instructions; When predicting coarse-grained acoustic tags, the masked text embedding sequence or semantic tag sequence is also generated unconditionally to further enhance the model's understanding of instructions and the naturalness of generated audio; In the training phase of the autoregressive model (AR), a coarse-grained acoustic unit sequence is predicted based on the partially masked text encoding representation to generate the overall duration and basic acoustic features of the audio; In the training phase of the non-autoregressive model (NAR), the output of the autoregressive model is used as a condition to predict the fine-grained acoustic unit sequence through a multi-layer iterative method, while implementing an unconditional guidance strategy to enhance the audio details and high-frequency information. Sampling semantic tags in logarithmic space and combining them with an unconditional guidance strategy to optimize the semantic relevance and naturalness of the generated audio; When sampling coarse-grained acoustic markers, two types of CFGs are used simultaneously, allowing the generation of acoustic markers to focus on different aspects, such as the control of audio features and the intelligibility of audio content; During the inference process, the CFG coefficients are dynamically adjusted to meet the requirements for audio expressiveness in different scenarios.

9. The method for generating highly expressive audio based on natural language description text according to claim 1, characterized in that: Jointly train the text encoder, AR model, and NAR model; In the pre-training stage, the pre-trained multilingual text encoder is trained using public audio data without description annotations to extract and understand the semantic information in the natural language description instructions. The public audio data without description annotations only provides the basic category labels of the speech transcription text or audio automatically annotated by ASR. Through model pre-training, the model has basic text-audio association modeling capabilities and audio generation capabilities, as well as stable and clear text-to-speech synthesis and zero-sample sound cloning capabilities. In the fine-tuning stage, the pre-trained text encoder is fine-tuned using instruction-audio pairing data to give the model the ability to understand descriptive information in human instructions; in the fine-tuning stage, a progressive fine-tuning strategy is adopted. First, a small-scale description annotation data is used to fine-tune the autoregressive model (AR) to improve the accuracy and expressiveness of the generated semantic unit sequence; in the fine-tuning stage, the parameters of the classifier-free guidance strategy (CFG) are adjusted to meet the adjustment of different requirements such as audio style description relevance, pronunciation intelligibility, and reference timbre similarity; thereby, the model is given the ability to understand the fine descriptive information in human instructions, thereby improving the accuracy of the generated semantic unit sequence and the expressiveness of the generated audio.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating highly expressive audio based on natural language description text as described in any one of claims 1 to 8 is implemented.

Images